search

webcompat / webcompat.com

Source code for webcompat.com
https://webcompat.com
359 stars 191 forks source link

Build global issues view #266

Closed miketaylr closed 9 years ago

miketaylr commented 10 years ago

Essentially the Flask/Backbone guts of whatever gets designed in https://github.com/webcompat/webcompat.com/issues/169

miketaylr commented 10 years ago

screnshot is roughly what we'll be building.

miketaylr commented 10 years ago

Just brain dumping/storming here:

We have API endpoints for untriaged, needs diagnosis, ready for outreach and site contacted at: https://github.com/webcompat/webcompat.com/blob/master/webcompat/api/endpoints.py#L86-L148

We could easily make another one for "closed".

It would also be nice to grab all issues and just do the filtering on the clientside--but this becomes less feasible as we grow. There is an option to get them all, via ?state=all. But we have to consider download size in the case that we have thousands of issues one day.

OK, scratch that. GitHub will actually paginate this for us, even if we ask for all (see Link header):

$ curl -I https://api.github.com/repos/webcompat/web-bugs/issues?state=all
HTTP/1.1 200 OK
Server: GitHub.com
Date: Tue, 23 Sep 2014 16:24:04 GMT
Content-Type: application/json; charset=utf-8
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1411493038
Cache-Control: public, max-age=60, s-maxage=60
ETag: "b60f7fe2fc277672d58dc20aa47fc4a6"
Vary: Accept
X-GitHub-Media-Type: github.v3
Link: <https://api.github.com/repositories/17914657/issues?state=all&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?state=all&page=12>; rel="last"
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Content-Length: 92394
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 425AD868:76F9:10C78C69:54219EA4
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: c436b2b44345c72ff906059f604991e7
karlcow commented 10 years ago

ooooh. :heart:

Link: <https://api.github.com/repositories/17914657/issues?state=all&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?state=all&page=12>; rel="last"

miketaylr commented 10 years ago

Ditching "Team" for now, #288.

miketaylr commented 10 years ago

So we've got two API sources for this: issues API and the search API. I'm going to attempt to get all the non-search data from the Issues API and leave the search API for user-initiated search.

(It's probably possible to do everything via the search API, but the rate limit would kill us.)

karlcow commented 10 years ago

@miketaylr when I was working on optimizing the code for #294 I realized that /api/issues/untriaged is in fact a filtered out version of the first page of all issues. And the same goes for the others. I don't know why I didn't realize that before. I guess that was the essence of your comment in https://github.com/webcompat/webcompat.com/issues/266#issuecomment-56547800 I need another pair of eyes for christmas ;) :eyes:

Trying to understand a bit more below.

https://api.github.com/repos/karlcow/webcompat-test/issues?labels=needsdiagnosis
https://api.github.com/repos/karlcow/webcompat-test/issues?labels=contactready
https://api.github.com/repos/karlcow/webcompat-test/issues?labels=sitewait
https://api.github.com/repos/karlcow/webcompat-test/issues

We never really have access to all issues.

Tell me if I'm wrong. In our current UI, how do we navigate older issues than the recent ones. Currently the home page is showing only the most recent ones. Luckily, we do not really offer the option to show the oldest ones first. We could not do it in the current design of the API.

Not sure there's an easy solution though.

Question:

miketaylr commented 10 years ago

We never really have access to all issues.

Not from the home page. As it's designed, it's just a "bite-sized" preview of our issues.

In our current UI, how do we navigate older issues than the recent ones.

From the homepage we don't. On the /issues page that we're building here in #266, there will be pagination and sorting, etc. Part of this task will be to add page/Link traversal to the API endpoints that the homepage uses. Or to make a more generic mechanism and ditch them altogether. :bomb: We'll see.

Not a reflection of the final product, but should give you an idea:

screenshot

miketaylr commented 10 years ago

what do you call non search data?

Sorry, very vague language on my part. In my mind the /issues page does three things: 1) filter issues 2) search issues 3) sort issues. 3 is probably just an operation on the results of 1 and 2. So non-search data is the result from a "filter" operation--clicking on "Needs Diagnosis", for example.

The point of my comment was just me thinking out loud--we can get all the same results from the Issues API with the Seach API. But it's way more expensive to do so. Probably OK if we could force all users to login, but not possible for the proxied "anonymous" approach.

karlcow commented 10 years ago

For the Issues page, aka, /issues, let me see if I understand correctly in terms of User actions and how it would work on the API side.

  1. filter issues: aka click on [contactready] and get the first page of contactready. it means a call to /api/issues/contactready and github will send back a paginated JSON.
  2. search issues: Take the search box input, send it to github and gets back the result to be displayed. (to be built)
  3. sort issues: the list is displayed and we reorganize the order of them.

Sorting issue is orthogonal to search and filter in my understanding and it is where the difficulty is without a local dump of all issues (all of that if I'm not making a fool of myself :cactus: living dangerously).

Let's say you displayed all the untriaged. Untriaged is a reduced version of the paginated all issues case. It's a list with holes. It also means that if you sort them by oldest, for example, you can't fech the last paginated link because it might contain no untriaged issues at all. You could go fetch one by one links starting from the oldest until you build up the 10 links you need for displaying in the UI, but that would be consuming resources. It's one of the issues.

Or Maybe for untriaged we need to use github search all the time.

I wonder if we have similar issues for the contactready/sitewait/etc cases. Need to think about it.

Also background thought, I wonder if it would be expensive to store all of that on the client side in one storage bucket so we could play with the data. Like how long does it take to fetch the 8 current pages of all issues compared to all the subsequent requests to create the searches, etc. :/ I guess it depends on the type of user.

miketaylr commented 10 years ago

Hmm, apparently my comment I wrote a while back (yesterday?) didn't stick.

Or Maybe for untriaged we need to use github search all the time.

I think so.

I wonder if we have similar issues for the contactready/sitewait/etc cases. Need to think about it.

Shouldn't be a problem. We can ask the Issues API to only return (paginated) issues filtered by a single label.

I wonder if it would be expensive to store all of that on the client side in one storage bucket so we could play with the data.

Would be a cool experiment. I considered this a few months back, https://github.com/pamelafox/lscache

miketaylr commented 10 years ago

OK, coming back to this after a week or travel and hacking on other things. Question for @calexity and @magsout:

If you check out staging, when you load the page you get all open issues loaded into the issues view. If you click on a filter button (untriaged, ready for outreach, etc. buttons) it will re-render the issues view with those categories.

How does the user get back to "all"? Should we click on the selected button to toggle it off and go back that way? Right now it only toggles if you click a different category.

miketaylr commented 10 years ago

(note: untriaged doesn't work yet--need to hook that up via the search API tomorrow).

calexity commented 10 years ago

Great question @miketaylr! It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

I was thinking you would just select (or click) View all open issues again and it would reset, but that might be more trouble than it's worth for a user.

miketaylr commented 10 years ago

It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

I like that suggestion. I think for now I'm going to add the "toggle" behavior, and we can tweak it after we launch.

magsout commented 10 years ago

@calexity @miketaylr

Great question @miketaylr! It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

yes, I see it like that too

miketaylr commented 10 years ago

Just deployed arbitary search over issues to http://staging.webcompat.com/issues

Next up: sorting, etc.

magsout commented 10 years ago

@miketaylr :+1:

magsout commented 10 years ago

@miketaylr You think it is possible to automatically search after typing ~3 letters. And just as like to reset automatically when the inuput empty?

miketaylr commented 10 years ago

@magsout I was thinking about that... it's also related to the question of needing a "search" button. Right now it just searches on "enter"/submit/whatever. I think it wouldn't be an issue for logged in users, but for anonymous people it will be too many requests for @webcompat-bot. The Search API rate limit is very limited (docs say 20 searches a minute, HTTP response says 30 a minute).

Thinking out loud... would it be super terrible to disable search for non-logged in users and have a "Log in to search" placeholder?

miketaylr commented 10 years ago

Ping @calexity for thoughts on previous comment about restricting search to logged in users. For comparison, we do require users to log in to set labels or comment on and close issues.

calexity commented 10 years ago

I'd rather have search open but if it's going to cause an issue with Github, its fine to say login with github to search.

Also, I forgot to call out that I used "Reported Bugs" because we use the term bug on the homepage. I'd like to replace using "issues" with "bugs " for clarity and consistency. Would also need to replace Browse issues on home with Reported Bugs. On Oct 24, 2014 8:37 AM, "Mike Taylor" notifications@github.com wrote:

Ping @calexity https://github.com/calexity for thoughts on previous comment about restricting search to logged in users. For comparison, we do require users to log in to set labels or comment on and close issues.

— Reply to this email directly or view it on GitHub https://github.com/webcompat/webcompat.com/issues/266#issuecomment-60404688 .

miketaylr commented 10 years ago

Thanks for the feedback Alexa. The other option is to just be smart about rate limits and show a message if we do run into them saying "hang on for N seconds before searching again." Or something. :) We might ship the first version requiring login, then work our way up to it, depending on how much effort it would be (I think @karlcow put in most of the plumbing for that already).

calexity commented 10 years ago

Excellent - a limit would be a solid solution. A regular user (not a bot) would only submit a few search requests at a time.

On Fri, Oct 24, 2014 at 1:09 PM, Mike Taylor notifications@github.com wrote:

Thanks for the feedback Alexa. The other option is to just be smart about rate limits and show a message if we do run into them saying "hang on for N seconds before searching again." Or something. :) We might ship the first version requiring login, then work our way up to it, depending on how much effort it would be (I think @karlpro put in most of the plumbing for that already).

— Reply to this email directly or view it on GitHub https://github.com/webcompat/webcompat.com/issues/266#issuecomment-60442194 .

miketaylr commented 9 years ago

Related ticket: #309

miketaylr commented 9 years ago

Need to investigate: https://github.com/webcompat/webcompat.com/commit/27482c77a009ce2dccc2560231f68c8f71a102bb#commitcomment-8359530

miketaylr commented 9 years ago

Oh, maybe https://twitter.com/githubstatus/status/527583627123912704.

We're actively mitigating a large DDoS attack.

Just don't tell @karlcow. ^_^

miketaylr commented 9 years ago

Oh, maybe https://twitter.com/githubstatus/status/527583627123912704.

Nope, just me writing bugs. Fixed now.

miketaylr commented 9 years ago

OK, #312 is out the door. Will file smaller bugs to track enabling the rest of the features. We should probably figure out our milestone process--seems like it would be useful to help us ship the rest of this feature.

magsout commented 9 years ago

We should probably figure out our milestone process--seems like it would be useful to help us ship the rest of this feature.

yes.. agree

miketaylr commented 9 years ago

I've created a Milestone "Ship full issues page" to track the smaller parts.

miketaylr commented 9 years ago

But let's close this issue. The milestone will serve as the "meta".