Create Github solution to display issues in multiple repos

[edrushka]

Driver: @edrushka Dev: @ScottDowne Consulted: @simonwex Informed: @OpenMatt @hannahkane @ldecoursy

Problem: tickets that PM's are responsible for driving are spread across several repos. There’s no central place for PM's to see all the things we’re working on (especially the small, misc tasks).

Can we build a way to see all the tickets that a single author has on the go, across repos?

I'm happy to use any means (labels, milestones, assignees, ticket author) - whatever is the easiest to implement, and would work well for others.

The program/product managers will shower you with love if you can help us with this! :gift_heart:

[davidascher]

If you come up with a design for a view that we display on build.w.o it should be quite simple to implement.

[edrushka]

Scott is going to investigate this and let us know if/when we need design.

[edrushka]

Further details:

I can search by:

Author (and/or) Assignee (and/or) Label (and/or) Milestone (and/or) Repo (and/or)

It generates a list that shows:

Repo name Ticket Author Assignee Milestone Labels

[ScottDowne]

First i'm looking at github's current search capabilities, because not many people know how to use it. It's complete, but uses a search box that you need to remember commands for, or a complex url. Not really a great option to rely on that alone.

If you navigate to https://github.com/issues you get some pretty simple search options, created, assigned, mentioned. This so far is only from the logged in user's point of view. So if I click those I see my assigned, created, or mentioned issues. Also not really super complete. I want to see issues assigned or created by other people, and filter by milestone, repo, or label.

Search box examples:

Author (and/or): author:edrushka Assignee (and/or): assignee:edrushka Label (and/or): label:platform Milestone (and/or): milestone:"Inspector Gadget" Repo (and/or): repo:mozilla-appmaker/appmaker

You can add as many of these search filters as you like, separated by a space, and multiple labels would look like this: label:platform label:curriculum

Remembering repo names, and user names is kinda annoying. It's not always obvious.

This search box just redirects to urls, which looks like this:

URL Examples:

Author (and/or): https://github.com/issues?utf8=%E2%9C%93&q=author%3Aedrushka+ Assignee (and/or): https://github.com/issues?utf8=%E2%9C%93&q=assignee%3Aedrushka+ Label (and/or): https://github.com/issues?utf8=%E2%9C%93&q=label%3A+platform+label%3Acurriculum+ Milestone (and/or): https://github.com/issues?utf8=%E2%9C%93&q=milestone%3A%22Inspector+Gadget%22+ Repo (and/or): https://github.com/issues?utf8=%E2%9C%93&q=repo%3Amozilla%2Fteach.webmaker.org

Documentation from github for all this: https://help.github.com/articles/searching-issues/

[edrushka]

Thanks! I see that there are various solutions, but not all in one place.

@davidascher what are your thoughts? Scott and I discussed a front end solution that generates URLs that we can plug into existing Github search boxes. This is kind of a 2-step hack, but it could be easy to build.

I'm also going to see if I can bring this to the attention of the Github team (Scott's suggestion). Maybe they have something in the works. At the very least, it's user feedback.

[davidascher]

A widget on build.w.o that when you hit "search" opens the search results page in github issues seems like a fine next step to me. In my opinion the hard part isn't the display of the results as much as building that front-end UI in such a way that it makes sense, especially given that the repos we care about are spread over multiple organizations but aren't all the repos in those organizations.

[edrushka]

@ScottDowne Here are the repos we use:

Mozillafoundation /plan /engagement /movement-building /2015-fundraising

Mozilla /mozillafestival /teach.webmaker.org /id.webmaker.org

MozillaHive /HiveGlobal/issues

[ScottDowne]

Wow, that's a much smaller list than what I expected. Thanks!

[xmatthewx]

There's also...

Mozilla /webmaker-app /webmaker-desktop /api.webmaker.org

[davidascher]

loads of other repos

popcorn thimble sawmill webmaker-curriculum …

[ScottDowne]

Initially I thought I wanted to make this static.

Then I got a bit sucked into doing something I have never done. I now get all repos under the orgs mozilla and MozillaFoundation. I store those server side every 10 minutes right now (probably want to increase this).

The ui drop down auto completes the list, and it's quite nice to use. I'm demoing it tomorrow.

[edrushka]

Whoa! All the repos?! Quite nice to use?! Here's a poodle for your good work: :poodle: Can't wait to see the demo!!!

[edrushka]

I got a reply from Github:

No, I don't think that exists currently. There's no search form which lists all repositories and all users from your organization (or across organizations even). We don't comment on product plans or timelines publicly, so I can't say if/when something like that might be available. Still, I'd be happy to pass your suggestion to the team to consider. Thanks!

In the meantime, you might consider using the API to build something that fits your workflow:

https://developer.github.com/v3/

The API allows you to fetch all issues in a repository and search for issues in a repository or organization using the same filters as on github.com. That should allow you to build a consoleapp or webapp which provides the features you want.

[ScottDowne]

I'm pushing up some changes later tonight when I'm back on my computer.

Something I don't like about the github API approach of it, it's a lot of requests. While I am caching it and making few requests, it makes dev a headache. One refresh of the server and you just used up 1/5th of your hourly github requests. Not amazing.

The majority of requests come from getting milestones and labels. We need to do this for labels once for each repo and then again once for each repo for milestones.

Another solution is setting up webhooks in github on all our repos and dumping that into redis. So github can tell us when a new milestone or label is created and we only need to update our data set then.

The code I have already written is going be adapted to create the webhooks (too many repos to do that manually) and to get the initial dataset.

I also mined a hige list of repos we used or have used that I'll drop in a gist and update in here. I want to trim it down with a human eye. A lot of dead repos.

[davidascher]

Get a token. You get unlimited queries.

[ScottDowne]

@davidascher

From: https://developer.github.com/v3/#rate-limiting

"For requests using Basic Authentication or OAuth, you can make up to 5,000 requests per hour. For unauthenticated requests, the rate limit allows you to make up to 60 requests per hour. "

I looked at other areas, but seems they no longer offer unlimited? Or I am missing something about their auth and tokens.

[davidascher]

Wow I guess you're just making a LOT of calls! :) On 17 Apr 2015 5:43 pm, "ScottDowne" notifications@github.com wrote:

@davidascher https://github.com/davidascher

From: https://developer.github.com/v3/#rate-limiting

"For requests using Basic Authentication or OAuth, you can make up to 5,000 requests per hour. For unauthenticated requests, the rate limit allows you to make up to 60 requests per hour. "

I looked at other areas, but seems they no longer offer unlimited? Or I am missing something about their auth and tokens.

— Reply to this email directly or view it on GitHub https://github.com/MozillaFoundation/plan/issues/425#issuecomment-94106750 .

[ScottDowne]

Haha, yeah. Initially when I added labels and milestones I hit 5000 just starting the server, I have it down to 360 now.

The current list of repos I grabbed is 143 repos, can probably cut that by 3/4.

Example, to get milestones, I need to make another request. A repo's data looks like this

https://api.github.com/repos/mozilla/popcorn.webmaker.org

Look for milestones and labels in there, and you'll see it's a link to another request instead of the data itself.

They also limit results to 30 per page, so I need to make multiple requests to get complete sets in some cases.

So say you have 100 repos, it's 200 calls to get labels and milestones, assuming everything is one page of data. My cache right now is in memory, so change and restart the server 25 times and you're locked out.

Been an interesting adventure learning their API, but it's not really designed for what I'm trying to do. I'm really leaning towards webhooks, or a finely tuned list of static repos. (moco has A LOT of repos that I removed)

[ScottDowne]

Here is the list of repos: https://gist.github.com/ScottDowne/a171add1c1a4efdd28e7

That is based on who has touched it. It's sorted by most recently changed to last, at the time I retrieved it. So likely repos at the end need to be dropped first.

[davidascher]

I have code using webhooks and keeping state in firebase. Will send out of band. On 17 Apr 2015 6:18 pm, "ScottDowne" notifications@github.com wrote:

Here is the list of repos: https://gist.github.com/ScottDowne/a171add1c1a4efdd28e7

That is based on who has touched it. It's sorted by most recent change to last, at the time I retrieved it. So likely repos at the end need to be dropped first.

— Reply to this email directly or view it on GitHub https://github.com/MozillaFoundation/plan/issues/425#issuecomment-94109577 .

[ScottDowne]

Live example here: http://thecount-build.herokuapp.com/issues

Pull request here: https://github.com/mozilla/build.webmaker.org/pull/71

It's been reviewed, but I'm not sure how to push it to prod yet, not seeing it in heroku under something I have access to.

[ScottDowne]

@davidascher or @thisandagain If I push this to master does it auto build the live site or do I need to do something to initiate that?

[davidascher]

Push to master should auto deploy via Travis

[ScottDowne]

@davidascher Awesome.

It has been R+'ed so I can merge it, but I wouldn't mind if you take a drive by if you have a chance :)

[edrushka]

@davidascher any updates? How can we get this lovely feature on build.wm.org?

[davidascher]

I shouldn't be in the critical path. Jbuck can find another reviewer. On Jun 4, 2015 3:19 PM, "Erika Drushka" notifications@github.com wrote:

@davidascher https://github.com/davidascher any updates? How can we get this lovely feature on build.wm.org?

— Reply to this email directly or view it on GitHub https://github.com/MozillaFoundation/plan/issues/425#issuecomment-109071384 .

[edrushka]

@jbuck - can you help us make this happen? https://github.com/mozilla/build.webmaker.org/pull/71

[ScottDowne]

Previously Travis wasn't updating the live site but I asked jbuck today and he said it's fixed. Kinda fell off my radar when mass surveillance and mozfest came up.

I'll give it a try tomorrow.

[edrushka]

@ScottDowne any progress? This is such a great tool and no one knows about it! Would love to find a home for this.

[edrushka]

It's live! http://build.webmaker.org/issues Woooo!