Link Issues <-> PRs

[mcomella]

Prior art: https://github.com/mcomella/github-issue-hoister/ Current efforts: https://github.com/mcomella/github-issue-pr-linker

Refined GitHub does something like this too: https://github.com/sindresorhus/refined-github/blob/master/source/features/highlight-closing-prs-in-open-issues.js

[mcomella]

To link PR -> issue, the PR must indicate which issue(s) it affects. We can identify this by searching in the:

• PR title: Issue/Closes/Resolves #... and #...
• Commits: ^ same
• PR comments: we can find Issue/Closes/Resolves #.... However, people often use #... to refer to other things so we shouldn't use it to link Issues and PRs.

[mcomella]

To link issue -> PR, we need to leverage the links that PRs have created. In an issue page, GitHub provides us:

Guaranteed issues -> PRs/commits

• Hover labels, via open PRs. In DOM: aria-label*=will close once pull request (via Refined GitHub, note English bias). These occur when:
  • [commit link] Commits contain Closes #... (example)
  • [issue link] Issue contains Closes #...
• "Closed this in..." via closed PRs.
  • [commit link] <commit> (example)
  • [issue link] <issue-num> (example)

It's unclear what happens if both the PR comments and commits reference an issue.

Non-guaranteed: PRs reference issue but may not address issue

• [commit link] "A commit referenced this issue"; these may not be in a Pull Request at all
• [issue link] "Referenced this issue"; these don't necessarily address this issue

---

To reverse this from a user perspective, references in:

• PR titles do nothing (medium for user)
• Commits (easy for user)
  • With "Closes" create less useful commit references (see below)
  • With "Issue" don't guarantee they're attached to a PR yet
• PR comments (hard for user)
  • "Closes" create true issue <-> PR links
  • "Issue" doesn't guarantee they're addressing this issue

Commit links aren't very useful because we have to crawl the commits of all open PRs in order to find which PR it belongs to (afaict). However, the preview API allows us to search issues by commit hash.

[mcomella]

To find guaranteed PR -> issues in all cases (not just "Closes"), we'll have to cache our own PR -> issue data:

• When visiting a PR, scrape the PR data (title, commit, comments) for issue data (see https://github.com/mcomella/apt_github_improvements/issues/4#issuecomment-409084808)
• When visiting the PR list, scrape the open/closed PRs (titles)
• Periodically ping the API to cache the open PR list (it'd be an additional API call each to check commits, comments, assuming less than a page worth so just titles?)
• ? crawl closed PRs to increase the database
• (later: use "closes", other?, commit references to look-up PR by commit)

Things we may want to report to the user:

• Is it an "issue" or "closes" reference?
• PR status: is it open? Closed? Reopened? Merged?
• PR data last updated...

Things to watch out for:

• There is a limited amount of storage: pruning (by last accessed?)
• API limits: how can we politely ask the user to add their API token?
• Can we avoid reporting PRs which later change their data? (e.g. typo Changes #3 and rewrite to Changes #4)?
• Attaching page mutation observers to get data when an open PR page changes?

[mcomella]

There is also fix and stuff: https://help.github.com/articles/closing-issues-using-keywords/

[mcomella]

There are two types of use cases:

• User in PR clicks Issue and expects to get back
  • Need to scrape
• User in Issue wants to see/access PR
  • Need to use API to be exhaustive; links provided in the page only cover "Closes" references

My suggestion algorithm is:

• If we're on a PR, scrape it and update the last updated time for this PR
• If we're on an issue, fetch the open PR list if we haven't in <5 min>. For each PR not fetched in <30 min? 60?>, fetch their commits and extract numbers.

Notes:

• We don't address issue numbers in PR comments: this would be an extra API call (or a few) so it seems fine to leave these out: I think most people link from the other two anyway.
• We fetch PRs less frequently than the PR list because they drive up the rate limit quickly
• If we were to use the search API preview, we might be able to search within PRs instead of crawling the PR data like we do
• We don't want to crawl the open PR list because it adds complexity but would only save us 1 API call. Crawling the closed PR list might be more useful so people can get back (but we can hoist "closed in commit" links too).

Concerns:

• Performance of making all of these requests?

---

My suggested schema:

v: <dbVersion int>,

/* Map issue -> PR; PR -> issue is just scraped */
issue-o/r/<num>: [{ /* PR num */
  num: <int>,
  closes: <boolean>
}]

/* last updates times for rate limiting; we can collapse these if no other keys */
issue-list-o/r: {
  lastUpdated: <int>
}

pr-o/r/<num>: {
  lastUpdated: <int>
}

Have to watch out for:

• Schema migration. If we don't want to crawl the whole DB,
• Pruning: we have no data to know which issues are oldest. We could do a one-time DB dump when we add this feature though

We're not handling:

• Cross-repo references

---

FYI: rate limits are:

• 60/hr unauthenticated (this may be slow even for this app; should we let users disable this feature? or disable if no API key?)
• 5000/hr authenticated

[mcomella]

I landed the basic implementation. Before release, we additionally need:

• The ability for users to add access tokens
• Additional testing
  • Verify recache duration is acceptable
• Readme for feature

We may need:

• Storage cleanup (automatic pruning or manual delete option)
• Onboard users to add access tokens
• Notify users when they hit the rate limit

Things we won't address intentionally, for implementation simplicity/time:

• If a PR content changes (e.g. original issue num was typo), we won't remove it's older content (since we don't store the original PR data)
• Adding page mutation observers to update the DB if a PR changes while we're watching
• Cross-repo issue references

[mcomella]

We also need to consider paging on the github api.

[mcomella]

The MVP is implemented and I filed follow-ups for additional features. In particular, we should consider doing #21 before we ship.