[GitHub] Reduce/eliminate sharing of personal access tokens (PAT)

[omgitsbillryan]

The Problem

GitHub personal access tokens (PAT) for bot users are used in many ways & places. If this PAT hits the 5k/hour rate limit, all of the places where the PAT is used will now be blocked. If more than one application is using the same bot, they should have unique tokens. The impact of hitting this limit is that build and deploy pipelines will fail.

More Context

This is a postmortem action item (link), where the github-exporter began spamming requests to GH API using a bot-user token that was used for all/most of our Jenkins jobs.

Tasks

• [ ] Create new tokens and/or new bot users for the various usages & "apps"
  • log into github as the bot and add new tokens.
• [ ] Update existing PAT usages to use new PATs (new bot users or new tokens from same bot user)

Acceptance Criteria

• [ ] Discrete applications that we own using GitHub PATs from the bot accounts are using distinct tokens

[dginther]

We definitely want to do this, as I would consider it best practice.

We have taken some actions to mitigate the issue and we made discoveries during the research of the issue that let us know more about what is using these tokens, and it would be easier to track down if it were to happen again.

It would be even more simple to troubleshoot once we have made this change.

[mchelen-gov]

Note: this helps with PAT sharing but does not provide visibility into GH API usage by bot accounts https://dsva.slack.com/archives/CJYRZK2HH/p1633550579382300?thread_ts=1633529226.365600&cid=CJYRZK2HH

[mchelen-gov]

Which Github bot does this refer to?