Scale repo-updater WIP

[tsenart]

Context

RFC191 has been written and this ticket details the implementation level issues that need to be completed roughly in the correct order.

Tasks

Scaffolding to allow second instance

• [ ] Create leader election package
  • [x] Use the Redis mutex package https://github.com/sourcegraph/sourcegraph/pull/12289
  • [ ] The current leader should temporarily write its hostname to redis so that we can route incoming requests correctly
• [ ] Test it in a service that already runs multiple instances, perhaps frontend
• [ ] Update repoupdater.Client to query for current leader from redis to route requests to correct instance
• [ ] Wrap all repo-updater tasks in leader election wrapper
  • [ ] Repo syncing
  • [ ] Repo updating
  • [ ] Phabricator syncing
  • [ ] GitolitePhabricator metadata syncing
  • [ ] Repository purging
  • [ ] Expired patchset deletion
  • [ ] Changeset syncing
  • [ ] Permission syncing
• [ ] Update deployment to allow multiple instances, perhaps include rename to code-host-syncer

RFC 191 Implementation

• [ ] Redis based rate limiting
  • [ ] Rate limiter
  • [ ] Semaphore for concurrency limiting
  • [ ] Switch out current in memory version

The following could be done concurrently:

• [ ] Implement db backed repo sync queue

  • [ ] Add new table for use with https://github.com/sourcegraph/sourcegraph/pull/12058
  • [ ] Update scheduler to write to queue
  • [ ] /enqueue-repo-update can be removed and client will talk directly to db
  • [ ] /repo-update-scheduler-info can be removed and client will talk directly to db

• [ ] DB backed changeset syncer

  • [ ] Move logic into SQL query so that it returns the next item to sync (or none)
  • [ ] Add item(s) to sync queue so we can use https://github.com/sourcegraph/sourcegraph/pull/12058
  • [ ] Remove syncer per code host and use a pool of workers
  • [ ] /enqueue-changeset-sync can be removed and client will talk directly to db

• [ ] DB backed permission syncing

  • [ ] Move queue into db and use https://github.com/sourcegraph/sourcegraph/pull/12058
  • [ ] /schedule-perms-sync can be removed and client will talk directly to db

Other ideas

(These are here from the original issue description so leaving them around for some context)

• Leverage the webhooks infrastructure we built for campaigns on repository and permissions syncing.
• Introduce scheduler based heuristic syncing for repositories (i.e. similar to what we've done to changesets and permissions).
  • Reduce memory usage of Syncer by processing each sourced repo with SyncSubset rather than aggregating all repos in memory and then making a NewDiff.
• Move git update schedule state from memory to Postgres.
• Move permissions update schedule state from memory to Postgres.
• Move changesets update schedule state from memory to Postgres.
• Move last sync error from memory to Postgres.
• Move rate limiter's state from memory to Redis (high write rate).

[keegancsmith]

FYI some of the original inspiration for how repo-updater's git scheduler was how phabricator does mirrored repos. (The simple heuristic). Phabricator stores the queue and state in the DB. I would encourage you to take a look at the code for some inspiration (and readable PHP code :P )

[ryanslade]

@keegancsmith @tsenart I've made a first pass at documenting the current responsibilities of repo-updater here (Under the Problem heading): https://docs.google.com/document/d/1Zvv9XqD6qH-Kmte5n0IPuPosUx4fblzsQFGi3kGmu8Y/edit

Is there anything I've missed?

[tsenart]

Is there anything I've missed?

Commenting on the RFC.

[daxmc99]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.18 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

Thank you