Core Services: 3.5 Tracking Issue

[keegancsmith]

Goal

Improve support for instances with 30,000 repositories.

Availability

• @mrnugget out on the 30th of May as well as the 10th, 20th and 21st of June. 8th-10th July (next iteration).
• @tsenart out for a wedding from Friday the 14th of June until the 18th.
• @keegancsmith out for a SA public holiday. Friday the 14th until 17th.

Dependencies

For this iteration we have some needs and wants that we think other teams are already focusing on. This list is based on our current understanding and is prone to change with new information.

• Ensure that our large scale environments have appropriate data in them and can support continuous load testing while reflecting the setups of our most important customers.
• Shortened feedback cycles after pushing code
  • CI time
  • Fast deployment to multiple environments
  • Understanding what is in each environment
  • More sentry integration?

Proposed Work

• [x] Postgres reliability

  • [x] insert: pq: duplicate key value violates unique constraint" on clean 3.4.1 instance #4210
  • [x] management-console: ... Failed to migrate the DB... can’t acquire database lock during upgrade from 3.2.1 to 3.4.1 #4208

• [x] Transfer ownership of auth{N|Z} by implementing Bitbucket Server authn and authz (ACLs) (#1108)

• [ ] Improve UX for site admins with 30k repos

  • [x] Fetch GitHub repositories in batches #4223
  • [x] "Not cloned" repositories page times out if there are a lot of repositories and none are cloned #4063
  • [x] Visiting a repository for the first time shows "starting to clone" but doesn't start #3447
  • [x] "Empty repository" banner shown on repo page in between enabling and cloning a repo #1744
  • [x] Special case org: in repositoryQuery so that it doesn't use the limited Search API. #4036
  • [x] Delay deleting repos and indexes to be resilient against admin mistakes in configuration (#2572)
  • [x] Cloning repositories return 404 Not Found (#4081)
  • [ ] Add a global "repositories are cloning"/"Sourcegraph is not ready yet" indicator #4120 [WIP @mrnugget]
  • [ ] When adding 1000's of github repos at once, there is no feedback in the UI for hours at a time #3413
  • [x] repos: Indicate schedule for cloning in UI #4232

• [ ] Improve Zoekt memory usage https://github.com/google/zoekt/issues/86 [WIP @keegancsmith]

• [ ] Synthetic load testing and complementary monitoring. Will do in testing week. Ancient implementation

  • [x] Load test Bitbucket Server ACLs
  • [ ] Load test repo page
  • [ ] Load test indexed search
  • [ ] Load test non-indexed search
  • [ ] Measure and alert on time from commit upstream to indexing
  • [ ] Measure and alert saving an external service config that yields 30k repos

Stretch Work

• [ ] Dogfood traces for common paths so that these are useful (don’t have to jump straight to logs)
• [x] Search results linking to wrong revision #3992
• [ ] Investigate feasibility of anonymized diagnostic (core)dumps of Postgres tables
• [ ] enqueue-repo-update should use an ID #3345
• [ ] Redis reliability
  • [ ] Redis AOF file grows unbounded due to frequent container restarts #3300
  • [ ] Redis AOF database corruption #651
  • [ ] redis: Log spam warning when redis is loading the dataset in memory #2904
  • [ ] Redis OOM error prevents sign in or using server #296

[mrnugget]

This comment by @slimsag on Slack and the ticket he mentions (https://github.com/sourcegraph/sourcegraph/issues/4063) might also be relevant to our overall goal. Since we finalized the planning yesterday, though, I'll just drop this here as a comment.

[keegancsmith]

This comment by @slimsag on Slack and the ticket he mentions (#4063) might also be relevant to our overall goal. Since we finalized the planning yesterday, though, I'll just drop this here as a comment.

Added to milestone and tracking issue. We may have to shift some stuff around so that it is reasonable for this milestone. But I agree, this fits into our goal for this milestone and is high prio so lets do it.

[keegancsmith]

Here is a very old implementation of load testing we did in 2016 using vegeta https://github.com/sourcegraph/enterprise/tree/a4c3283198105242f994ca5a2d56a9d6addcaf3b/test/load