Source tracking issue: Stop the bleeding

[mrnugget]

Goal

Make the system predictable, reliable, debuggable; ensure team’s mastery of it by getting the code and the team to a level of confidence about our systems that allows us to reason about it and provide great support.

Work

Port legacy code to new patterns (or delete), simplify, add debug features, e2e tests; cherry-on-top: remove repo-updater

Impact

Reduced support load, happier site-admins, increased future velocity, enabled more innovation, stretch: easier deployment

Milestones/Tasks

This section contains high level goals, and is not a fully exhaustive list of what we'll be working on in detail. We expect to find bugs along the way, which we will prioritize as well. After all, whatever can create support load we want to fix, next priority is working towards our north stars.

Reduce support load 33d

• add e2e tests 15d
  • Cover basic cloning from all supported code hosts 5d
  • Cover repo syncing from all supported code hosts 5d
  • Have basic authz tests for all supported providers 5d
• Git repositories don't get corrupted or corruption is automatically fixed 6d
  • https://github.com/sourcegraph/sourcegraph/issues/39028 6d
  • Improved repo lifecycle will allow us to not clean up repos while they're being updated
• Write Playbooks/troubleshooting guides 12d

Improve visibility into repository lifecycle 26d

• This is probably going to be a revival of what was attempted in the last quarter: Move the clone/fetch queue and jobs into a DB worker: 20d
  • Make clear when and why a repo is cloned and fetched (audit logs)
  • Make clear when we run cleanup tasks
  • Access to logs from the above processes
  • Allow to cancel a running fetch or clone from the UI, in case it got stuck
  • https://github.com/sourcegraph/sourcegraph/issues/57631
• Add history of code host connection edits like site-config 4d
• https://github.com/sourcegraph/sourcegraph/issues/57634 2d

Increase confidence in our codebase by cleaning up tech debt 36d

• Gitserver doesn't have a unique mapping to a repo in the DB (name is not unique) #56712 5d
• Gitserver should manage lifecycle of repos, including discovering when to clone and when to delete after repo was removed 3d
• We need to figure out how customers use authz/code hosts/rate limits/… 3d
  • Pings data! We need to know what configs are used.
• Timeboxed effort towards North star: Get rid of repo-updater 10d
• Timeboxed effort towards North star: Pluggable git backends and understand usage/performance of functions by introducing new gRPC endpoints 15d

Performance: Fix the biggest memory/CPU sinks 15d (timebox)

• https://github.com/sourcegraph/sourcegraph/issues/57522
• https://github.com/sourcegraph/sourcegraph/issues/57474
• https://github.com/sourcegraph/sourcegraph/issues/57635

North stars

_This section describes directions we want to work towards, but not necessarily ship this quarter. Decisions we make should be influenced by these to provide a path way for achieving them eventually.

North star: Get rid of repo-updater

• Helps with deployment, debugging, reduces resource requirements, less distributed systems
• Good forcing function to refresh” the code that’s in repo-updater right now, when moving it over, and reconsidering concepts

North star: Pluggable git backends and understand usage/performance of functions

• The team has a deep understanding of the internals of git, we need to know how the ODB works, how the git cli "frontend" commands operate
• Get rid of the exec endpoint in favor of specific well-typed RPC calls that we can test and instrument individually
  • No more parsing logic in gitserver.Client
• Have instrumentation in place that helps us understand where load is coming from and what improvements we can make for stability and performance
  • Add scopes to gitserver calls to understand what is calling us
• Map the RPC calls to generic VCS backend calls, so in the future we can plug in other backends (libgit2-based, pure go implementation, potentially another backend (hg, perforce,..) altogether)

North star: Services can react to events they're interested in, no more high baseline load from constant polling

• Today, we do a lot of polling from various background processes to understand when to trigger a new job to do something
• We should understand what the processes doing polling are trying to achieve, and emit events to decouple gitserver logic from applications that build on top of it
  • Event ideas: reference updated, repo cloned, maybe others

/cc @sourcegraph/source

[eseliger]

Made substantial progress towards more stable services.

• Gitserver is restructured a lot to be easier to reason about
• Git interactions have a much better test bed and coverage now, we're still expanding this to ALL interactions
• repo-updater has less responsibilities, as a result we're closer to being able to sunset it
• Numerous fixes for resharding events, as have happened on dotcom recently
• Supported cost reduction effort to drive down cost of Sourcegraph.com
• Many more small changes, I don't want to list 100 PRs here that made various improvements that will help us really nail down on stability next quarter

[eseliger]

QA Test Plan

Summary

See summary above.

Test Plan

Functional testing: Verify that the application functions as expected and meets the requirements. This will include testing of user interfaces, inputs, outputs, and all other functionalities.

Dotcom is our test bed, thousands of users use it every day and help us discover issues. At the time of writing this, none that are newly introduced by our changes are known.

Performance testing: Verify that the application performs efficiently and effectively under different loads and stress levels.

We've monitored the performance of Gitserver and repo-updater throughout this work in prometheus, and have not seen any potential bottlenecks.

Cody Checklist (If applicable)

1. Does your feature respect the global Cody feature flag? Default experience is OFF for Self Hosted

   • [ ] Yes
   • [ ] No
   • [x] N/A

2. Have you backtested prompts and questions that have worked before? IE - Can you summarize the following file?

   • [ ] Yes
   • [ ] No
   • [x] N/A

   If yes, please specify your test prompts and the response:

3. Have you made any client-side changes to Cody? To prevent breakages, please ensure your changes have been tested on other Cody platforms (Web, App, etc)

   • [ ] Yes, I've tested my changes across the different applicable Cody platforms
   • [ ] No
   • [x] N/A

4. Which clients have your changes been tested on?

   • [ ] VS Code
   • [ ] Cody App
   • [ ] Sourcegraph Web
   • [ ] IntelliJ
   • [ ] Other

   If other, please specify:

   5. Please provide any additional testing you've done that has not been covered above:

   If you notice any bugs related to the setup experience or Cody in general please raise them in #team-cody and tag @teamcody & @design-team.

QA Checklist

1. Have you made any infra related changes to environment variables, new services, or deployment methods that could affect customers?

   • [ ] Yes - I've informed #team-cloud and #team-delivery of the changes.
   • [ ] Yes - Changelog or documentation has been updated, this includes changes to defaults and site config flags.
   • [x] No

   If your change is non-trivial, please review the Cloud Launch process.

   If you've made changes to documentation, please link them in the comments below.

   Comments:

2. Which environments have the changes been tested on?

   • [ ] rctest.sourcegraph.com
   • [x] sourcegraph.sourcegraph.com
   • [x] sourcegraph.com
   • [ ] cody-dev.sgdev.dev (running based off 5.2 release branch)
   • [ ] scaletesting.sgdev.org
   • [ ] other

   If other, please specify:

3. Experimental features have been marked and behind a feature flag?

   • [ ] Yes
   • [ ] No
   • [x] N/A

   If no, please specify why:

4. Has telemetry been added as part of the product requirements?

   • [ ] Yes
   • [ ] No
   • [x] N/A

5. Completed entry to release post.

   • [ ] Yes
   • [ ] No
   • [x] N/A

6. Is a feature flagged in a way when turns the feature off, it behaves as-if the feature does not exist?

   • [ ] Yes
   • [ ] No
   • [x] N/A

   If no, please specify why:

7. A CHANGELOG entry has been added for the feature/change?

   • [x] Yes, for the changes that warranted one
   • [ ] No
   • [ ] N/A

8. Please provide any additional testing you've done that has not been covered above:

---

Tech Lead/DRI sign off: @eseliger