Investigate & Enable Sharded Gitaly

[jacobbmay]

Gitaly cluster is still not supported in kubernetes. In order to support larger target user counts, we should investigate supporting configuring sharded Gitaly with our bundle as a compromise to not having Gitaly cluster.

### Tasks
- [ ] Aggregate relevant documentation
- [ ] Produce testing strategy
- [ ] Produce design approach
- [ ] Get it working in dev
- [ ] Update docs
- [ ] Get PR merged
- [ ] Update UDS_CONFIG in 1 password

[JoeHCQ1]

Links:

• https://docs.gitlab.com/ee/administration/gitaly/ general gitaly page

• https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html the k8s discussion, most of what's relevant - actually no, only relevant to general tuning, not to shards.

• https://docs.gitlab.com/ee/administration/gitaly/gitaly_geo_capabilities.html#gitaly-capabilities has a reference to sharding. Sharding is a manual process.... This sounds sub-optimal for sure. Do we know we need sharding and not just replicated repo read/writers?

• [ ] Address pod disruption

• [ ] Address resource contention and saturation - variable insert whole resource blocks for easy manipulation, dito on labels/taints to make it easy to move Gitaly nodes elsewhere.

[JoeHCQ1]

• https://dev.gitlab.org/cookbooks/runbooks/-/blob/gitaly-total-cpu/howto/sharding.md points me at docs re: sharding with summary
• https://docs.gitlab.com/ee/administration/repository_storage_paths.html#configure-where-new-repositories-are-stored looks like sharding is more something we do manually than something we enable - perhaps why "shard" and "sharding" aren't returning hits in the docs search. It's seeming that you do this by giving extra storage paths to GitLab, and then telling the admin UI where to normally put stuff. Not sure if each path corresponds to a gitaly install. This line _Physical storage configured with a gitalyaddress that points to a Gitaly node. suggests to me it is indeed a gitaly node per shard.
• Gitaly is a sub-chart: https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html?tab=Helm+chart+%28Kubernetes%29
• https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html?tab=Helm+chart+%28Kubernetes%29#network-architecture is super useful in understanding how Gitaly, repository paths, and gitlab interconnect.

[jacobbmay]

That is correct. "Enabled" in the ticket title meant enable configuring it in our bundle. Sharded gitaly is just deploying multiple separate instances of Gitaly so you can distribute repositories and the related API calls between them. The Gitaly chart has a value for defining multiple Gitaly instances that it should create. The manual steps would be in GitLab configuring the distribution of resources between the shards.

[jacobbmay]

I believe it is the 'internal.names' value in the chart. It defaults to list containing a single value named 'default'. If we override it and pass multiple it should create a Gitaly instances per item in the list. The first one should be 'default' to enable deploying multiple as an upgrade. Guessing it won't like it if the name changes.

https://docs.gitlab.com/charts/charts/gitlab/gitaly/

[jacobbmay]

I thought I had included that in the item description but I guess I forgot to.