Configure gitaly with recommended resiliency settings

[jacobbmay]

Gitlab recommends if you are going to run Gitaly in kubernetes to configure it cgroups enabled in the chart and that the pod is able to access a node mountpoint to /sys/fs/cgroup.

The instructions for configuring that via chart values are here.

EDIT: Turned this into a general improved tuning ticket as there were a few other settings I encountered in performing #213 which should be checked as well.

### Tasks
- [ ] cgroups control enabled and configurable at deploy time
- [x] Address pod disruption (https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html)

[jacobbmay]

Specific recommendations about limits relative to pod resources are mentioned here. This helps prevent gitaly from getting OOM terminated.

[JoeHCQ1]

Removed myself b/c not actively working this as I switch to the RKE2 installation instructions. Need to redeploy the EKSD clusters into RKE2.

[jacobbmay]

Ticket title was misleading as we are not configuring HA Gitaly until GitLab finishes getting Gitaly cluster in Kubernetes ready for general availability. Changed to "HA" to resiliency.

[JoeHCQ1]

• [X] Add Priority Class to make gitaly the last pod to be evicted when node is under pressure - this was hardcoded into the bundle and additional manifests.
• Am not adding an annotation to control eviction under node auto-scaling, that is system specific and I'm aware of no auto-scaling, and certainly of no relevant annotation to guide the draining behavior.

Pepr requires that I setup a policy exception to enable cgroups.

[JoeHCQ1]

Links for future reference:

• https://docs.gitlab.com/ee/administration/gitaly/kubernetes.html#constrain-git-processes-resource-usage (linked before)
• cgroups specific docs for gitaly: https://docs.gitlab.com/charts/charts/gitlab/gitaly/#cgroups
• The cgroups dockerfile: https://gitlab.com/gitlab-org/build/CNG/-/tree/master/gitaly-init-cgroups?ref_type=heads
• The registry which does include oldish images: https://gitlab.com/gitlab-org/build/CNG/container_registry/6395062
• Useful to get the right number of Bytes: https://www.convertunits.com/from/GiB/to/bytes
• The gitaly helm chart: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/gitaly/values.yaml?ref_type=heads
• The gitaly stateful set: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/gitaly/templates/_statefulset_spec.yaml?ref_type=heads
• Example of exception: https://github.com/defenseunicorns/uds-core/blob/0c19633313c14a8983553e3e3916a8d86d8440f0/src/vector/chart/templates/uds-exemption.yaml#L15
• Pepr reference on exception: https://github.com/defenseunicorns/uds-core/blob/0c19633313c14a8983553e3e3916a8d86d8440f0/docs/reference/configuration/pepr-policies.md
• RKE2 image we might want to swap in for the nodes currently in use. A similar error to the one we're getting happened before the fix linked here: https://github.com/defenseunicorns/uds-rke2-image-builder/blob/fa1128387d87c991e1ed501e5b5ef0edc8951df6/packer/scripts/os-prep.sh#L15
• Cool reading on cgroups themselves - who knew linux had such great admin docs: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
• Here is the code Gitaly is running to modify the cgroups.

[JoeHCQ1]

Internal slack convo on security risks with the permissions the gitaly init container is requiring: https://defense-unicorns.slack.com/archives/C06QJAUHWFN/p1730316451522489

The current error: Error: changing permissions for Gitaly pod 5d02861e-0cce-48d1-8789-a7f0354e6c6c cgroups: chown cgroup path "": chown /run/gitaly/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d02861e_0cce_48d1_8789_a7f0354e6c6c.slice: permission denied

[JoeHCQ1]

Adding the DisallowPrivileged exception didn't fix it either. I might need to get exceptions added for: Restrict hostPath Volume Mountable Paths - note there are two policies by that name back to back.