Allow configure git autoPackLimit

[jsolana]

Summary

Introduce a configurable autoPackLimit for Git in repo servers to enabling automatic garbage collection and reducing resource consumption.

Motivation

Git uses an object storage system to efficiently manage the contents of a repository. These objects represent key pieces of information, such as commits, trees (directory structures), blobs (file contents), and tags. Over time, as a repository grows, these objects accumulate as individual files, which can impact performance and storage usage.

Git packfiles are compressed files that group many loose objects into a single file, optimizing storage and improving access speed

In monorepos, the number of objects can grow significantly due to frequent changes to manifests, branches, or large sets of commits. Without proper management, Git accumulates these objects as loose files. Over time, this increases the load on the repository server, as more disk space and resources are consumed to handle them.

In environments where ephemeral storage is configured (common in Kubernetes), these objects can exhaust the available storage, causing pods to fail. Even after container restarts, the storage is not always released because ephemeral storage is tied to the pod lifecycle. This can lead to pods remaining in an Error state, locking up resources and requiring manual intervention.

In the picture, ephemeral storage usage accumulated for 40 repo-servers (decreasing after a kubectl delete pod command)

Proposal

Limiting the number of loose objects via a configurable autoPackLimit would trigger Git's internal garbage collection and repack -d operations. This would automatically pack loose objects and free up disk space, preventing the repo server from consuming excessive resources.

Allow configure repo-server to specifies the limit for automatic git repacking and apply this git configuration.

With this change we auto trigger the git repack -d after N pack files being generated avoiding increase the size of storage required for repo-server in time

References

https://git-scm.com/docs/git-gc#Documentation/git-gc.txt-gcautoPackLimit https://cloud-native.slack.com/archives/C01TSERG0KZ/p1713892767011309

[andrii-korotkov-verkada]

Hm, this size increase doesn't seem normal for a deploy repository. Can you check if there are git fetch timeouts which can cause runaway storage usage as described here https://github.com/argoproj/argo-cd/issues/18831?

[jsolana]

Hi @andrii-korotkov-verkada, the problem is related with the amount of pack files Git managed. I can't see timeouts. Just to know, are you able to reproduce the increase of pack files caused by timeouts too?

[andrii-korotkov-verkada]

I'm not sure it was related to pack files specifically :(