[BUG]: Maintenance job does not clean up git submodules

[ChristianStadelmann]

What happened?

Steps to reproduce:

1. Have a git repository with multiple submodules
2. Have a pipeline run on the git repo
3. Modify both the parent repo and the submodules, update the submodules in the parent repo
4. Run the pipeline
5. Repeat steps 3 and 4 a few times
6. Have a maintenance job enabled on the pool
7. Wait until maintenance job runs at least once

What happens:

With steps 3 and 4, the .git folder grows in size. It will accumulate lots of .pack files in .git/modules/[submodule name]/objects/pack/.

In my case, even though the maintenance job was configured to run once a week, some submodules had old .pack files lying around which were some months old. Over time with many changes on the repo and large submodules, the pipeline will consume a lot of disk space. (In my case, one of the submodules has a .pack file of a few GB, easily accumulating to tens of GB within a few days.)

According to the maintenance logs (see attached logs), the maintenance job only runs

git repack -adfl
git prune -v

What should happen:

The maintenance job should recursively run on submodules too, otherwise they will never be cleaned (up as long as the pipeline is executed often enough not to be deleted by maintenance job criteria).

In technical terms, the maintenance job should run git commands prefixed with git submodule foreach --recursive for both git repack and git prune.

Versions

Agent version: 3.232.3 OS version: Windows 10 22H2 (10.0.19045)

Environment type (Please select at least one enviroment where you face this issue)

• [X] Self-Hosted
• [ ] Microsoft Hosted
• [ ] VMSS Pool
• [ ] Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version control system

Git

Relevant log output

2024-02-13T00:48:27.6762113Z Run maintenance operation on repository: D:\agent\_work\5\s
2024-02-13T00:48:27.6832338Z Prepending Path environment variable with directory containing 'git.exe'.
2024-02-13T00:48:27.6876237Z ##[command]git version
2024-02-13T00:48:28.3060912Z git version 2.39.1.windows.1
2024-02-13T00:48:28.3075273Z ##[command]git lfs version
2024-02-13T00:48:29.2087613Z git-lfs/3.3.0 (GitHub; windows amd64; go 1.19.3; git 77deabdf)
2024-02-13T00:48:29.2095873Z Repository status before executing 'git repack'
2024-02-13T00:48:29.2110827Z ##[command]git count-objects -v -H
2024-02-13T00:48:29.3159856Z count: 44
2024-02-13T00:48:29.3160060Z size: 26.47 KiB
2024-02-13T00:48:29.3160183Z in-pack: 10705
2024-02-13T00:48:29.3160259Z packs: 1
2024-02-13T00:48:29.3160311Z size-pack: 61.73 MiB
2024-02-13T00:48:29.3160367Z prune-packable: 0
2024-02-13T00:48:29.3160428Z garbage: 0
2024-02-13T00:48:29.3160477Z size-garbage: 0 bytes
2024-02-13T00:48:29.3208330Z ##[command]git repack -adfl
2024-02-13T00:48:35.6534036Z ##[command]git prune -v
2024-02-13T00:48:35.7811570Z ##[command]git lfs version
2024-02-13T00:48:36.0039588Z git-lfs/3.3.0 (GitHub; windows amd64; go 1.19.3; git 77deabdf)
2024-02-13T00:48:36.0053260Z ##[command]git lfs prune
2024-02-13T00:48:37.0204706Z prune: 0 local objects, 1 retained, done.
2024-02-13T00:48:37.0320392Z Repository status after executing 'git repack'
2024-02-13T00:48:37.0321225Z ##[command]git count-objects -v -H
2024-02-13T00:48:37.1157216Z count: 0
2024-02-13T00:48:37.1157483Z size: 0 bytes
2024-02-13T00:48:37.1157560Z in-pack: 10749
2024-02-13T00:48:37.1157633Z packs: 1
2024-02-13T00:48:37.1157706Z size-pack: 61.75 MiB
2024-02-13T00:48:37.1157852Z prune-packable: 0
2024-02-13T00:48:37.1157967Z garbage: 0
2024-02-13T00:48:37.1158049Z size-garbage: 0 bytes
2024-02-13T00:48:37.1206867Z Total time for executing maintenance for repository 'D:\agent\_work\5\s': 7.9110806 seconds.

[aleksandrlevochkin]

Hi @ChristianStadelmann thank you for bringing this issue up. We're working on more prioritized issues at the moment but will get back to this one as soon as we can.

[github-actions[bot]]

This issue has had no activity in 180 days. Please comment if it is not actually stale