datalad / git-annex

A non-official clone of git-annex established for DataLad purposes. No PRs will be merged, but could be used to test perspective git-annex patches. Official git-annex repository: https://git.kitenet.net/index.cgi/git-annex.git/
14 stars 3 forks source link

Recommend to reduce gc.reflogExpire in hopes to reduce footprint on storage #158

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago

I am yet to see if it has effect "automagically" or we should force some "gc" invocation

We got to GBs of used storage and triggered quota limits on one HPC

yarikoptic commented 1 year ago

FWIW here is somewhat of a demo

$> git gc --prune=now
Counting objects: 2236, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (795/795), done.
Writing objects: 100% (2236/2236), done.
Total 2236 (delta 1421), reused 2236 (delta 1421)
(git)/net/vast-storage/scratch/vast/gablab/yhalchen/jobs:[result-openmind7-1037]
#43 !1034 [0].....................................:Tue Mar 07 16:48:16:.
yhalchen@openmind7:/om2/user/yhalchen/jobs
$> du -scm .
1204    .
1204    total
(git)/net/vast-storage/scratch/vast/gablab/yhalchen/jobs:[result-openmind7-1037]
#44 !1035 [0].....................................:Tue Mar 07 16:48:21:.
yhalchen@openmind7:/om2/user/yhalchen/jobs
$> git config gc.reflogExpire 10
(git)/net/vast-storage/scratch/vast/gablab/yhalchen/jobs:[result-openmind7-1037]
#45 !1036 [0].....................................:Tue Mar 07 16:48:29:.
yhalchen@openmind7:/om2/user/yhalchen/jobs
*$> git reflog expire
(git)/net/vast-storage/scratch/vast/gablab/yhalchen/jobs:[result-openmind7-1037]
#46 !1037 [0].....................................:Tue Mar 07 16:48:44:.
yhalchen@openmind7:/om2/user/yhalchen/jobs
$> git gc --prune=now
Counting objects: 2076, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (719/719), done.
Writing objects: 100% (2076/2076), done.
Total 2076 (delta 1339), reused 2071 (delta 1337)
(git)/net/vast-storage/scratch/vast/gablab/yhalchen/jobs:[result-openmind7-1037]
#47 !1038 [0].....................................:Tue Mar 07 16:48:49:.
yhalchen@openmind7:/om2/user/yhalchen/jobs
$> du -scm .
48      .
48      total
yarikoptic commented 1 year ago

we are staying at 48M there, so I guess it is working. Lets proceed