Disallow gc in background, do explicitly after `commit`

[yarikoptic]

Might be a reason for #309 but it is 50/50 chance ;)

000347 was found to be dirty, I looked through recent emails to find

2022-11-08T01:54:32-0500 [WARNING ] backups2datalad: Retrying GET request to /dandisets/000156/versions/draft/ in 0.981522 seconds as it raised ReadTimeout:
fatal: Unable to create '/mnt/backup/dandi/dandisets/000347/.git/objects/info/commit-graph.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
2022-11-08T01:58:07-0500 [WARNING ] backups2datalad: `git gc` in /mnt/backup/dandi/dandisets/000347 exited with code 128

which gives the idea on what could potentially go wrong -- many git commands can trigger gc to run in background. In datalad we used to add ['-c', 'receive.autogc=0', '-c', 'gc.auto=0'] to git calls, but then stopped since https://github.com/datalad/datalad/pull/3458 and it seems to work ok, besides one case on appveyor

I suspect that somehow/somewhere we keep running collision into this gc process potentially running in background. So let's

• add those two settings to each git call
• invoke git gc after git commit we do (instead of datalad save)

this way we would avoid collision and keep repos lean.

[jwodder]

@yarikoptic Should these options also be supplied when git-annex is called? If so, should they come between "git" and "annex", or can they be after "git-annex"?

[jwodder]

@yarikoptic

invoke git gc after git commit we do (instead of datalad save)

Do you mean that you just want git gc run after each git commit, or do you also want each occurrence of datalad save changed to git commit+git gc?

[yarikoptic]

@yarikoptic Should these options also be supplied when git-annex is called?

yes, any invocation of git including git annex.

If so, should they come between "git" and "annex", or can they be after "git-annex"?

AFAIK git OPTIONS annex is the way since otherwise AFAIK git would not see them in git-annex.

Do you mean that you just want git gc run after each git commit, or do you also want each occurrence of datalad save changed to git commit+git gc?

the former -- only add after each present git commit.

[jwodder]

@yarikoptic git OPTIONS annex doesn't seem to work. Such invocations fail with git-annex: user error (git ["config","--null","--list"] exited 128).

[yarikoptic]

Took me awhile to get what you are talking about -- please point to the failed CI run next time and/or provide more context such as :

2022-11-30T20:41:32.9277932Z ----------------------------- Captured stderr call -----------------------------
2022-11-30T20:41:32.9278342Z [INFO] Running procedure cfg_text2git 
2022-11-30T20:41:32.9278683Z [INFO] == Command start (output follows) ===== 
2022-11-30T20:41:32.9279175Z [INFO] == Command exit (modification check follows) ===== 
2022-11-30T20:41:32.9279532Z [INFO] Running procedure cfg_text2git 
2022-11-30T20:41:32.9279866Z [INFO] == Command start (output follows) ===== 
2022-11-30T20:41:32.9280228Z [INFO] == Command exit (modification check follows) ===== 
2022-11-30T20:41:32.9281498Z error: bogus format in GIT_CONFIG_PARAMETERS
2022-11-30T20:41:32.9281925Z fatal: unable to parse command-line config
2022-11-30T20:41:32.9282394Z git-annex: user error (git ["config","--null","--list"] exited 128)
2022-11-30T20:41:32.9282938Z ------------------------------ Captured log call -------------------------------

searching for GIT_CONFIG_PARAMETERS lead me to https://git-annex.branchable.com/forum/error__58___bogus_format_in_GIT__95__CONFIG__95__PARAMETERS/ which reminded that I did have similar problem 4 months ago.

on CI we have ubuntu jammy and git-annex 10.20220822-1~ndall+1 which is odd since http://neuro.debian.net/pkgs/git-annex.html#binary-pkg-git-annex says that it should be 10.20221003-2~nd22.04+1... tested in docker image neurodebian:nd22.04 -- indeed the case! and the screw up happened in binary package lacking -standalone part thus overloading regular stock git-annex build :-/

root@7b06f5cf14c4:/# apt-cache policy git-annex-standalone
git-annex-standalone:
  Installed: (none)
  Candidate: 10.20220822-1~ndall+1
  Version table:
     10.20220822-1~ndall+1 500
        500 http://neuro.debian.net/debian jammy/main amd64 Packages
root@7b06f5cf14c4:/# apt-cache policy git-annex 
git-annex:
  Installed: (none)
  Candidate: 10.20221003-2~nd22.04+1
  Version table:
     10.20221003-2~nd22.04+1 500
        500 http://neuro.debian.net/debian jammy/main amd64 Packages
     8.20210223-2ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

apparently I (and forgot, or may be it was @mih ) provided native build of git-annex without providing git-annex-standalone for one of the recent ones for jammy.

NB moral -- besides native builds we still should provide -standalone for fresh releases

So ideally we could have just switched to install git-annex instead of -standalone in the action, but we do it all via datalad-installer which for -m neurodebian hardcodes installation of git-annex-standalone. I guess could just "force" its reinstallation using apt install git-annex after datalad-installer call (which would still be useful to enable neurodebian).

alternative is to just tune up PATH to point to git-annex dir with its git: `export PATH=/usr/lib/git-annex.linux/:$PATH`

```shell root@7b06f5cf14c4:/tmp/11# git -c foo.bar=baz annex info error: bogus format in GIT_CONFIG_PARAMETERS fatal: unable to parse command-line config git-annex: user error (git ["config","--null","--list"] exited 128) root@7b06f5cf14c4:/tmp/11# export PATH=/usr/lib/git-annex.linux/:$PATH root@7b06f5cf14c4:/tmp/11# git -c foo.bar=baz annex info trusted repositories: 0 semitrusted repositories: 3 00000000-0000-0000-0000-000000000001 -- web 00000000-0000-0000-0000-000000000002 -- bittorrent 16a3f2cd-3a0f-4c69-bf00-1062dbef3621 -- root@7b06f5cf14c4:/tmp/11 [here] untrusted repositories: 0 transfers in progress: none available local disk space: 8.67 gigabytes (+1 megabyte reserved) local annex keys: 0 local annex size: 0 bytes annexed files in working tree: 0 size of annexed files in working tree: 0 bytes bloom filter size: 32 mebibytes (0% full) backend usage: ```

Choose the workaround you like.

[jwodder]

@yarikoptic OK, the tests are now back to only failing because of https://github.com/dandi/dandi-archive/issues/1365.