git-annex non-fast-forward issue on Gin

[adswa]

Pushing different datasets (e.g., studyforrest, machinelearning-books) from a Windows machine to Gin failed with a non-fast-forward error of the git-annex branch (see also #346).

Errors look like this:

-> Failed: Dataset('C:\\Users\\adina\\Desktop\\ml-books').push(to='gin4')

Traceback (most recent call last):
  File "C:\Users\adina\repos\datalad-gooey\datalad_gooey\dataladcmd_exec.py", line 164, in _cmdexec_thread
    for res in cmd(**cmdkwargs):
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\interface\utils.py", line 357, in generator_func
    for r in _process_results(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\interface\utils.py", line 544, in _process_results
    for res in results:
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\core\distributed\push.py", line 262, in __call__
    yield from _push(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad_next\patches\push_optimize.py", line 179, in _push
    yield from mod_push._push_refspecs(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\core\distributed\push.py", line 726, in _push_refspecs
    push_res = repo.push(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\support\gitrepo.py", line 1983, in push
    push_res.extend(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\support\gitrepo.py", line 2000, in push_
    yield from self._fetch_push_helper(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\support\gitrepo.py", line 2098, in _fetch_push_helper
    out = self._git_runner.run(
  File "C:\Users\adina\env\gooyey\lib\site-packages\datalad\runner\runner.py", line 211, in run
    raise CommandError(
datalad.runner.exception.CommandError: CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "push" "--progress" "--porcelain" "gin4" "master" "git-annex:git-annex"' failed with exitcode 1 under C:\Users\adina\Desktop\ml-books [out: 'To gin.g-node.org:/adswa/ml-books-only-ssh.git
*   refs/heads/master:refs/heads/master [new branch]
!   refs/heads/git-annex:refs/heads/git-annex   [rejected] (non-fast-forward)
Done'] [err: 'Delta compression using up to 16 threads
Total 422 (delta 198), reused 149 (delta 33), pack-reused 0
remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         181/198)        remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         remote: remote:                                                                                         error: failed to push some refs to 'gin.g-node.org:/adswa/ml-books-only-ssh.git'
hint: Updates were rejected because a pushed branch tip is behind its remote
hint: counterpart. Check out this branch and integrate the remote changes
hint: (e.g. 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.']

I have tried this with various access protocols (https-ssh and ssh), private and non-private repos, and with both the --data auto, anything and auto-if-wanted configuration.

The local git annex branch carries a range of commits I haven't seen before, of which @mih says they happen when one intentionally severs the git-annex branch.

(gooyey) C:\Users\adina\Desktop\ml-books2>git log git-annex
commit 4e226892a69de8989b56cef5f41c49f138aee09e (git-annex)
Author: Adina Wagner <adina.wagner@t-online.de>
Date:   Fri Oct 14 09:22:57 2022 +0200

    continuing transition ["forget git history"]

commit 38be5a7d07b019e2a7e42c8dff0734926c276f7d
Author: Adina Wagner <adina.wagner@t-online.de>
Date:   Fri Oct 14 09:17:56 2022 +0200

    update

commit 72cd967f9648209aab5c55aebf5b60f1aea41099 (origin/git-annex)
Author: Adina Wagner <adina.wagner@t-online.de>
Date:   Tue Apr 19 13:29:07 2022 +0200

    update

The remote git-annex branch, interestingly, has a "Gogs" committer id: https://gin.g-node.org/adswa/ml-books-only-ssh/src/git-annex

My local git-annex version is 10.20220928-g82dab0749. I do not see a similar problem pushing to a freshly created github sibling.

[adswa]

I have reproduced this on Linux with git-annex version 10.20220504, in the gooey and the command line. I suspect this is something on Gin's end. Pure git commands confirm that the histories of the local and remote git-annex branch are unrelated:

(gooey) adina@muninn in /tmp/machinelearning-books on git:git-annex
❱ git pull gin git-annex
From https://gin.g-node.org/adswa/mlbooksmoretests
 * branch            git-annex  -> FETCH_HEAD
fatal: refusing to merge unrelated histories

The problem persists in a clone of the repository:

(gooey) adina@muninn in /tmp/mlbooksmoretests on git:master
❱ git-annex whereis A.Shashua-Introduction_to_Machine_Learning.pdf          1 !
whereis A.Shashua-Introduction_to_Machine_Learning.pdf (0 copies) failed
whereis: 1 failed

[mih]

Is git annex merge able to recover from this situation? I would not expected a standard git pull to do it. I assume that GIN is (for some reason) running git annex forget, and git annex merge should be able to work with the outcome.

If that is the case, we could look into automating this kind of recovery.

[adswa]

I haven't tried but I will. What we discovered is that a git push gin git-annex --force fixes the branch on gins end. Both merge and push wouldn't be possible for someone cloning someone's repo

[adswa]

This issue has nothing to do with datalad-gooey, but is a yet unresolved issue of Gin, that luckily hasn't reoccurred so far. I'm closing this, therefore.