"Remote origin not usable by git-annex"

[jbwexler]

When I try to install this dataset, I get the following errors:

(main) login4.frontera(1136)$ datalad install https://github.com/ReproNim/containers.git [INFO ] Remote origin not usable by git-annex; setting annex-ignore [INFO ] https://github.com/ReproNim/containers.git/config download failed: Not Found install(ok): /scratch1/03201/jbwexler/openneuro_derivatives/containers (dataset)

It still seems to work properly and allows me to datalad get files. However, things periodically break and I have to delete the entire dataset and reinstall. For example, I recently updated the dataset (datalad update --merge) and tried to get the newest fmriprep image, but ran into the following error:

(main) login4.frontera(1116)$ datalad get bids-fmriprep--20.2.6.sing get(error): images/bids/bids-fmriprep--20.2.6.sing (file) [not available; (Note that these git remotes have annex-ignore set: origin)]

[yarikoptic]

(main) login4.frontera(1136)$ datalad install https://github.com/ReproNim/containers.git [INFO ] Remote origin not usable by git-annex; setting annex-ignore

that is ok. git-annex just says that indeed github remote from which you install/clone has no git-annex, i.e. does not contain any annexed content (since github does not natively support git-annex).

It still seems to work properly and allows me to datalad get files

good. That is so because after you install, you get actually two git remotes (well -- there is also a datalad but it is a special git-annex remote, ignore it for now):

lena:/tmp
$>  datalad install https://github.com/ReproNim/containers.git
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore                                                                                                                                                
[INFO   ] https://github.com/ReproNim/containers.git/config download failed: Not Found                                                                                                                               
install(ok): /tmp/containers (dataset)
(dev3) 1 26119 [1].....................................:Fri 26 Nov 2021 01:44:21 PM EST:.
lena:/tmp
$> git -C containers remote
datalad
datasets.datalad.org
origin
(dev3) 1 26119 [1].....................................:Fri 26 Nov 2021 01:44:25 PM EST:.
lena:/tmp
$> git -C containers remote show datasets.datalad.org
* remote datasets.datalad.org
  Fetch URL: http://datasets.datalad.org/repronim/containers/.git
  Push  URL: falkor.datalad.org:/srv/datasets.datalad.org/www/repronim/containers/.git
  HEAD branch: master
  Remote branches:
    git-annex     tracked
    master        tracked
    synced/master tracked
  Local refs configured for 'git push':
    git-annex pushes to git-annex (fast-forwardable)
    master    pushes to master    (up to date)

so it is that datasets.datalad.org which points to http://datasets.datalad.org/?dir=/repronim/containers is where it gets those annexed files.

However, things periodically break and I have to delete the entire dataset and reinstall. For example, I recently updated the dataset (datalad update --merge) and tried to get the newest fmriprep image, but ran into the following error:

(main) login4.frontera(1116)$ datalad get bids-fmriprep--20.2.6.sing get(error): images/bids/bids-fmriprep--20.2.6.sing (file) [not available; (Note that these git remotes have annex-ignore set: origin)]

"need more data" -- what remotes you have available in that clone? how you ended up without having datasets.datalad.org one -- may be it got marked as ignore for git-annex? (check within .git/config)

[jbwexler]

Thanks for the quick reply. I unfortunately deleted my previous version of the dataset before reinstalling so I can't check .git/config. I'll close the issue for now and reopen if I have problems again.

[jbwexler]

So here's an issue I've had in the past with this dataset (perhaps just due to my bad understanding of datalad): I wanted to add an image of a pre-release of mriqc to my clone of this dataset for testing purposes. So I copied the image bids-mriqc--21.0.0rc2.sing to containers/images/bids and tried to datalad save. This resulted in the following errors:

(main) login3.frontera(1034)$ datalad save [WARNING] Received an exception CommandError(CommandError: 'git -c diff.ignoreSubmodules=none annex add --json --json-error-messages -c annex.dotfiles=true -- images/bids/bids-mriqc--21.0.0rc2.sing' failed with exitcode 1 under /scratch1/03201/jbwexler/openneuro_derivatives/containers [info keys: stdout_json] [err: 'error: invalid object 100644 4e82683af4cb5cf7bd77d3cd9c35ca453d41fc7d for '415/c16/MD5E-s181080095--f66b6967abee5173c47cac03fd0697d6.6.sing.log' fatal: git-write-tree: error building trees git-annex: failed to read sha from git write-tree CallStack (from HasCallStack): error, called at ./Git/Sha.hs:23:15 in main:Git.Sha']). | Canceling not-yet running jobs and waiting for completion of running. | You can force earlier forceful exit by Ctrl-C. [INFO ] Canceled 0 out of 0 jobs. 0 left running. Total: 0.00 datasets [00:16, ? datasets/s]CommandError: 'git -c diff.ignoreSubmodules=none annex add --json --json-error-messages -c annex.dotfiles=true -- images/bids/bids-mriqc--21.0.0rc2.sing' failed with exitcode 1 under /scratch1/03201/jbwexler/openneuro_derivatives/containers [info keys: stdout_json] error: invalid object 100644 4e82683af4cb5cf7bd77d3cd9c35ca453d41fc7d for '415/c16/MD5E-s181080095--f66b6967abee5173c47cac03fd0697d6.6.sing.log' fatal: git-write-tree: error building trees git-annex: failed to read sha from git write-tree CallStack (from HasCallStack): error, called at ./Git/Sha.hs:23:15 in main:Git.Sha

[yarikoptic]

oh -- you won some kind of a prize -- I never saw "git-write-tree: error building trees" although it was mentioned also in https://github.com/datalad/datalad/issues/4388 but there preceded with a clear permissions error, for which permissions were fixed and things started to work. The "fatal: git-write-tree: error building trees" is truly new to me (https://github.com/search?q=org%3Adatalad+fatal%3A+git-write-tree%3A+error+building+trees). googling suggests a series of git reset --hard etc. Was it in a clean (git status output) state before you added that file?

also knowing datalad wtf could be helpful (how recent git/datalad)

[jbwexler]

Thanks, I feel very proud of this accomplishment. I'm pretty sure it was in a clean status before adding that file, and I've tried git reset (hard and mixed) but still get the same issue when I try to add the file. This is about the fourth time I've reinstalled the dataset due to this issue. But I have a theory that it's a TACC-related (or me-related) issue. This dataset is stored on $SCRATCH which gives TACC the right to delete things that haven't been used or viewed for more than ten days. And I do notice some missing files. Maybe the type of operation they use to delete files doesn't get registered by git and thus appears to be clean until you actually try to commit? I'm checking with them now to see if they have a record of deleting files from this directory.

[jbwexler]

Here's `datalad wtf` just in case:

> # WTF > ## configuration > ## credentials > - keyring: > - active_backends: > - PlaintextKeyring with no encyption v.1.0 at /home1/03201/jbwexler/.local/share/python_keyring/keyring_pass.cfg > - config_file: /home1/03201/jbwexler/.config/python_keyring/keyringrc.cfg > - data_root: /home1/03201/jbwexler/.local/share/python_keyring > ## datalad > - full_version: 0.15.2 > - version: 0.15.2 > ## dataset > - branches: > - git-annex@832a449 > - master@300307a > - id: b02e63c2-62c1-11e9-82b0-52540040489c > - metadata: > - path: /scratch1/03201/jbwexler/openneuro_derivatives/containers > - repo: AnnexRepo > ## dependencies > - annexremote: 1.5.0 > - appdirs: 1.4.4 > - boto: 2.49.0 > - cmd:annex: 8.20210903-ga4d179c > - cmd:bundled-git: UNKNOWN > - cmd:git: 2.33.0 > - cmd:system-git: 2.33.0 > - cmd:system-ssh: 8.7p1 > - humanize: 3.11.0 > - iso8601: 0.1.16 > - keyring: 23.2.1 > - keyrings.alt: 4.1.0 > - msgpack: 1.0.2 > - requests: 2.26.0 > - wrapt: 1.12.1 > ## environment > - LANG: en_US.UTF-8 > - PATH: /work2/01329/poldrack/software/fsl/fsl-6.0.4/bin:/work2/03201/jbwexler/frontera/Sea/bin:/work/03201/jbwexler/frontera/anaconda3/envs/main/bin:/work/03201/jbwexler/frontera/anaconda3/condabin:/opt/apps/xalt/xalt/bin:/work/01329/poldrack/software/nodejs/node-v17.1.0-linux-x64/bin:/work/01329/poldrack/tacc-software/launch:/opt/apps/tacc-singularity/3.7.2/bin:/opt/apps/hwloc/1.11.12/bin:/opt/apps/pmix/3.1.4/bin:/opt/apps/cmake/3.20.3/bin:/opt/apps/intel19/python3/3.7.0/bin:/opt/apps/autotools/1.2/bin:/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin:/opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64:/opt/apps/gcc/8.3.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/opt/ddn/ime/bin:. > - PYTHONPATH: /opt/apps/intel19/impi19_0/python3/3.7.0/lib/python3.7/site-packages > ## extensions > - container: > - description: Containerized environments > - entrypoints: > - datalad_container.containers_add.ContainersAdd: > - class: ContainersAdd > - load_error: None > - module: datalad_container.containers_add > - names: > - containers-add > - containers_add > - datalad_container.containers_list.ContainersList: > - class: ContainersList > - load_error: None > - module: datalad_container.containers_list > - names: > - containers-list > - containers_list > - datalad_container.containers_remove.ContainersRemove: > - class: ContainersRemove > - load_error: None > - module: datalad_container.containers_remove > - names: > - containers-remove > - containers_remove > - datalad_container.containers_run.ContainersRun: > - class: ContainersRun > - load_error: None > - module: datalad_container.containers_run > - names: > - containers-run > - containers_run > - load_error: None > - module: datalad_container > - version: 1.1.5 > ## git-annex > - build flags: > - Assistant > - Webapp > - Pairing > - Inotify > - DBus > - DesktopNotify > - TorrentParser > - MagicMime > - Feeds > - Testsuite > - S3 > - WebDAV > - dependency versions: > - aws-0.22 > - bloomfilter-2.0.1.0 > - cryptonite-0.26 > - DAV-1.3.4 > - feed-1.3.0.1 > - ghc-8.8.4 > - http-client-0.6.4.1 > - persistent-sqlite-2.10.6.2 > - torrent-10000.1.1 > - uuid-1.3.13 > - yesod-1.6.1.0 > - key/value backends: > - SHA256E > - SHA256 > - SHA512E > - SHA512 > - SHA224E > - SHA224 > - SHA384E > - SHA384 > - SHA3_256E > - SHA3_256 > - SHA3_512E > - SHA3_512 > - SHA3_224E > - SHA3_224 > - SHA3_384E > - SHA3_384 > - SKEIN256E > - SKEIN256 > - SKEIN512E > - SKEIN512 > - BLAKE2B256E > - BLAKE2B256 > - BLAKE2B512E > - BLAKE2B512 > - BLAKE2B160E > - BLAKE2B160 > - BLAKE2B224E > - BLAKE2B224 > - BLAKE2B384E > - BLAKE2B384 > - BLAKE2BP512E > - BLAKE2BP512 > - BLAKE2S256E > - BLAKE2S256 > - BLAKE2S160E > - BLAKE2S160 > - BLAKE2S224E > - BLAKE2S224 > - BLAKE2SP256E > - BLAKE2SP256 > - BLAKE2SP224E > - BLAKE2SP224 > - SHA1E > - SHA1 > - MD5E > - MD5 > - WORM > - URL > - X* > - local repository version: 8 > - operating system: linux x86_64 > - remote types: > - git > - gcrypt > - p2p > - S3 > - bup > - directory > - rsync > - web > - bittorrent > - webdav > - adb > - tahoe > - glacier > - ddar > - git-lfs > - httpalso > - borg > - hook > - external > - supported repository versions: > - 8 > - upgrade supported from repository versions: > - 0 > - 1 > - 2 > - 3 > - 4 > - 5 > - 6 > - 7 > - version: 8.20210903-ga4d179c > ## location > - path: /scratch1/03201/jbwexler/openneuro_derivatives/containers > - type: dataset > ## metadata_extractors > - annex (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: None > - module: datalad.metadata.extractors.annex > - version: None > - audio (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: ModuleNotFoundError(No module named 'mutagen') > - module: datalad.metadata.extractors.audio > - datacite (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: None > - module: datalad.metadata.extractors.datacite > - version: None > - datalad_core (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: None > - module: datalad.metadata.extractors.datalad_core > - version: None > - datalad_rfc822 (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: None > - module: datalad.metadata.extractors.datalad_rfc822 > - version: None > - exif (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: ModuleNotFoundError(No module named 'exifread') > - module: datalad.metadata.extractors.exif > - frictionless_datapackage (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: None > - module: datalad.metadata.extractors.frictionless_datapackage > - version: None > - image (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: ModuleNotFoundError(No module named 'PIL') > - module: datalad.metadata.extractors.image > - xmp (datalad 0.15.2): > - distribution: datalad 0.15.2 > - load_error: ModuleNotFoundError(No module named 'libxmp') > - module: datalad.metadata.extractors.xmp > ## metadata_indexers > ## python > - implementation: CPython > - version: 3.9.7 > ## system > - distribution: centos/7/Core > - encoding: > - default: utf-8 > - filesystem: utf-8 > - locale.prefered: UTF-8 > - max_path_length: 325 > - name: Linux > - release: 3.10.0-1160.45.1.el7.x86_64 > - type: posix > - version: #1 SMP Wed Oct 13 17:20:51 UTC 2021

[yarikoptic]

great, thanks for the details. I will try to recover details on how to login to tacc and try it out myself . Is git-annex and git coming from anaconda environment you have? (might be worth to try newer versions... didn't check though if newer git is available there)

[yarikoptic]

ok. logged in to stampede2. Realized that it is the "lovely" lustre filesystem (some old annex issues lustre1 lustre2 just for the reference), filed https://github.com/datalad/datalad/issues/6294 . Created fresh conda env via conda create -n datalad-conda-forge -c conda-forge -y datalad, cd $SCRATCH, and failed (I hate failing! ;)) to reproduce

(base) login2.stampede2(679)$ conda activate datalad-conda-forge
(datalad-conda-forge) login2.stampede2(680)$ pwd
/scratch/03372/yoh
(datalad-conda-forge) login2.stampede2(681)$ datalad clone ///repronim/containers
install(ok): /scratch/03372/yoh/containers (dataset)                                                      
(datalad-conda-forge) login2.stampede2(682)$ cd containers/images/bids/                                   
(datalad-conda-forge) login2.stampede2(683)$ cat /dev/random > random-bits.simg
^C
(datalad-conda-forge) login2.stampede2(684)$ ls -ld random-bits.simg
-rw------- 1 yoh G-803075 10484 Dec  9 11:53 random-bits.simg
(datalad-conda-forge) login2.stampede2(685)$ file random-bits.simg
random-bits.simg: data
(datalad-conda-forge) login2.stampede2(686)$ datalad save -m "Saving random file"
add(ok): images/bids/random-bits.simg (file)                                                              
save(ok): . (dataset)                                                                                     
action summary:                                                                                           
  add (ok: 1)
  save (ok: 1)
(datalad-conda-forge) login2.stampede2(690)$ datalad wtf --flavor=short
# WTF
- datalad: version=0.15.3
- dependencies: cmd:git=2.34.0 cmd:annex=8.20211118-g23ee48898 cmd:bundled-git=2.34.0 cmd:system-git=2.24.1 cmd:system-ssh=7.5p1-hpn14v13NMOD cmd:7z=16.02 annexremote=1.5.0 appdirs=1.4.4 boto=2.49.0 exifread=2.3.2 humanize=3.13.1 iso8601=1.0.2 keyring=23.4.0 keyrings.alt=UNKNOWN msgpack=1.0.3 mutagen=1.45.1 requests=2.26.0 wrapt=1.13.3

so, @jbwexler please first try creating similar conda env with recent versions, and see if reproduces.

FTR, my git config is also pretty plain:

(datalad-conda-forge) login2.stampede2(692)$ cat ~/.gitconfig 
[user]
        name = Yaroslav Halchenko
        email = debian@onerussian.com

setting annex.pidlock did not effect

```shell (datalad-conda-forge) login2.stampede2(694)$ git config annex.pidlock true (datalad-conda-forge) login2.stampede2(695)$ git config annex.pidlock true (datalad-conda-forge) login2.stampede2(696)$ cat /dev/random > random-bits2.simg ^C (datalad-conda-forge) login2.stampede2(697)$ datalad save -m "Saving random file #2" add(ok): images/bids/random-bits2.simg (file) save(ok): . (dataset) action summary: add (ok: 1) save (ok: 1) ```

[yarikoptic]

oh -- searching git-annex website brings up many reports which might relate!!! (didn't investigate yet) . Seems to relate to corrupt git repos (did you run out of free 2.7P there? are there quotas? may be path too long (interestingly, your WTF says max_path_length: 325 and mine 297)?

In some @joeyh recommended to run git annex repair to fix them up

[yarikoptic]

also may be try to reproduce with plain git/git-annex commands (git clone ...; cd ...; git annex init; cp whatever somewherehere; git annex add somewherehere; git commit). If fails -- cool! if not -- rinse/repeat with datalad but run as datalad -l debug mode, share the outputs etc

[jbwexler]

Sorry for the delay, I was waiting to hear back from TACC (still no word) as to whether they had purged some files from that directory. But I just encountered this error on a fresh copy of the dataset after trying to datalad save. So I tried git annex repair which produced the following output:

(main) login2.frontera(1033)$ git annex repair repair Running git fsck ... Fsck found no problems. Checking for broken branches. Found problems, attempting repair. Unpacking all pack files. Unpacking objects: 100% (6214/6214), 683.55 KiB | 1021.00 KiB/s, done. No missing objects found, but the index file is corrupt! To force a recovery to a usable state, retry with the --force parameter. Had to delete the .git/annex/index file as it was corrupt. No data was lost. failed repair: 1 failed

But then I tried datalad save again and it worked!!

[yarikoptic]

Oh well, even though some human beasts are God like in my perception, no software is made by Gods, and every software has bugs. Operation on "interesting" setups is particularly entertaining ;-) I will close for now, but feel welcome to reopen if anything else could be done