test_openfmri_pipeline1 errors with latest git-annex

kyleam commented 4 years ago

In a test run (from datalad/datalad#3648) with the latest git-annex, which auto-upgrades to v7 repos, test_openfmri.test_openfmri_pipeline1 errors:

https://travis-ci.org/datalad/datalad/jobs/588507087#L1238

======================================================================
ERROR: datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1
----------------------------------------------------------------------
Traceback (most recent call last):
[...]
 line 478, in test_openfmri_pipeline1
    out = run_pipeline(pipeline)
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 114, in run_pipeline
    output = list(xrun_pipeline(*args, **kwargs))
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 194, in xrun_pipeline
    for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps
    for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps
    for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps
    for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps
    for data_ in data_in_to_loop:
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 194, in xrun_pipeline
    for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps
    for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps
    for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output):
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps
    for data_ in data_in_to_loop:
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/nodes/annex.py", line 1174, in _remove_other_versions
    current_version = versions_db.version
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/dbs/versions.py", line 67, in version
    return self._db['version']
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/dbs/base.py", line 64, in _db
    self._assure_loaded()
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/dbs/base.py", line 56, in _assure_loaded
    self.load()
  File "/home/travis/virtualenv/python3.5.6/lib/python3.5/site-packages/datalad_crawler/dbs/base.py", line 69, in load
    with open(self._filepath) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/datalad_temp_test_openfmri_pipeline1daibzfc8/.datalad/crawl/versions/incoming.json'

yarikoptic commented 4 years ago

grr... failed to reproduce

```shell (git)hopa:~datalad/datalad-crawler[enh-github]git $> python -m nose -s -v datalad_crawler/pipelines/tests/test_openfmri.py:test_openfmri_pipeline1 datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ... Connecting to bucket: openneuro Bucket info: Versioning: S3ResponseError: 403 Forbidden Website: S3ResponseError: 403 Forbidden ACL: S3ResponseError: 403 Forbidden ERROR: No output was provided for prefix 'ds666' Connecting to bucket: openneuro Bucket info: Versioning: S3ResponseError: 403 Forbidden Website: S3ResponseError: 403 Forbidden ACL: S3ResponseError: 403 Forbidden ERROR: No output was provided for prefix 'ds666' Connecting to bucket: openneuro Bucket info: Versioning: S3ResponseError: 403 Forbidden Website: S3ResponseError: 403 Forbidden ACL: S3ResponseError: 403 Forbidden ERROR: No output was provided for prefix 'ds666' ok Versions: appdirs=1.4.3 boto=2.44.0 cmd:annex=7.20190912-1~ndall+1 cmd:bundled-git=2.20.1 cmd:git=2.20.1 cmd:system-git=2.23.0 cmd:system-ssh=7.9p1 exifread=2.1.2 git=2.1.11 gitdb=2.0.5 humanize=0.5.1 iso8601=0.1.11 keyring=17.1.1 keyrings.alt=3.1.1 msgpack=0.5.6 mutagen=1.40.0 patoolib=1.12 requests=2.21.0 scrapy=1.6.0 six=1.12.0 tqdm=4.28.1 wrapt=1.10.11 Obscure filename: str=b' "\';a&b&c\xce\x94\xd0\x99\xd7\xa7\xd9\x85\xe0\xb9\x97\xe3\x81\x82 `| ' repr=' "\';a&b&cΔЙקم๗あ `| ' Encodings: default='utf-8' filesystem='utf-8' locale.prefered='UTF-8' Environment: LANG='en_US.utf8' PATH='/home/yoh/proj/datalad/datalad-crawler/venvs/dev3/bin:/home/yoh/gocode/bin:/home/yoh/gocode/bin:/home/yoh/bin:/home/yoh/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/sbin:/usr/sbin:/usr/local/sbin' GIT_PAGER='less --no-init --quit-if-one-screen' GIT_PYTHON_GIT_EXECUTABLE='/usr/lib/git-annex.linux/git' ---------------------------------------------------------------------- Ran 1 test in 66.338s OK python -m nose -s -v 39.68s user 17.31s system 82% cpu 1:08.71 total (dev3) 1 28778.....................................:Tue 24 Sep 2019 09:40:06 AM EDT:. (git)hopa:~datalad/datalad-crawler[enh-github]git $> welp datalad PATH : /home/yoh/proj/datalad/datalad-master/datalad/__init__.py SRC PATH : /home/yoh/proj/datalad/datalad-master/datalad/__init__.py VERSION : 0.11.6 __version__: '0.11.6.dev114' ```

kyleam commented 4 years ago

Thank for trying it locally. I can trigger the issue locally with the latest git-annex (7.20190912), crawler at 0.4.1, and datalad 0.11.x (60542a0da).

failure 0

``` % python -m nose -vs -x datalad_crawler/pipelines/tests/test_openfmri.py:test_openfmri_pipeline1 datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ... Connecting to bucket: openneuro Connecting to bucket: openneuro ERROR Versions: appdirs=1.4.3 boto=2.49.0 cmd:annex=7.20190912 cmd:bundled-git=UNKNOWN cmd:git=2.23.0 cmd:system-git=2.23.0 cmd:system-ssh=7.9p1 exifread=2.1.2 git=3.0.2 gitdb=2.0.5 humanize=0.5.1 iso8601=0.1.12 keyring=19.1.0 keyrings.alt=3.1.1 msgpack=0.6.1 mutagen=1.42.0 patoolib=1.12 requests=2.22.0 scrapy=1.7.3 six=1.12.0 tqdm=4.35.0 wrapt=1.11.2 Obscure filename: str=b' "\';a&b&c\xce\x94\xd0\x99\xd7\xa7\xd9\x85\xe0\xb9\x97\xe3\x81\x82 `| ' repr=' "\';a&b&cΔЙקم๗あ `| ' Encodings: default='utf-8' filesystem='utf-8' locale.prefered='UTF-8' Environment: PATH='/home/kyle/src/python/venvs/datalad-crawler/bin:/home/kyle/src/python/venvs/datalad-crawler-py2//bin:/home/kyle/.guix-profile/bin:/home/kyle/.config/guix/current/bin:/home/kyle/.cabal/bin:/usr/lib/ccache/bin:/home/kyle/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games' PYTHONPATH='/home/kyle/.local/lib/python' GIT_EXEC_PATH='/home/kyle/.guix-profile/libexec/git-core' GIT_SSL_CAINFO='/home/kyle/.guix-profile/etc/ssl/certs/ca-certificates.crt' LANG='en_US.UTF-8' ====================================================================== ERROR: datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/kyle/src/python/venvs/datalad-crawler/lib/python3.7/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 442, in newfunc return t(*(arg + (d,)), **kw) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 554, in newfunc return tfunc(*(args + (path, url)), **kwargs) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipelines/tests/test_openfmri.py", line 478, in test_openfmri_pipeline1 out = run_pipeline(pipeline) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 114, in run_pipeline output = list(xrun_pipeline(*args, **kwargs)) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/kyle/src/python/datalad-crawler/datalad_crawler/nodes/annex.py", line 1174, in _remove_other_versions current_version = versions_db.version File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/versions.py", line 67, in version return self._db['version'] File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 64, in _db self._assure_loaded() File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 56, in _assure_loaded self.load() File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 69, in load with open(self._filepath) as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/datalad_temp_test_openfmri_pipeline1r373xom8/.datalad/crawl/versions/incoming.json' -------------------- >> begin captured logging << -------------------- [...] --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 1 test in 24.662s FAILED (errors=1) ```

I can also trigger it changing datalad to the v7-default (de6b6c308, before your latest force push).

failure 1

``` datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ... Connecting to bucket: openneuro Connecting to bucket: openneuro ERROR ====================================================================== ERROR: datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/kyle/src/python/venvs/datalad-crawler/lib/python3.7/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 442, in newfunc return t(*(arg + (d,)), **kw) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 554, in newfunc return tfunc(*(args + (path, url)), **kwargs) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/kyle/src/python/datalad/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipelines/tests/test_openfmri.py", line 478, in test_openfmri_pipeline1 out = run_pipeline(pipeline) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 114, in run_pipeline output = list(xrun_pipeline(*args, **kwargs)) File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/kyle/src/python/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/kyle/src/python/datalad-crawler/datalad_crawler/nodes/annex.py", line 1174, in _remove_other_versions current_version = versions_db.version File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/versions.py", line 67, in version return self._db['version'] File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 64, in _db self._assure_loaded() File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 56, in _assure_loaded self.load() File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 69, in load with open(self._filepath) as f: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/datalad_temp_test_openfmri_pipeline1hbbxy0ys/.datalad/crawl/versions/incoming.json' [...] Ran 1 test in 24.622s FAILED (errors=1) ```

If I downgrade git-annex to 7.20190819, the test passes. Since I can trigger this, I'll try to look into it more.

kyleam commented 4 years ago

A couple of notes:

I've been trying to trace the code in various places to see when the .datalad/crawler json files start being tracked by git-annex. The switch happens with the first run_pipeline() call in test_openfmri_pipeline1. During that call, there is a point where the files are in git rather than annexed, but I haven't been able to find what causes the switch.
As a brute-force approach, I tried converting all the custom git calls to use -c annex.largefiles=nothing and the test still had the same failure.

yarikoptic commented 4 years ago

difference from my environment: in your first output I see cmd:annex=7.20190912 cmd:bundled-git=UNKNOWN, so it is a system wide git used then I assume, which is 2.23.0, which is the same as mine systemwide...

but bundled version I have is 2.20.1!

```shell $> /usr/lib/git-annex.linux/git --version git version 2.20.1 ```

but that git-annex-standalone should be the same one as travis uses, and seems to be the case:

  - cmd:bundled-git: 2.20.1
  - cmd:annex: 7.20190912-1~ndall+1
  - cmd:git: 2.20.1

so don't know yet where the difference is... but could you try with git-annex-standalone package from neurodebian?

FTR: With unpatched 0.11.x version of datalad I think my failure matches yours:

``` $> python -m nose -s -v datalad_crawler/pipelines/tests/test_openfmri.py:test_openfmri_pipeline1 datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ... Connecting to bucket: openneuro Bucket info: Versioning: S3ResponseError: 403 Forbidden Website: S3ResponseError: 403 Forbidden ACL: S3ResponseError: 403 Forbidden ERROR: No output was provided for prefix 'ds666' Connecting to bucket: openneuro Bucket info: Versioning: S3ResponseError: 403 Forbidden Website: S3ResponseError: 403 Forbidden ACL: S3ResponseError: 403 Forbidden ERROR: No output was provided for prefix 'ds666' ERROR Versions: appdirs=1.4.3 boto=2.44.0 cmd:annex=7.20190912-1~ndall+1 cmd:bundled-git=2.20.1 cmd:git=2.20.1 cmd:system-git=2.23.0 cmd:system-ssh=7.9p1 exifread=2.1.2 git=2.1.11 gitdb=2.0.5 humanize=0.5.1 iso8601=0.1.11 keyring=17.1.1 keyrings.alt=3.1.1 msgpack=0.5.6 mutagen=1.40.0 patoolib=1.12 requests=2.21.0 scrapy=1.6.0 six=1.12.0 tqdm=4.28.1 wrapt=1.10.11 Obscure filename: str=b' "\';a&b&c\xce\x94\xd0\x99\xd7\xa7\xd9\x85\xe0\xb9\x97\xe3\x81\x82 `| ' repr=' "\';a&b&cΔЙקم๗あ `| ' Encodings: default='utf-8' filesystem='utf-8' locale.prefered='UTF-8' Environment: LANG='en_US.utf8' PATH='/home/yoh/proj/datalad/datalad-crawler/venvs/dev3/bin:/home/yoh/gocode/bin:/home/yoh/gocode/bin:/home/yoh/bin:/home/yoh/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/sbin:/usr/sbin:/usr/local/sbin' GIT_PAGER='less --no-init --quit-if-one-screen' GIT_PYTHON_GIT_EXECUTABLE='/usr/lib/git-annex.linux/git' ====================================================================== ERROR: datalad_crawler.pipelines.tests.test_openfmri.test_openfmri_pipeline1 ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py", line 442, in newfunc return t(*(arg + (d,)), **kw) File "/home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py", line 554, in newfunc return tfunc(*(args + (path, url)), **kwargs) File "/home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py", line 615, in newfunc return t(*(arg + (filename,)), **kw) File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipelines/tests/test_openfmri.py", line 478, in test_openfmri_pipeline1 out = run_pipeline(pipeline) File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 114, in run_pipeline output = list(xrun_pipeline(*args, **kwargs)) File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 194, in xrun_pipeline for idata_out, data_out in enumerate(xrun_pipeline_steps(pipeline, data_in, output=output_sub)): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 286, in xrun_pipeline_steps for data_out in xrun_pipeline_steps(pipeline_tail, data_, output=output): File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/pipeline.py", line 270, in xrun_pipeline_steps for data_ in data_in_to_loop: File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/nodes/annex.py", line 1174, in _remove_other_versions current_version = versions_db.version File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/dbs/versions.py", line 67, in version return self._db['version'] File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/dbs/base.py", line 64, in _db self._assure_loaded() File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/dbs/base.py", line 56, in _assure_loaded self.load() File "/home/yoh/proj/datalad/datalad-crawler/datalad_crawler/dbs/base.py", line 69, in load with open(self._filepath) as f: FileNotFoundError: [Errno 2] No such file or directory: '/home/yoh/.tmp/datalad_temp_test_openfmri_pipeline1cznj4m33/.datalad/crawl/versions/incoming.json' ```

kyleam commented 4 years ago

in your first output I see cmd:annex=7.20190912 cmd:bundled-git=UNKNOWN, so it is a system wide git used then I assume, which is 2.23.0, which is the same as mine systemwide

Correct. I typically don't have git-annex-standalone installed unless I need it to test something.

but could you try with git-annex-standalone package from neurodebian?

Sure. I see the same failure when I use git-annex-standalone with datalad at either 0.11.x or on the v7-default branch.

DataLad 0.11.6.dev114 WTF (git-annex, datalad, dependencies)

# WTF ## git-annex - version: 7.20190912-1~ndall+1 - build flags: - Assistant - Webapp - Pairing - S3 - WebDAV - Inotify - DBus - DesktopNotify - TorrentParser - MagicMime - Feeds - Testsuite - dependency versions: - aws-0.20 - bloomfilter-2.0.1.0 - cryptonite-0.25 - DAV-1.3.3 - feed-1.0.0.0 - ghc-8.4.4 - http-client-0.5.13.1 - persistent-sqlite-2.8.2 - torrent-10000.1.1 - uuid-1.3.13 - yesod-1.6.0 - key/value backends: - SHA256E - SHA256 - SHA512E - SHA512 - SHA224E - SHA224 - SHA384E - SHA384 - SHA3_256E - SHA3_256 - SHA3_512E - SHA3_512 - SHA3_224E - SHA3_224 - SHA3_384E - SHA3_384 - SKEIN256E - SKEIN256 - SKEIN512E - SKEIN512 - BLAKE2B256E - BLAKE2B256 - BLAKE2B512E - BLAKE2B512 - BLAKE2B160E - BLAKE2B160 - BLAKE2B224E - BLAKE2B224 - BLAKE2B384E - BLAKE2B384 - BLAKE2BP512E - BLAKE2BP512 - BLAKE2S256E - BLAKE2S256 - BLAKE2S160E - BLAKE2S160 - BLAKE2S224E - BLAKE2S224 - BLAKE2SP256E - BLAKE2SP256 - BLAKE2SP224E - BLAKE2SP224 - SHA1E - SHA1 - MD5E - MD5 - WORM - URL - remote types: - git - gcrypt - p2p - S3 - bup - directory - rsync - web - bittorrent - webdav - adb - tahoe - glacier - ddar - git-lfs - hook - external - operating system: linux x86_64 - supported repository versions: - 7 - upgrade supported from repository versions: - 0 - 1 - 2 - 3 - 4 - 5 - 6 ## datalad - version: 0.11.6.dev114 - full_version: 0.11.6.dev114-gde6b6 ## dependencies - cmd:annex: 7.20190912-1~ndall+1 - cmd:git: 2.20.1 - cmd:bundled-git: 2.20.1 - cmd:system-git: 2.23.0 - cmd:system-ssh: 7.9p1 - appdirs: 1.4.3 - boto: 2.49.0 - exifread: 2.1.2 - git: 3.0.2 - gitdb: 2.0.5 - humanize: 0.5.1 - iso8601: 0.1.12 - keyring: 19.1.0 - keyrings.alt: 3.1.1 - msgpack: 0.6.1 - mutagen: 1.42.0 - requests: 2.22.0 - scrapy: 1.7.3 - six: 1.12.0 - wrapt: 1.11.2

yarikoptic commented 4 years ago

~~I think~~ I reproduced after downgrade of systemwide git to 1:2.20.1-2 . Something in the back of my head says that we (or @joeyh) had some intermittent git issue which was resolved... and not sure if worth time investment ATM from our side if we know that it is not present with more recent git. What do you think @kyleam ?

yarikoptic commented 4 years ago

may be quick and dirty workaround would be to skip this test whenever systemwide git is 2.20.1, and release datalad-crawler. I will submit a PR for that shortly so we are ready

yarikoptic commented 4 years ago

gy gy -- that test even marked as @known_failure_v6 #FIXME ;) dedicated issue now: https://github.com/datalad/datalad/issues/3706

kyleam commented 4 years ago

had some intermittent git issue which was resolved

That doesn't seem consistent with what I'm seeing. I can trigger the issue with the latest git.

joeyh commented 4 years ago

git < 2.22 has the smudge memory leak, and so git-annex refuses to upgrade a direct mode repo to v7 when that version of git is in use.

I don't know of anything else specific to git 2.20.

-- see shy jo

yarikoptic commented 4 years ago

Thank you @joeyh

datalad / datalad-crawler

test_openfmri_pipeline1 errors with latest git-annex #55