search

mih / datalad-mihextras

DataLad extension with special interest functionality or drafts of future additions to DataLad proper
Other
1 stars 1 forks source link

have been red in datalad-extensions for awhile #27

Closed yarikoptic closed 2 years ago

yarikoptic commented 2 years ago

see https://github.com/datalad/datalad-extensions/

mih commented 2 years ago

I tried to replicate the failure locally, but I cannot reproduce it. Moreover, coincidentally, https://github.com/mih/datalad-mihextras/pull/25 also tests against datalad's master, but does not fail (in this particular way).

yarikoptic commented 2 years ago

TL;DR edit: install neurodebian version of git-annex-standalone to reproduce

there was a question on matrix either it is still pertinent, so FWIW, failing testing didn't go away since they came at:

(git)smaug:/mnt/datasets/datalad/ci/datalad-extensions[master]2022
$> ls -1d */*/cron/*/*/*mih* | grep -3 -e -failed | head
02/10/cron/20220210T024704/6e4081d/github-test-datalad_mihextras-maint-335-success/
02/11/cron/20220211T024806/6e4081d/github-test-datalad_mihextras-master-336-success/
02/11/cron/20220211T024817/6e4081d/github-test-datalad_mihextras-maint-336-success/
02/12/cron/20220212T024951/6e4081d/github-test-datalad_mihextras-master-337-failed/
02/12/cron/20220212T024957/6e4081d/github-test-datalad_mihextras-maint-337-failed/
02/13/cron/20220213T024747/6e4081d/github-test-datalad_mihextras-master-338-failed/

seems to remain at stable FAILED (SKIP=2, errors=8) for both datalad-git-annex-latest and neurodebian environments.

looking at a sample they all are due to

2022-04-18T03:09:25.3531150Z ======================================================================
2022-04-18T03:09:25.3531745Z ERROR: datalad_mihextras.gitremote.tests.test_datalad_annex.test_annex_remote
2022-04-18T03:09:25.3532649Z ----------------------------------------------------------------------
2022-04-18T03:09:25.3533100Z Traceback (most recent call last):
2022-04-18T03:09:25.3543096Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
2022-04-18T03:09:25.3543896Z     self.test(*self.arg)
2022-04-18T03:09:25.3545007Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 214, in _wrap_skip_if_on_windows
2022-04-18T03:09:25.3545582Z     return func(*args, **kwargs)
2022-04-18T03:09:25.3546329Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile
2022-04-18T03:09:25.3547120Z     return t(*(arg + (filename,)), **kw)
2022-04-18T03:09:25.3547988Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile
2022-04-18T03:09:25.3548540Z     return t(*(arg + (filename,)), **kw)
2022-04-18T03:09:25.3549357Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad_mihextras/gitremote/tests/test_datalad_annex.py", line 67, in test_annex_remote
2022-04-18T03:09:25.3549906Z     _check_push_fetch_cycle(ds, dlaurl, remotepath)
2022-04-18T03:09:25.3550702Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile
2022-04-18T03:09:25.3551252Z     return t(*(arg + (filename,)), **kw)
2022-04-18T03:09:25.3552064Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad_mihextras/gitremote/tests/test_datalad_annex.py", line 99, in _check_push_fetch_cycle
2022-04-18T03:09:25.3552709Z     dsrepo.call_git(['push', 'dla'])
2022-04-18T03:09:25.3553462Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/gitrepo.py", line 437, in call_git
2022-04-18T03:09:25.3553992Z     read_only=read_only))
2022-04-18T03:09:25.3554772Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/gitrepo.py", line 482, in call_git_items_
2022-04-18T03:09:25.3555227Z     sep=sep):
2022-04-18T03:09:25.3555974Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/dataset/gitrepo.py", line 345, in _generator_call_git
2022-04-18T03:09:25.3556530Z     for file_no, content in generator:
2022-04-18T03:09:25.3558928Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/_collections_abc.py", line 317, in __next__
2022-04-18T03:09:25.3559342Z     return self.send(None)
2022-04-18T03:09:25.3560056Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/runner/nonasyncrunner.py", line 98, in send
2022-04-18T03:09:25.3560500Z     self._check_result()
2022-04-18T03:09:25.3561172Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/runner/nonasyncrunner.py", line 80, in _check_result
2022-04-18T03:09:25.3561637Z     self.runner._check_result()
2022-04-18T03:09:25.3562335Z   File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/datalad/runner/nonasyncrunner.py", line 259, in _check_result
2022-04-18T03:09:25.3562821Z     stderr=decoded_output.get("stderr", None))
2022-04-18T03:09:25.3563583Z datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none push dla' failed with exitcode 128 [err: 'fatal: The current branch dl-test-branch has no upstream branch.
2022-04-18T03:09:25.3564203Z To push the current branch and set the remote as upstream, use
2022-04-18T03:09:25.3592562Z 
2022-04-18T03:09:25.3592985Z     git push --set-upstream dla dl-test-branch']

with FWIW

Versions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=10.20220322+git36-g6bd23e172-1~ndall+1 cmd:bundled-git=2.30.2 cmd:git=2.30.2 cmd:system-git=2.35.1 cmd:system-ssh=8.2p1 git=3.1.27 gitdb=4.0.9 humanize=4.0.0 iso8601=1.0.2 keyring=23.5.0 keyrings.alt=UNKNOWN msgpack=1.0.3 platformdirs=2.5.1 requests=2.27.1

of the datalad/git-annex bulid and then for neurodebian one

2022-04-18T03:09:39.7606458Z Versions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=8.20211123+git12-g02e3756bd-1~ndall+1 cmd:bundled-git=2.30.2 cmd:git=2.30.2 cmd:system-git=2.35.1 cmd:system-ssh=8.2p1 git=3.1.27 gitdb=4.0.9 humanize=4.0.0 iso8601=1.0.2 keyring=23.5.0 keyrings.alt=UNKNOWN msgpack=1.0.3 platformdirs=2.5.1 requests=2.27.1

with the ones in appveyor run of https://github.com/mih/datalad-mihextras/pull/25

Versions: annexremote=1.5.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=10.20220121-gdf6a8476e cmd:bundled-git=UNKNOWN cmd:git=2.34.1 cmd:system-git=2.34.1 cmd:system-ssh=8.2p1 git=3.1.27 gitdb=4.0.9 humanize=4.0.0 iso8601=1.0.2 keyring=23.5.0 keyrings.alt=UNKNOWN msgpack=1.0.3 platformdirs=2.5.1 requests=2.27.1

but I don't think that should matter since error looks more like missing the required...

at first I failed to reproduce with git-annex from debian, but installing one from neurodebian did reproduce the sample fail ```shell $> python -m nose -s -v datalad_mihextras/gitremote/tests/test_datalad_annex.py:test_export_remote datalad_mihextras.gitremote.tests.test_datalad_annex.test_export_remote ... create(ok): . (dataset) ERROR Versions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=8.20210223-1~ndall+1 cmd:bundled-git=2.24.0 cmd:git=2.24.0 cmd:system-git=2.35.1.473.g83b2b277ed cmd:system-ssh=8.7p1 git=3.1.27 gitdb=4.0.9 humanize=4.0.0 iso8601=1.0.2 keyring=23.5.0 keyrings.alt=UNKNOWN msgpack=1.0.3 platformdirs=2.5.2 requests=2.27.1 Obscure filename: str=b' |;&%b5{}\'"\xce\x94\xd0\x99\xd7\xa7\xd9\x85\xe0\xb9\x97\xe3\x81\x82 .datc ' repr=' |;&%b5{}\'"ΔЙקم๗あ .datc ' Encodings: default='utf-8' filesystem='utf-8' locale.prefered='UTF-8' Environment: PATH='/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/bin:/home/yoh/gocode/bin:/home/yoh/gocode/bin:/home/yoh/bin:/home/yoh/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/usr/local/sbin' LANG='en_US.UTF-8' GIT_PAGER='less --no-init --quit-if-one-screen' GIT_CONFIG_PARAMETERS="'init.defaultBranch=master' 'clone.defaultRemoteName=origin'" GIT_ASKPASS='true' ====================================================================== ERROR: datalad_mihextras.gitremote.tests.test_datalad_annex.test_export_remote ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile return t(*(arg + (filename,)), **kw) File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile return t(*(arg + (filename,)), **kw) File "/home/yoh/proj/datalad/datalad-mihextras/datalad_mihextras/gitremote/tests/test_datalad_annex.py", line 79, in test_export_remote _check_push_fetch_cycle(ds, dlaurl, remotepath) File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/tests/utils.py", line 874, in _wrap_with_tempfile return t(*(arg + (filename,)), **kw) File "/home/yoh/proj/datalad/datalad-mihextras/datalad_mihextras/gitremote/tests/test_datalad_annex.py", line 99, in _check_push_fetch_cycle dsrepo.call_git(['push', 'dla']) File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/dataset/gitrepo.py", line 432, in call_git return "\n".join( File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/dataset/gitrepo.py", line 478, in call_git_items_ for file_no, line in self._generator_call_git( File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/dataset/gitrepo.py", line 345, in _generator_call_git for file_no, content in generator: File "/usr/lib/python3.9/_collections_abc.py", line 330, in __next__ return self.send(None) File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py", line 98, in send self._check_result() File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py", line 80, in _check_result self.runner._check_result() File "/home/yoh/proj/datalad/datalad-mihextras/venvs/dev3/lib/python3.9/site-packages/datalad/runner/nonasyncrunner.py", line 256, in _check_result raise CommandError(cmd=self.cmd, datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none push dla' failed with exitcode 128 [err: 'fatal: The current branch master has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream dla master'] ---------------------------------------------------------------------- Ran 1 test in 0.378s FAILED (errors=1) ```
mih commented 2 years ago

Here is the dependency info for the CI run that just passed again

Versions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=10.20220223-ge37f0f227 cmd:bundled-git=UNKNOWN cmd:git=2.35.1 cmd:system-git=2.35.1 cmd:system-ssh=8.2p1 git=3.1.27 gitdb=4.0.9 humanize=4.0.0 iso8601=1.0.2 keyring=23.5.0 keyrings.alt=UNKNOWN msgpack=1.0.3 platformdirs=2.5.2 requests=2.27.1

Compared to the failure you were able to replicate

cmd:annex=8.20210223-1~ndall+1 cmd:bundled-git=2.24.0

According to the note in .appveyor.yml this would indeed not work:

      # datalad-annex git remote needs something after git-annex_8.20211x
      INSTALL_GITANNEX: git-annex -m snapshot

But in the PR migrating this code to -next I see the failure on github-actions too, and it gives me

annexremote: 1.6.0 boto: 2.49.0 cmd:7z: 16.02 cmd:annex: 8.20211123+git12-g02e3756bd-1~ndall+1 cmd:bundled-git: 2.30.2  cmd:git: 2.30.2 cmd:system-git: 2.35.1 cmd:system-ssh: 8.2p1 humanize: 4.0.0 iso8601: 1.0.2 keyring: 23.5.0 keyrings.alt: 4.1.0 msgpack: 1.0.3 platformdirs: 2.5.2 requests: 2.27.1

So 8.20211+ might not even be sufficient, or it is some interaction of git-version and annex version.

mih commented 2 years ago

Release is imminent now https://github.com/mih/datalad-mihextras/pull/36 (pending a datalad-next >= 0.2 release)

mih commented 2 years ago

Release was made. Should be good now.