search

datalad / datalad-crawler

DataLad extension for tracking web resources as datasets
http://datalad.org
Other
5 stars 16 forks source link

datalad core testing started to fail #124

Open yarikoptic opened 1 year ago

yarikoptic commented 1 year ago 
(git)smaug:/mnt/datasets/datalad/ci/logs/2022/08[master]git
$> git grep -l 'getSymbolicLinkStatus: does not exist'
11/pr/6935/9d6dc72/github-Extensions-5378-failed/4_test (datalad-crawler).txt
11/pr/6935/9d6dc72/github-Extensions-5378-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
12/pr/6932/cc859ae/github-Extensions-5386-failed/4_test (datalad-crawler).txt
12/pr/6932/cc859ae/github-Extensions-5386-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
12/pr/6934/f4e6f33/github-Extensions-5382-failed/4_test (datalad-crawler).txt
12/pr/6934/f4e6f33/github-Extensions-5382-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
12/pr/6937/44525c2/github-Extensions-5385-failed/4_test (datalad-crawler).txt
12/pr/6937/44525c2/github-Extensions-5385-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
12/push/maint/804b3c9/github-Extensions-5384-failed/4_test (datalad-crawler).txt
12/push/maint/804b3c9/github-Extensions-5384-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
14/pr/6942/f43826e/github-Extensions-5387-failed/4_test (datalad-crawler).txt
14/pr/6942/f43826e/github-Extensions-5387-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt
15/pr/6942/3c8f7c1/github-Extensions-5388-failed/4_test (datalad-crawler).txt
15/pr/6942/3c8f7c1/github-Extensions-5388-failed/test (datalad-crawler)/13_datalad-crawler tests using pytest.txt

not yet sure if new git annex version (likely, I think I uploaded new standalone to debian back then)

yarikoptic commented 1 year ago

FWIW -- yes, relates to git-annex new version

$> grep Versions {10/pr/6932/f80a141/github-Extensions-5374-success,11/pr/6935/9d6dc72/github-Extensions-5378-failed}/4_test\ \(datalad-crawler\).txt
10/pr/6932/f80a141/github-Extensions-5374-success/4_test (datalad-crawler).txt:2022-08-10T21:29:48.4083725Z tests/test_utils.py::test_get_func_kwargs_doc PASSEDVersions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=10.20220322-1~ndall+1 cmd:bundled-git=2.30.2 cmd:git=2.30.2 cmd:ssh=8.2p1 cmd:system-git=2.37.1 cmd:system-ssh=8.2p1 datalad=0.17.2+99.g3e5dfe344 exifread=3.0.0 humanize=4.3.0 iso8601=1.0.2 keyring=23.8.2 keyrings.alt=4.1.1 mock=4.0.3 msgpack=1.0.4 mutagen=1.45.1 platformdirs=2.5.2 requests=2.28.1 scrapy=2.6.2
11/pr/6935/9d6dc72/github-Extensions-5378-failed/4_test (datalad-crawler).txt:2022-08-11T22:26:38.3468591Z tests/test_utils.py::test_get_func_kwargs_doc PASSEDVersions: annexremote=1.6.0 boto=2.49.0 cmd:7z=16.02 cmd:annex=10.20220724-1~ndall+1 cmd:bundled-git=2.30.2 cmd:git=2.30.2 cmd:ssh=8.2p1 cmd:system-git=2.37.1 cmd:system-ssh=8.2p1 datalad=0.17.2+80.g2fd2f30bb exifread=3.0.0 humanize=4.3.0 iso8601=1.0.2 keyring=23.8.2 keyrings.alt=4.1.1 mock=4.0.3 msgpack=1.0.4 mutagen=1.45.1 platformdirs=2.5.2 requests=2.28.1 scrapy=2.6.2

so went from 10.20220322-1~ndall+1 to 10.20220724-1~ndall+1 and now we are observing errors like

2022-08-11T22:26:38.4004204Z E           datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none annex add -c 'annex.largefiles=exclude=Makefile and exclude=LICENSE* and exclude=ISSUES* and exclude=CHANGES* and exclude=README* and exclude=ReadMe.txt and exclude=*.[mc] and exclude=dataset*.json and exclude=license.txt and (exclude=*.txt or include=*/*.txt) and (exclude=*.json or include=*/*.json) and (exclude=*.tsv or include=*/*.tsv)' --json --json-error-messages -c annex.dotfiles=true -- .' failed with exitcode 1 under /tmp/datalad_temp_test_openfmri_pipeline1vv3jbs5q [info keys: stdout_json] [err: 'git-annex: ds666.tar.gz: getSymbolicLinkStatus: does not exist (No such file or directory)']

and it reproduces locally with annex 10.20220724+git120-g7286f408f-1~ndall+1 -- so now need to figure out what lead to this new problem, uff (attn @joeyh)

yarikoptic commented 1 year ago

filed against git-annex https://git-annex.branchable.com/bugs/regression__58___fails_to___39__add_.__39___if_file_was_renamed/?updated