kyleam
commented
4 years ago [publish failure]
I wish ERROR message provided a more informative message (i.e. how it failed, some stderr or returned record from
datalad publishinvocation if done in Python).
Indeed, that was pretty hard to do anything with. It should be more helpful after 45baab076 (ENH: orchestrators: Provide access to datalad-publish results, 2020-05-14), which is in gh-506.
Running with local submitter and datalad-pair-run (I still can't figure out why my tree is left dirty when using datalad-pair), I was able to trigger a publish error, or rather multiple. The first one a saw was due to another "git tries to fetch non-existent origin" issue (gh-499). A similar issue was fixed in update, but this one is in publish.
error
``` [ERROR ] CommandError: 'git fetch --verbose --progress local' failed with exitcode 1 under /home/kyle/scratch/reproman/gh-438/out5 [err: 'remote: Total 25 (delta 4), reused 0 (delta 0), pack-reused 0 | From /home/kyle/.reproman/run-root/1e5d7eba-9ace-11ea-a5bd-54e1ada290c0 | 30381ed..97b1944 git-annex -> local/git-annex | 7dc5d91..b9c9f0a master -> local/master | Fetching submodule data/mriqc | fatal: 'origin' does not appear to be a git repository | fatal: Could not read from remote repository. | | Please make sure you have the correct access rights | and the repository exists. | Errors during submodule fetch: | data/mriqc'] [cmd.py:run:477] (CommandError) ```DataLad's 8343899d3 (BF: publish: Tell git-fetch to not recurse into submodules, 2020-05-20), which you reviewed/merged yesterday (thanks!), resolves this issue. Running things again gets farther, but hits another publish error. Based on the spot were the failure happened in your log, I'd guess this might be the one you hit.
[ERROR ] 'datalad publish' failed: {'action': 'publish', 'path': '/home/kyle/scratch/reproman/gh-438/out6/data/fmriprep', 'type': 'dataset', 'refds': '/home/kyle/scratch/reproman/gh-438/out6', 'status': 'error', 'mode_src': 0, 'mode': 57344, 'revision_src': None, 'revision': '0cb506b90c06a7d7d7b88e30d2808fb50d7ebc27', 'parentds': '/home/kyle/scratch/reproman/gh-438/out6', 'state': 'added', 'type_src': None, 'registered_subds': True, 'process_content': True, 'message': ("Unknown target sibling '%s' for publication", 'local')} [orchestrators.py:prepare_remote:792] (OrchestratorErrorSo, it looks like run needs to do a better job of setting up the sibling in subdatasets that pop up after the initial run. You mentioned that running the suggested publish command manually worked, but I think that's just because the suggestion left out --recursive :/
yarikoptic
This was an attempt to run with @kyleam 's version of the script with only minor tune ups, as available from the branch scratch/use-case-local here. Command I had was
and full log try1.log is now at http://www.onerussian.com/tmp/reproman-try1.txt (renamed).
1. The overall failure was due to
while it went just fine upon checking and retrying after crash
```shell lena:~/temp-test-reproman $> datalad status -d /home/yoh/temp-test-reproman/try1 nothing to save, working tree clean $> datalad siblings -d /home/yoh/temp-test-reproman/try1 .: here(+) [git] .: local(+) [/home/yoh/.reproman/run-root/ac410208-90ae-11ea-889c-c3382a41054b (git)] $> datalad status -d /home/yoh/.reproman/run-root/ac410208-90ae-11ea-889c-c3382a41054b nothing to save, working tree clean $> datalad publish -d /home/yoh/temp-test-reproman/try1 --to=local ```which is strange. I wish ERROR message provided a more informative message (i.e. how it failed, some stderr or returned record from
datalad publishinvocation if done in Python).2. In the log you could see above this failure
for running mriqc. so there is a WARNING but overall it seems it didn't cause the process to fail. @kyleam shouldn't overall
run --followfail if subjob fails?trusting noone including myself I have tested that it is not somehow my demo script which should fail if individual command fails due to `set -e` on the top... it fails as expected with the following patch
```shell (git)lena:~/proj/repronim/reproman[scratch/use-case-local] $> git diff diff --git a/docs/usecases/bids-fmriprep-workflow-NP.sh b/docs/usecases/bids-fmriprep-workflow-NP.sh index 3ac4eca..c54473c 100755 --- a/docs/usecases/bids-fmriprep-workflow-NP.sh +++ b/docs/usecases/bids-fmriprep-workflow-NP.sh @@ -97,7 +97,8 @@ unknown_runner () { # Common invocation of ReproMan # TODO: just make it configurable per project/env? reproman_run () { - reproman run --follow -r "${RM_RESOURCE}" --sub "${RM_SUB}" --orc "${RM_ORC}" "$@" + /bin/ls bogus + # reproman run --follow -r "${RM_RESOURCE}" --sub "${RM_SUB}" --orc "${RM_ORC}" "$@" } ```FTR, the content of that stderr file was
my singularity-container from neurodebian is 2.6.1-2~nd100+1
DataLad 0.13.0rc1 WTF (dependencies, datalad)
# WTF ## datalad - full_version: 0.13.0rc1 - version: 0.13.0rc1 ## dependencies - appdirs: 1.4.3 - boto: 2.49.0 - cmd:7z: 16.02 - cmd:annex: 7.20190708+git9-gfa3524b95-1~ndall+1 - cmd:bundled-git: 2.20.1 - cmd:git: 2.20.1 - cmd:system-git: 2.26.2 - cmd:system-ssh: 8.1p1 - exifread: 2.1.2 - git: 3.1.0 - gitdb: 4.0.2 - humanize: 2.3.0 - iso8601: 0.1.12 - keyring: 18.0.1 - keyrings.alt: 3.4.0 - msgpack: 0.6.2 - mutagen: 1.40.0 - requests: 2.23.0 - tqdm: 4.43.0 - wrapt: 1.11.2