search

ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

run: failed due to non output directory was fetched (and "conflicted" upon extraction) #471

Open yarikoptic opened 4 years ago

yarikoptic commented 4 years ago

We ran

(git)smaug:…eproNim_Workshop_2019_1018/ReproNim_Workshop_2019_1018-reproman[master]git
$> reproman run --follow -r my-nitrc-large --orchestrator datalad-pair-run --input "*/sub-*/ses-*/anat/*T1w.nii.gz" --output "simple_workflow_output" --jp "container=containers/repronim-simple-workflow" ./code/simple_workflow/run_demo_workflow.sh -o simple_workflow_output -w workdir --plugin_args 'dict(n_procs=96)' */sub-*/ses-*/anat/*T1w.nii.gz

which failed with

2019-10-18 20:41:02,825 [INFO   ] Waiting on job 32295: running
2019-10-18 20:41:13,005 [INFO   ] Fetching results for 20191018-173634-245d
2019-10-18 20:49:13,256 [ERROR  ] [Errno 13] Permission denied: '././workdir/metaflow/sub-000303269784_ses-20050906_T1w/segment_brain/sub-000303269784_ses-20050906_T1w_reoriented_brain_seg.nii.gz' [tarfile.py:makefile:2155] (PermissionError)
reproman run --follow -r my-nitrc-large --orchestrator datalad-pair-run        204.84s user 45.81s system 2% cpu 3:12:41.00 total

(ignore permission denied somewhat for now) which is strange since workdir is not part of the outputs (so I guess should have not even being fetched or considered for fetching results from the remote resource), and it is also ignored:

$> cat .gitignore
workdir

(git)smaug:…eproNim_Workshop_2019_1018/ReproNim_Workshop_2019_1018-reproman[master]git
$> git status
On branch master
Your branch is ahead of 'origin/master' by 4 commits.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)

        modified:   simple_workflow_output (untracked content)

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .reproman/jobs/my-nitrc-large/20191018-173634-245d/
        20191018-173634-245d.tar.gz

so workdir/ should have not been fetched or considered for any addition. and apparently it was transferred (present in that 20191018-173634-245d.tar.gz)

the files for which permission was denied are actually are part of our initial run on a sample subject, but apparently they were fetched as well... so probably the "Permission denied" is just a side effect of workdir/ being fetched whenever it should have not been

yarikoptic commented 4 years ago

note: apparently underlying command we ran itself failed as well. So may be reproman has different logic on what to fetch in case of command erroring out?