search

ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

[run] Q: we `save` aux .repronim/* output even if no actual output is produced, intentional? #459

Closed yarikoptic closed 5 years ago

yarikoptic commented 5 years ago
With ` reproman run --follow -r localshell --sub condor --orc datalad-pair --jp working_directory=$PWD --bp "sl=1,2" sleep "{p[sl]}"` ```shell (git-annex)hopa:…im/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3[master]git-annex $> git show --stat commit 7aa5f0d99eb6c5e5a865d98d335a17664fdde806 (HEAD -> master) Author: Yaroslav Halchenko Date: Fri Sep 20 12:17:35 2019 -0400 [ReproMan] Configure jobs directory .reproman/jobs/.gitattributes | 5 +++++ .reproman/jobs/.gitignore | 3 +++ 2 files changed, 8 insertions(+) (dev3) 1 28809.....................................:Fri 20 Sep 2019 12:33:00 PM EDT:. (git-annex)hopa:…im/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3[master]git-annex $> reproman run --follow -r localshell --sub condor --orc datalad-pair --input data/bids --jp working_directory=$PWD --bp "sl=1,2,3,4,5,6,7,8" sleep (dev3) 1 28809 ->130.....................................:Fri 20 Sep 2019 12:33:15 PM EDT:. (git-annex)hopa:…im/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3[master]git-annex $> reproman run --follow -r localshell --sub condor --orc datalad-pair --jp working_directory=$PWD --bp "sl=1,2" sleep "{p[sl]}" 2019-09-20 12:33:34,333 [INFO ] No root directory supplied for localshell; using '/home/yoh/.reproman/run-root' 2019-09-20 12:33:34,489 [INFO ] Submitting 20190920-123333-4276 2019-09-20 12:33:34,523 [INFO ] Submitting /home/yoh/proj/repronim/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3/.reproman/jobs/localshell/20190920-123333-4276/submit 2019-09-20 12:33:34,579 [INFO ] Job 20190920-123333-4276 submitted as condor job 15 2019-09-20 12:33:34,587 [INFO ] Registered job 20190920-123333-4276 2019-09-20 12:33:34,603 [INFO ] Waiting on job 15: idle 2019-09-20 12:33:44,650 [INFO ] Waiting on job 15: idle 2019-09-20 12:33:54,688 [INFO ] Waiting on job 15: running 2019-09-20 12:34:04,732 [INFO ] Fetching results for 20190920-123333-4276 2019-09-20 12:34:04,786 [INFO ] Unregistered job 20190920-123333-4276 (dev3) 1 28810.....................................:Fri 20 Sep 2019 12:34:04 PM EDT:. (git-annex)hopa:…im/reproman-master/docs/usecases/bids-fmriprep-workflow-NP/out3[master]git-annex $> git show --stat commit 55d3a635e4522c65962a174539ae7ff09dcec3f9 (HEAD -> master, refs/reproman/20190920-123333-4276) Author: Yaroslav Halchenko Date: Fri Sep 20 12:33:59 2019 -0400 [ReproMan] save results for 20190920-123333-4276 .reproman/jobs/localshell/20190920-123333-4276/command-array | 1 + .reproman/jobs/localshell/20190920-123333-4276/idmap | 1 + .reproman/jobs/localshell/20190920-123333-4276/runscript | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ .reproman/jobs/localshell/20190920-123333-4276/status.0 | 1 + .reproman/jobs/localshell/20190920-123333-4276/status.1 | 1 + .reproman/jobs/localshell/20190920-123333-4276/stderr.0 | 1 + .reproman/jobs/localshell/20190920-123333-4276/stderr.1 | 1 + .reproman/jobs/localshell/20190920-123333-4276/stdout.0 | 3 +++ .reproman/jobs/localshell/20190920-123333-4276/stdout.1 | 5 +++++ .reproman/jobs/localshell/20190920-123333-4276/submit | 15 +++++++++++++++ 10 files changed, 102 insertions(+) ```

I saw that there is a commit generated with all job relevant information committed, whenever there were no actual output.

It might actually be by design I guess -- this way, even if command fails, we would get access to the outputs/logs.

Was that "by design" @kyleam ?

kyleam commented 5 years ago

Yes, it was intentional. The main benefit is for failed commands as you suggest. But more generally, the user asked us to record execution, so we record what there is to record. If the user wants to drop that information, they can easily do so.

yarikoptic commented 5 years ago

Great, thanks!