ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

datalad* orchestrators do not care to unlock outputs #546

Open yarikoptic opened 3 years ago

yarikoptic commented 3 years ago

I was composing the FOI success report but decided to do the diff between sequential datalad containers-run and reproman run

using this script ```shell #!/bin/bash set -eu export PS4='> ' set -x get_participant_ids () { # Would go through provided paths and current directory to find participants.tsv # and return participant ids, comma-separated for p in "$@" .; do f="$p/participants.tsv" if [ -e "$f" ]; then awk -F'\t' '/^sub-/{print $1}' "$f" \ | sed 's/sub-//' \ | tr '\n' ',' \ | sed -e 's/,$//g' break fi done } reproman run \ --follow \ -r local \ --sub condor \ --orc datalad-no-remote \ --jp container=.datalad/containers/bids-bidsonym \ --input "sub-{p[pl]}" \ --output "sub-{p[pl]}" \ --output "sourcedata/bidsonym/sub-{p[pl]}" \ --output "sourcedata/bidsonym/dataset_description_no_deid.json" \ --output "sourcedata/bidsonym/participants_no_deid.json" \ --bp "pl=$(get_participant_ids . )" \ $PWD participant '--participant_label={p[pl]}' --brainextraction bet --bet_frac 0.5 --deid mri_deface ```

but results diverged "badly" so I realized that:

yarikoptic commented 3 years ago

per weekly chat discussion: most likely should just add a job parameter to unlock-outputs. Could possibly become not just a flag but a regex or smth like that which would "filter" corresponding to the job outputs to the ones matching or not. Behavior could be similar to datalad to simply "remove" outputs which were not provided as inputs, or we could add yet another one to be an alternative to "unlock" - remove-outputs