containers/scripts/singularity_cmd: Argument list too long

[DVSneuro]

Hi -- I'm trying to use the ReproNim container for HeuDiConv in datalad, and I'm getting and error with singularity_cmd (containers/scripts/singularity_cmd: Argument list too long) when using datalad containers-run. I wasn't sure if I should post this here or the datalad issues page, but I think the singularity_cmd is from this repository.

Example code is below.

# create dataset and provide inputs and code
sub=10369
datalad create -c yoda "datalad-testing-rf1"
cd datalad-testing-rf1
mkdir -p inputs/rawdata
cp -r /data/sourcedata/rf1-sra/Smith-SRA-$sub inputs/rawdata/.
eval "datalad save -m 'add dicoms for sub-${sub}' inputs/rawdata --jobs 16"
cp ../rf1-sra-data/code/heuristics.py code/.
cp ../rf1-sra-data/code/prepdata.sh code/.
datalad save -m 'add code' code

# install repronim containers
datalad install -d . ///repronim/containers

# run heudiconv
datalad containers-run -m "Convert sub-10369 DICOMs into BIDS" \
  --container-name containers/nipy-heudiconv \
  'heudiconv -d inputs/rawdata/Smith-SRA-10369/*/scans/*/*/DICOM/files/*.dcm -f code/heuristics.py -s 10369 -c dcm2niix -b --minmeta -o outputs/sub-10369'

And here's the full output of the error:

[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
/bin/sh: 1: containers/scripts/singularity_cmd: Argument list too long
[INFO   ] == Command exit (modification check follows) ===== 
run(error): /data/projects/datalad-testing-rf1-05 (dataset) [containers/scripts/singularity_cmd run c...]
action summary:                                                                                                                                                                                                                           
  get (notneeded: 2)
  run (error: 1)
  save (notneeded: 2)

Thanks for any tips or pointers!

David

[yarikoptic]

• you don't need to include heudiconv command in your invocation, since it is already an entry point in that container:

  
  $> datalad containers-run --container-name nipy-heudiconv
  [INFO   ] Making sure inputs are available (this may take some time) 
  [INFO   ] == Command start (output follows) ===== 
  WARNING: Nothing to be done - displaying usage help
  usage: heudiconv [-h] [--version] [-d DICOM_DIR_TEMPLATE | --files [FILES ...]] [-s [SUBJS ...]] [-c {dcm2niix,none}]
               [-o OUTDIR] [-l LOCATOR] [-a CONV_OUTDIR] [--anon-cmd ANON_CMD] [-f HEURISTIC] [-p] [-ss SESSION]
               [-b [BIDSOPTION1 [BIDSOPTION2 ...]]] [--overwrite] [--datalad] [--dbg]
               [--command {heuristics,heuristic-info,ls,populate-templates,sanitize-jsons,treat-jsons,populate-intended-for}]
               [-g {studyUID,accession_number,all,custom}] [--minmeta] [--random-seed RANDOM_SEED] [--dcmconfig DCMCONFIG]
               [-q {SLURM,None}] [--queue-args QUEUE_ARGS]

Example: heudiconv -d 'rawdata/{subject}' -o . -f heuristic.py -s s1 s2 s3

...

- the actual issue is probably in the large number of those dicom files you are pointing to, i.e. do smth like that to discover that

/bin/ls -1 inputs/rawdata/Smith-SRA-10369//scans///DICOM/files/.dcm | nl | tail -n 1

which is what triggers that error message
- I don't think you want to use `-d` argument (since not using template) but rather directly `--files` and then do not bother listing the files but point to folders e.g. `--files inputs/rawdata/Smith-SRA-10369/*/scans/*/*/DICOM/files`

[DVSneuro]

Thanks! That fixed it! Closing this issue.