singularity exec fails when dataset is in directory that contains a symlink

[mih]

containers-run output:

[INFO   ] == Command start (output follows) =====                                            
INFO: Running heudiconv version 0.5.dev1
Traceback (most recent call last):
  File "/usr/local/bin/heudiconv", line 11, in <module>
    load_entry_point('heudiconv==0.5.dev1', 'console_scripts', 'heudiconv')()
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/cli/run.py", line 120, in main
    process_args(args)
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/cli/run.py", line 244, in process_args
    args.subjs, grouping=args.grouping)
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/parser.py", line 164, in get_study_sessions
    for _, files_ex in get_extracted_dicoms(files):
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/parser.py", line 84, in get_extracted_dicoms
    if not tarfile.is_tarfile(t):
  File "/usr/lib/python3.5/tarfile.py", line 2450, in is_tarfile
    t = open(name)
  File "/usr/lib/python3.5/tarfile.py", line 1559, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib/python3.5/tarfile.py", line 1685, in xzopen
    fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset)
  File "/usr/lib/python3.5/lzma.py", line 118, in __init__
    self._fp = builtins.open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mih/pd/scratch/multires3t/bids/sourcedata/REDACTED/dicoms'
[INFO   ] == Command exit (modification check follows) ===== 

pd is a symlink.

I tried the following (which works on the commandline), but causes some yet-to-be-determined issue:

diff --git a/.datalad/config b/.datalad/config
index b139e7b..f7f7752 100644
--- a/.datalad/config
+++ b/.datalad/config
@@ -3,4 +3,4 @@
 [datalad "containers.conversion"]
        updateurl = shub://mih/ohbm2018-training:heudiconv
        image = .datalad/environments/conversion/image
-       cmdexec = [\"singularity\", \"exec\", \"{img}\", \"{cmd}\"]
+       cmdexec = [\"singularity\", \"exec\", \"--bind\", \"$(readlink -f $(pwd))\", \"{img}\", \"{cmd}\"]

% datalad hirni-spec2bids -s sourcedata/REDACTED
[INFO   ] == Command start (output follows) ===== 
ERROR  : Target /var/lib/singularity/mnt/final/$(readlink -f $(pwd)) doesn't exist
ABORT  : Retval = 255
[INFO   ] == Command exit (modification check follows) ===== 
[INFO   ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -r -F.git/COMMIT_EDITMSG .' 
Failed to run ['singularity', 'exec', '--bind', '$(readlink -f $(pwd))', '.datalad/environments/conversion/image', 'heudiconv', '-f', '/home/mih/datalad-hirni/datalad_hirni/support/hirni_heuristic.py', '-s', 'REDACTED', '-c', 'dcm2niix', '-o', '/home/mih/pd/scratch/multires3t/bids/.git/stupid/REDACTED', '-b', '-a', '/home/mih/pd/scratch/multires3t/bids', '-l', '', '--minmeta', '--files', '/home/mih/pd/scratch/multires3t/bids/sourcedata/REDACTED/dicoms'] under '/home/mih/pd/scratch/multires3t/bids'. Exit code=255. out= err=%                               

(also: there is "stupid" in the path...)

[mih]

https://github.com/datalad/datalad-container/issues/34

[mih]

Situation is more complex: https://github.com/datalad/datalad-container/pull/35

[bpoldrack]

I'll leave that one open for now, but probably should move to https://github.com/psychoinformatics-de/hirni-toolbox/ since hirni itself doesn't care anymore.

[mih]

If hirni doesn't care, the toolbox (that is inside the container) shouldn't care either.