search

psychoinformatics-de / datalad-hirni

DataLad extension for (semi-)automated, reproducible processing of (medical/neuro)imaging data
http://datalad.org
Other
5 stars 8 forks source link

hirni-spec2bids > [Cannot find procedure with name 'hirni-dicom-converter'] #106

Closed ted-strauss-K1 closed 5 years ago

ted-strauss-K1 commented 5 years ago

This error seems to be unrelated to the studyspec file. Let me know what background env you may need to know.

# datalad --dbg hirni-spec2bids -d . sourcedata/studyspec2.json 
[WARNING] Cannot find procedure with name 'hirni-dicom-converter' [run_procedure(hirni-dicom-converter)] 
run_procedure(impossible): hirni-dicom-converter [Cannot find procedure with name 'hirni-dicom-converter']
Traceback (most recent call last):
  File "/root/env/datalad/bin/datalad", line 8, in <module>
    main()
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/cmdline/main.py", line 494, in main
    ret = cmdlineargs.func(cmdlineargs)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/base.py", line 628, in call_from_parser
    ret = list(ret)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad_hirni/commands/spec2bids.py", line 264, in __call__
    return_type='generator'
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 467, in generator_func
    msg="Command did not complete successfully")
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully [{'action': 'run_procedure', 'path': 'hirni-dicom-converter', 'refds': '/scratch/box/webdav/box/MRI0/hirni-bids', 'status': 'impossible', 'message': "Cannot find procedure with name 'hirni-dicom-converter'"}]

> /root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py(467)generator_func()
-> msg="Command did not complete successfully")
(Pdb)
bpoldrack commented 5 years ago

Cannot find procedure with name 'hirni-dicom-converter'

This suggests, that you didn't install the toolbox, which is supposed to contain this procedure. Did you run the hirni setup procedure on your dataset? Can you check whether you can find a directory code/hirni-toolbox/ in your dataset and a directory procedures underneath it?

ted-strauss-K1 commented 5 years ago

Ok, I ran datalad run-procedure setup_hirni_dataset and now have the toolbox installed. Now I'm getting this:

# datalad --dbg hirni-spec2bids -d . sourcedata/studyspec2.json
[INFO   ] == Command start (output follows) ===== 
[WARNING] The module 'datalad_revolution.revstatus' is deprecated. The `RevStatus` class can be imported with: `from datalad.core.local.sport Status as RevStatus` 
Traceback (most recent call last):
  File "/path/to/hirni-bids/code/hirni-toolbox/procedures/hirni-dicom-converter.py", line 24, in <module>
    location = sys.argv[5]
IndexError: list index out of range
[INFO   ] == Command exit (modification check follows) ===== 
Traceback (most recent call last):
  File "/root/env/datalad/bin/datalad", line 8, in <module>
    main()
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/cmdline/main.py", line 494, in main
    ret = cmdlineargs.func(cmdlineargs)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/base.py", line 628, in call_from_parser
    ret = list(ret)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad_hirni/commands/spec2bids.py", line 264, in __call__
    return_type='generator'
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run_procedure.py", line 462, in __call__
    return_type='generator'
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 216, in __call__
    sidecar=sidecar):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 617, in run_command
    raise exc
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 385, in _execute_command
    expect_fail=True,
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/cmd.py", line 550, in run
    raise CommandError(text_type(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command 'python /path/to/hirni-bids/code/hirni-toolbox/procedui-dicom-converter.py /path/to/hirni-bids sourcedata/studyspec2.json False  sourcedata/.' failed with exitcode 1
Failed to run 'python /path/to/hirni-bids/code/hirni-toolbox/procedures/hirni-dicom-converter.py /path/to/hirni-bids sourcedata/studyspec2.json False  sourcedata/.' under '/path/to/hirni-bids'. Exit code=1.

> /root/env/datalad/lib/python3.6/site-packages/datalad/cmd.py(550)run()
-> raise CommandError(text_type(cmd), msg, status, out[0], out[1])
(Pdb)
bpoldrack commented 5 years ago

At the very least I should take care of having a proper error message, so I'll mark this issue as a bug for awareness that there's something to be done.

On the actual error: This looks like your spec tells hirni-spec2bids to convert dicoms, but the respective snippet is missing the field 'location'. Can you please post your spec or at least a simplified version if there's confidential data to leave out?

bpoldrack commented 5 years ago

Alternatively your spec might lack what we call a call format string for the procedure 'hirni-dicom-converter'. Either way: This shouldn't happen by hirni's own spec creation routines. Did that spec came to be by other means?

ted-strauss-K1 commented 5 years ago

Here is the spec that was generated by running datalad hirni-dicom2spec -d . --anon-subject 001 -s studyspec2.json . on a dicom dataset.

# cat sourcedata/studyspec2.json 
{"anon-subject":{"approved":false,"value":null},"bids-acquisition":{"approved":false,"value":null},"bids-contrast-enhancement":{"approved":false,"value":null},"bids-direction":{"approved":false,"value":null},"bids-echo":{"approved":false,"value":null},"bids-modality":{"approved":false,"value":null},"bids-reconstruction-algorithm":{"approved":false,"value":null},"bids-run":{"approved":false,"value":"1"},"bids-session":{"approved":false,"value":null},"bids-task":{"approved":false,"value":null},"comment":{"approved":false,"value":""},"dataset-id":"e4fec81a-7687-11e9-81af-00163eacdccb","dataset-refcommit":"6fad52b064035fa4a3d04134e79d4c6370ff1a1d","description":{"approved":false,"value":"GLOBAL_8ch_norm_Sag"},"id":{"approved":false,"value":2},"location":".","procedures":[{"on-anonymize":{"approved":false,"value":false},"procedure-call":{"approved":false,"value":null},"procedure-name":{"approved":false,"value":"hirni-dicom-converter"}}],"subject":{"approved":false,"value":""},"type":"dicomseries:all"}
{"anon-subject":{"approved":false,"value":null},"bids-acquisition":{"approved":false,"value":null},"bids-contrast-enhancement":{"approved":false,"value":null},"bids-direction":{"approved":false,"value":null},"bids-echo":{"approved":false,"value":null},"bids-modality":{"approved":false,"value":null},"bids-reconstruction-algorithm":{"approved":false,"value":null},"bids-run":{"approved":false,"value":"1"},"bids-session":{"approved":false,"value":null},"bids-task":{"approved":false,"value":null},"comment":{"approved":false,"value":""},"dataset-id":"e4fec81a-7687-11e9-81af-00163eacdccb","dataset-refcommit":"6fad52b064035fa4a3d04134e79d4c6370ff1a1d","description":{"approved":false,"value":"GLOBAL_8ch_norm_Sag"},"id":{"approved":false,"value":2},"location":".","subject":{"approved":false,"value":""},"tags":[],"type":"dicomseries","uid":"1.3.12.2.1107.5.2.14.16505.30000013092013431240600000005"}
ted-strauss-K1 commented 5 years ago

Ok, I was able to regenerate the spec after re-aggregating the dicom dataset. This is the new spec

{"anon-subject":{"approved":false,"value":"001"},"bids-acquisition":{"approved":false,"value":null},"bids-contrast-enhancement":{"approved":false,"value":null},"bids-direction":{"approved":false,"value":null},"bids-echo":{"approved":false,"value":null},"bids-modality":{"approved":false,"value":null},"bids-reconstruction-algorithm":{"approved":false,"value":null},"bids-run":{"approved":false,"value":"1"},"bids-session":{"approved":false,"value":null},"bids-task":{"approved":false,"value":null},"comment":{"approved":false,"value":""},"dataset-id":"e4fec81a-7687-11e9-81af-00163eacdccb","dataset-refcommit":"6fad52b064035fa4a3d04134e79d4c6370ff1a1d","description":{"approved":false,"value":"GLOBAL_8ch_norm_Sag"},"id":{"approved":false,"value":2},"location":".","procedures":[{"on-anonymize":{"approved":false,"value":false},"procedure-call":{"approved":false,"value":null},"procedure-name":{"approved":false,"value":"hirni-dicom-converter"}}],"subject":{"approved":false,"value":""},"type":"dicomseries:all"}
{"anon-subject":{"approved":false,"value":"001"},"bids-acquisition":{"approved":false,"value":null},"bids-contrast-enhancement":{"approved":false,"value":null},"bids-direction":{"approved":false,"value":null},"bids-echo":{"approved":false,"value":null},"bids-modality":{"approved":false,"value":null},"bids-reconstruction-algorithm":{"approved":false,"value":null},"bids-run":{"approved":false,"value":"1"},"bids-session":{"approved":false,"value":null},"bids-task":{"approved":false,"value":null},"comment":{"approved":false,"value":""},"dataset-id":"e4fec81a-7687-11e9-81af-00163eacdccb","dataset-refcommit":"6fad52b064035fa4a3d04134e79d4c6370ff1a1d","description":{"approved":false,"value":"GLOBAL_8ch_norm_Sag"},"id":{"approved":false,"value":2},"location":".","subject":{"approved":false,"value":""},"tags":[],"type":"dicomseries","uid":"1.3.12.2.1107.5.2.14.16505.30000013092013431240600000005"}

Oh, actually, now I realize that's the same spec as before. I have successfully extracted and aggregated dicom metadata, but it does not end up in the spec using dicom2spec. Any ideas on that?

Next, I run spec2bids and get this:

# datalad hirni-spec2bids --anonymize -d . studyspec3.json
[WARNING] Cannot find procedure with name 'hirni-dicom-converter' [run_procedure(hirni-dicom-converter)] 
run_procedure(impossible): hirni-dicom-converter [Cannot find procedure with name 'hirni-dicom-converter']
(datalad) endex:/path/to/hirni-bids/sourcedata# cd ..
(datalad) endex:/path/to/hirni-bids# datalad hirni-spec2bids --anonymize -d . sourcedata/studyspec3.json
[INFO   ] == Command start (output follows) ===== 
[WARNING] The module 'datalad_revolution.revstatus' is deprecated. The `RevStatus` class can be imported with: `from datalad.core.local.status import Status as RevStatus` 
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] To obtain some keys we need to fetch an archive of size 14.8 MB                                                                
[INFO   ] == Command start (output follows) =====                                                                                        
WARNING: Skipping user bind, non existent bind point (directory) in container: '/path/to/hirni-bids'
INFO: Running heudiconv version 0.5.2-dev
Traceback (most recent call last):
  File "/usr/local/bin/heudiconv", line 11, in <module>
    load_entry_point('heudiconv==0.5.2.dev0', 'console_scripts', 'heudiconv')()
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/cli/run.py", line 125, in main
    process_args(args)
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/cli/run.py", line 248, in process_args
    heuristic = load_heuristic(args.heuristic)
  File "/usr/local/lib/python3.5/dist-packages/heudiconv/utils.py", line 282, in load_heuristic
    mod = __import__(fname.split('.')[0])
ImportError: No module named 'hirni_heuristic'
[INFO   ] == Command exit (modification check follows) ===== 
[INFO   ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad add -d . -r -F .git/COMMIT_EDITMSG .' 
Traceback (most recent call last):
  File "/path/to/hirni-bids/code/hirni-toolbox/procedures/hirni-dicom-converter.py", line 114, in <module>
    "".format(subject),
  File "/root/env/datalad/lib/python3.6/site-packages/wrapt/wrappers.py", line 603, in __call__
    args, kwargs)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/distribution/dataset.py", line 546, in apply_func
    return f(**kwargs)
  File "/root/env/datalad/lib/python3.6/site-packages/wrapt/wrappers.py", line 564, in __call__
    args, kwargs)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 491, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/root/env/datalad/lib/python3.6/site-packages/wrapt/wrappers.py", line 564, in __call__
    args, kwargs)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 479, in return_func
    results = list(results)
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad_container/containers_run.py", line 106, in __call__
    return_type='generator'):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 428, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/utils.py", line 521, in _process_results
    for res in results:
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 216, in __call__
    sidecar=sidecar):
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 617, in run_command
    raise exc
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/interface/run.py", line 385, in _execute_command
    expect_fail=True,
  File "/root/env/datalad/lib/python3.6/site-packages/datalad/cmd.py", line 550, in run
    raise CommandError(text_type(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command 'singularity exec --bind /path/to/hirni-bids code/hirni-toolbox/converters/heudiconv/heudiconv.simg heudiconv -f code/hirni-toolbox/converters/heudiconv/hirni_heuristic.py -s 001 -c dcm2niix -o .git/hirni-tmp-kbpo9at4 -b -a '/path/to/hirni-bids' -l '' --minmeta --files sourcedata/.' failed with exitcode 1
Failed to run "singularity exec --bind /path/to/hirni-bids code/hirni-toolbox/converters/heudiconv/heudiconv.simg heudiconv -f code/hirni-toolbox/converters/heudiconv/hirni_heuristic.py -s 001 -c dcm2niix -o .git/hirni-tmp-kbpo9at4 -b -a '/path/to/hirni-bids' -l '' --minmeta --files sourcedata/." under '/path/to/hirni-bids'. Exit code=1.
[INFO   ] == Command exit (modification check follows) ===== 
CommandError: command 'python /path/to/hirni-bids/code/hirni-toolbox/procedures/hirni-dicom-converter.py /path/to/hirni-bids sourcedata/studyspec3.json True 001 sourcedata/.' failed with exitcode 1
Failed to run 'python /path/to/hirni-bids/code/hirni-toolbox/procedures/hirni-dicom-converter.py /path/to/hirni-bids sourcedata/studyspec3.json True 001 sourcedata/.' under '/path/to/hirni-bids'. Exit code=1.
jbpoline commented 5 years ago

would it be worth looking at what's in the studyspec3.json ?

bpoldrack commented 5 years ago

Not sure, on what dataset(s) you are running those calls and how those datasets look like. I can tell, however, that this is not the structure hirni intends. This is because you apparently have the spec file in the same dataset as the DICOMs themselves. But it's part of the entire idea to have the dicoms in a subdataset, get the metadata and aggregate it in the super dataset, to be able to operate on the metadata without even having the DICOM dataset.

I'd suggest to have video call, so we can walk through. Seems the easiest way to figure it out. I'll probably not be able to make to the regular datalad call on Tue, but may be we can find some time Wed-Fri?

mih commented 5 years ago

Is that resolved?

jbpoline commented 5 years ago

not sure : but I know Ted is actively working on this !

On Sat, Jun 1, 2019 at 4:26 AM Michael Hanke notifications@github.com wrote:

Is that resolved?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/psychoinformatics-de/datalad-hirni/issues/106?email_source=notifications&email_token=AACDE2A26YSRF3AUV3SB6NDPYIXC7A5CNFSM4HNHDWB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWW3XKA#issuecomment-497925032, or mute the thread https://github.com/notifications/unsubscribe-auth/AACDE2HXMX5ZVQ2S2IGYASDPYIXC7ANCNFSM4HNHDWBQ .

ted-strauss-K1 commented 5 years ago

Pretty sure it was determined (through a sigularity debug session) that there was a missing/corrupted dicom frame in the dataset that was causing this error. Closing the issue. Thanks.