Make a Node put its data in the workflow directories

[TheChymera]

Following my misfortunes with trying to extend the Dcmstack interface, I am setting out to wrap a function of my own in place of Dcmstack's parse_and_stack.

This has worked nicely (I am defining the function here, and using it via a Node here).

How can I make the stacker Node put the data it generates into the preprocessing directories (like the realigner node is doing)?

[oesteban]

Nodes set their directory as cwd when run, so results should be written in that directory by default. Were are placed results of parse_and_stack? Can you modify the output directory?

[TheChymera]

I am not using parse_and_stack, as it is not picklable (see my first link). I am using a separate function called dcm_to_nii seen in my second link.

This function uses the dcmstack.DcmStack() interface and passes it an output path formatted based on the input path. I can modify the output directory manually, yes, but how can I make my Node automatically use the workflow directory?

set their directory as cwd when run

what does that mean?

[oesteban]

When a node is run, the current working directory (cwd) is changed to the folder corresponding to that node

[TheChymera]

you mean it is changed to workflow.base_dir, yes? So how do I configure the node to use workflow.base_dir/name to store the files it generates?

[oesteban]

Ok, so you are using a function called parse_and_stack_wrapper right?

Inside it you use dcmstack.parse_and_stack isn't it?

You probably want dcmstack.parse_and_stack to save results directly on the corresponding path. Inside parse_and_stack_wrapper you can call os.getcwd() and pass it to dcmstack.parse_and_stack to be used as output directory.

As nodes automatically change the current dir when run, os.getcwd() will return the directory of the node.

[TheChymera]

I am sorry, it appears my second link was mis-pasted. No, parse_and_stack_wrapper, and anything else with the name parse_and_stack is a failed attempt to make this function nipype-Node compatible.

I am trying to work around that by using this function, which I use via a Node.

But I guess the same thing you said applies to my own case as well, except, I would like the function to only behave like that when it's wrapped a a node - how can I check for this inside a function?

[satra]

in the interface you can use copyfiles=False as metadata to symlink the corresponding input files to the local directory when the interface is run as a node. then in your function if you test the path of the input, it should leave you in the working directory.

[satra]

for copyfiles to work you have to create a proper nipype interface. with a function node. all you can do is check if the input file path is different from the current working directory and then it's your function's call how to deal with it.

[TheChymera]

in the interface you can use copyfiles=False as metadata to symlink the corresponding input files to the local directory when the interface is run as a node.

I am not looking to create symlinks, I am looking to let the function put its data in the directory it was putting it in if it is called as a function, and in os.getcwd() if it is called as a Node. How can I do that?

in the interface you can use copyfiles=False as metadata to symlink the corresponding input files to the local directory when the interface is run as a node.

So what I am using now is not a "proper" nipype interface? Why not?

[oesteban]

No, it's not a proper interface. This is an example of a simple interface for a pure python script: http://nipy.sourceforge.net/nipype/devel/python_interface_devel.html

[TheChymera]

@oesteban so the actual python script being wrapped in that examle is _run_interface(self, runtime) - yes?

Can I, in the body of that function just a function from another file (so that I still get to keep my function separate from the nipype wrapper?).

[oesteban]

Exactly, that's the idea: inside _run_interface you call parse_and_stack.

Whatever you define in the InputSpec class will be available there as self.inputs.my_parameter_name.

[TheChymera]

@oesteban ok, (I think) that part worked. But what can I do about the output?

My interface currently looks like this.

But whatever I try I seem to not be able to get the output file names in _list_outputs

[oesteban]

Here https://github.com/TheChymera/chyMRI/blob/75088e51ec4a306578f42f049d2c61d61005dbe1/extra_interfaces.py#L30 you use self.result that has not been set elsewhere.

You can set it in _run_interface, typically dcm_to_nii would return you the list of files or you compute it explicitly.

Some other interfaces implement a private function to generate names that will be used when run.

[TheChymera]

@oesteban how do I set it? I tried to just replace https://github.com/TheChymera/chyMRI/blob/75088e51ec4a306578f42f049d2c61d61005dbe1/extra_interfaces.py#L23 with

result = dcm_to_nii(dcm_dir, group_by, node=True)

But that apparently does not set it.

[oesteban]

Well, it depends in how you are designing your calls. I meant that typically dcm_to_nii should return the list of paths. Then you would get it doing self.result = dcm_to_nii ...

Of course, that's assuming that dcm_to_nii returns this info. Otherwise, you'll need to find out how to compose the list of actual files written and set it in self.result.

[TheChymera]

@oesteban yes, dcm_to_nii does return a list of the files it creates (as tested witha print call in the function. This revision of my files returns the following when I run preprocessing.py:

IOError: [Errno 2] No such file or directory: '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/_0x289dd62a997c2c42797640b02bb20fc1_unfinished.json'

150520-17:28:44,653 workflow INFO:
     ***********************************
150520-17:28:44,653 workflow ERROR:
     could not run node: preproc.dcm_to_nii
150520-17:28:44,653 workflow INFO:
     crashfile: /home/chymera/src/chyMRI/crash-20150520-172744-chymera-dcm_to_nii.pklz
150520-17:28:44,653 workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "/home/chymera/src/chyMRI/preprocessing.py", line 45, in <module>
    preproc_workflow("/home/chymera/data/dc.rs/export_ME/dicom/4459/1/EPI/", workflow_base="/home/chymera/data/dc.rs/export_ME/")
  File "/home/chymera/src/chyMRI/preprocessing.py", line 42, in preproc_workflow
    workflow.run(plugin="MultiProc")
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 700, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 269, in run
    report_nodes_not_run(notrun)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 92, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details
['/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI23.nii.gz']
['/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI23.nii.gz', '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI11.nii.gz']
['/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI23.nii.gz', '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI11.nii.gz', '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI17.nii.gz']
{'nii_files': <undefined>}
['/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI23.nii.gz', '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI11.nii.gz', '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/EPI17.nii.gz']
[Finished in 782.897s]

Why would the file '/home/chymera/data/dc.rs/export_ME/preproc/dcm_to_nii/_0x289dd62a997c2c42797640b02bb20fc1_unfinished.json' be needed?

[mwaskom]

It might be easier to debug by just importing the interface and calling the .run() method, rather than trying to iterate over it in the context of a workflow.

[TheChymera]

@mwaskom which interface do you mean, specifically?

[mwaskom]

The one you are trying to write.

[TheChymera]

@mwaskom ok, I did it. What can I learn based on this?

>>> from extra_interfaces import DcmToNii
>>> DcmToNii.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unbound method run() must be called with DcmToNii instance as first argument (got nothing instead)
>>> d2n = DcmToNii()
>>> d2n.run
<bound method DcmToNii.run of <extra_interfaces.DcmToNii object at 0x7f03902fd310>>
>>> d2n.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 970, in run
    self._check_mandatory_inputs()
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 908, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: DcmToNii requires a value for input 'dcm_dir'. For a list of required inputs, see DcmToNii.help()
>>> d2n.inputs.dcm_dir = "/home/chymera/data/dc.rs/export_ME/dicom/4459/1/EPI/"
>>> d2n.inputs.group_by = "EchoTime"
>>> d2n.run()
['/home/chymera/src/chyMRI/EPI23.nii.gz']
['/home/chymera/src/chyMRI/EPI23.nii.gz', '/home/chymera/src/chyMRI/EPI11.nii.gz']
['/home/chymera/src/chyMRI/EPI23.nii.gz', '/home/chymera/src/chyMRI/EPI11.nii.gz', '/home/chymera/src/chyMRI/EPI17.nii.gz']
{'nii_files': <undefined>}
['/home/chymera/src/chyMRI/EPI23.nii.gz', '/home/chymera/src/chyMRI/EPI11.nii.gz', '/home/chymera/src/chyMRI/EPI17.nii.gz']
<nipype.interfaces.base.InterfaceResult object at 0x7f0390315d50>

It looks like everything is working nicely, no? The interface returns a list with my files, yes?

[mwaskom]

I don't know enough about the function you're trying to wrap to be more helpful than that, sorry, but hopefully that will let you iterate a little bit more quickly and transparently.

[TheChymera]

What do you mean by iterate? I am not using this function in a loop.

[satra]

@TheChymera - you may want to read this page - even though it's for command line tools.

http://nipy.org/nipype/devel/cmd_interface_devel.html

what @mwaskom is suggesting is to check that the interface functions and returns the correct results.

in you output above, it looks like you are printing things in your interface.

if you do:

results = d2n.run()
print(results.outputs)

that is what should be correct. from the looks of it, the outputs are never being set properly.

[TheChymera]

@satra

I removed the internal print calls, but other than that the output looks ok to me, is it not?

>>> from extra_interfaces import DcmToNii
>>> d2n = DcmToNii()
>>>  d2n.inputs.dcm_dir = "/home/chymera/data/dc.rs/export_ME/dicom/4459/1/EPI/"
  File "<stdin>", line 1
    d2n.inputs.dcm_dir = "/home/chymera/data/dc.rs/export_ME/dicom/4459/1/EPI/"
    ^
IndentationError: unexpected indent
>>> d2n.inputs.dcm_dir = "/home/chymera/data/dc.rs/export_ME/dicom/4459/1/EPI/"
>>> d2n.inputs.group_by = "EcoTime"
>>> d2n.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 1003, in run
    runtime = self._run_interface(runtime)
  File "extra_interfaces.py", line 23, in _run_interface
    self.result = dcm_to_nii(dcm_dir, group_by, node=True)
  File "functions_preprocessing.py", line 21, in dcm_to_nii
    echo_times += [float(meta[group_by])]
KeyError: 'EcoTime\nInterface DcmToNii failed to run. '
>>> d2n.inputs.group_by = "EchoTime"
>>> d2n.run()
<nipype.interfaces.base.InterfaceResult object at 0x7f8e4c592e10>
>>> results = d2n.run()
>>> print(results.outputs)

nii_files = ['/home/chymera/src/chyMRI/EPI23.nii.gz', '/home/chymera/src/chyMRI/EPI11.nii.gz', '/home/chymera/src/chyMRI/EPI17.nii.gz']

[satra]

yes the outputs look reasonable.

[TheChymera]

Many thanks guys for guiding me along the way, my basic interface is now working as of this commit. I think the original question is way past answered, and if I will need further advice I will ask in a more appropriate issue.