How to assign a filename to an output without using parameter completion?

[ylep]

I am trying to implement a very simple use-case, with a pipeline that has:

• one input file input
• one output file output

I want users of my pipeline to be able to specify where the output should be written, i.e.

p = capsul.api.executable("highres_cortex.capsul.pipelines.FilteredSumcurvs")
p.input = os.path.join(self.test_dir, "reference_laplacian.nii.gz")
p.output = os.path.join(self.test_dir, "curvature.nii.gz")
with self.capsul_engine as ce:
    ce.run(p)

... and I cannot find a way to achieve this. Here is the structure of the pipeline I have implemented:

The difficulty stems from the fact that my output parameter should actually be an input of the pipeline from the point of view of the file’s name, but is actually an output from the point of view of the file’s contents. The pipeline exports it as an output parameter, therefore the direction of the sumcurvs.output_image->output link is correct for the contents, but is not followed when I set the output file’s name on the pipeline.

How should we handle such (simple) cases?

For reference, here is the code I used to implement the pipeline:

class FilteredSumcurvs(capsul.api.Pipeline):
    """Compute the filtered sum of principal curvatures of isophote surfaces

    This is equivalent to computing the divergence of the normalized gradient
    of the input scalar field.

    .. note::

      A Gaussian smoothing of width sigma is first applied to the input
      image, in order to limit the appearance of local high curvature values
      (e.g. due to the discontinuity of second-order derivative at the
      borders of the cortex).
    """
    sigma: float = field(default=1.0,
                         doc="standard deviation of the gaussian filter "
                         "[default=1.0 mm]")

    def pipeline_definition(self):
        self.add_process('smoothing',
                         "highres_cortex.capsul.processes.GaussianSmoothing")
        self.add_process('sumcurvs',
                         "highres_cortex.capsul.processes.IsoCurvature",
                         do_not_export=['mode'])
        self.nodes['sumcurvs'].mode = 'sum'

        # Inputs
        self.export_parameter('smoothing', 'input_image', 'input')
        # self.export_parameter('smoothing', 'xsigma', 'sigma')
        self.add_link('sigma->smoothing.xsigma')
        self.add_link('sigma->smoothing.ysigma')
        self.add_link('sigma->smoothing.zsigma')
        self.export_parameter('sumcurvs', 'verbosity')

        self.add_link('smoothing.output_image->sumcurvs.input_image')

        # Outputs
        self.export_parameter('sumcurvs', 'output_image', 'output')

[ylep]

Note: the sumcurvs.output_image parameter is defined as:

    output_image: File = field(
        write=True, extensions=VOLUME_EXTENSIONS,
        doc="output image volume containing the curvature of the isosurfaces "
        "of the input field"
    )

i.e. with write=True, but output=False

[denisri]

File and Directory parameters are more complex than other paramerters (strings, numbers), bcause they are actually 2 things: a file name, and a file content. They are mixed, but are actually different. To distinguish them, we use the metadata output for the file name part, and the metadata write for the file content. Thus, in your case, you should declare the output_image param in IsoCurvature with write=True, but not output=True.

Well, during the time I write this, you have added a message, which says that you have already done so, so I think your pipeline is OK. There must be a bug in the execution infrastructure...

What's ehe excat symptom ? The file is not written ? The output param value is not present any longer after execution ? The IsoCurvature process doesn't get its output_image param during execution ?

[ylep]

It seems that during execution the output_image parameter of sumcurvs in unset:

  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/host/src/capsul/master/capsul/run.py", line 90, in <module>
    process.execute(execution_context)
  File "/casa/host/build/python/highres_cortex/capsul/processes.py", line 205, in execute
    "--output", self.output_image
  File "/casa/host/build/python/soma/controller/controller.py", line 455, in __getattribute__
    raise AttributeError('{} object has no attribute {}'.format(repr(self.__class__), repr(name)))
AttributeError: <class 'highres_cortex.capsul.processes.GaussianSmoothing'> object has no attribute 'output_image'. Did you mean: 'input_image'?
E
======================================================================
ERROR: test_filtered_sumcurvs (__main__.SphereTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/casa/host/build/python/highres_cortex/test/test_capsul.py", line 103, in test_filtered_sumcurvs
    ce.run(p)
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 197, in run
    self.raise_for_status(status)
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 152, in raise_for_status
    raise RuntimeError(f'{error}\n\n{detail}')
RuntimeError: Command '['python', '-m', 'capsul.run', 'process', 'd24f3359-8021-4a78-89e8-a39b16bab428']' returned non-zero exit status 1.

Traceback (most recent call last):
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 269, in <module>
    subprocess.check_call(command, env=env, stdout=sys.stdout,
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '-m', 'capsul.run', 'process', 'd24f3359-8021-4a78-89e8-a39b16bab428']' returned non-zero exit status 1.

[sapetnioc]

Your code seems correct, we need to figure out where the value is lost. Do you see the correct value on output_image in the GUI when you set output on the pipeline ? If yes, debugging will be easier but I expect a no here.

Is it possible to access the complete pipeline in order to test it ?

[ylep]

It seems to be set:

It seems that the error happens in the execution subprocess

[ylep]

I will try to provide the code if you need it, as part of a branch of the highres-cortex toolbox

[sapetnioc]

Yes, I need the code because debugging this part is tricky.

[ylep]

I tried to write a minimal test-case for reproducing this issue, but did not succeed so far :confused: I will try again on Monday

[ylep]

I didn't manage to write a minimal test-case so here is my complete code: https://github.com/neurospin/highres-cortex/commit/c906f563a745a2bf4f092a1815960379aed3b152 (branch WIP_capsul_v3 of highres-cortex).

The failing testcase is:

$ python -m highres_cortex.test.test_capsul SphereTestCase.test_filtered_sumcurvs
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/host/src/capsul/master/capsul/run.py", line 90, in <module>
    process.execute(execution_context)
  File "/casa/host/build/python/highres_cortex/capsul/processes.py", line 208, in execute
    "--output", self.output_image
  File "/casa/host/build/python/soma/controller/controller.py", line 455, in __getattribute__
    raise AttributeError('{} object has no attribute {}'.format(repr(self.__class__), repr(name)))
AttributeError: <class 'highres_cortex.capsul.processes.GaussianSmoothing'> object has no attribute 'output_image'. Did you mean: 'input_image'?
E
======================================================================
ERROR: test_filtered_sumcurvs (__main__.SphereTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/casa/host/build/python/highres_cortex/test/test_capsul.py", line 101, in test_filtered_sumcurvs
    ce.run(p)
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 197, in run
    self.raise_for_status(status)
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 152, in raise_for_status
    raise RuntimeError(f'{error}\n\n{detail}')
RuntimeError: Command '['python', '-m', 'capsul.run', 'process', 'ea23d459-a94c-49a7-be5b-1bb4ef8deffb']' returned non-zero exit status 1.

Traceback (most recent call last):
  File "/casa/host/src/capsul/master/capsul/engine/local.py", line 269, in <module>
    subprocess.check_call(command, env=env, stdout=sys.stdout,
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '-m', 'capsul.run', 'process', 'ea23d459-a94c-49a7-be5b-1bb4ef8deffb']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 1 test in 2.287s

FAILED (errors=1)

[ylep]

I just realized that the “bug” is not the one I thought... I misread the exception message, and was convinced that the second process was missing a value, whereas it was in fact the first process. Sigh. I will open a new issue with my updated question, sorry.