search

nipype / pydra

Pydra Dataflow Engine
https://nipype.github.io/pydra/
Other
120 stars 59 forks source link

Dissociate task cmdline from validation #654

Open ghisvail opened 1 year ago

ghisvail commented 1 year ago

As part of the process of designing task packages, I find myself providing some doctest of style:

"""
Examples
--------

>>> task = MyTask(...)
>>> task.cmdline
'cmd ...'
"""

which allow quick verification that the task specifications and definition produce the corresponding commandline that would have been run in a shell.

However, if there are fields in the input specifications which are of type pydra.engine.specs.File, then the doctest fails with a validation error if the files don't exist. Because of this, the doctest setup needs to be cluttered with setup and teardown mechanisms to provide empty tempfiles.

I wonder whether the file validation logic could be pushed further down the task runtime to facilitate simple commandline tests through this pattern? So far, I have been bypassing it using os.PathLike, but the latter does not work with copyfile. Also, with the recent discussions to normalize file and directory fields to os.PathLike, I have got a feeling this will not work any longer and all my packages CI will start failing at a later Pydra release.

ghisvail commented 1 year ago

cc @effigies who's involved in the File and Directory refactoring

tclose commented 1 year ago

Ah, I have been advocating pushing the validation logic pushed higher up (to task/workflow construction time), and to include file format detection, which would make things more complicated.

Although as long as conftest puts generic files in the working dir for each of the main format types of the form, test-nifti.nii.gz, test-dicom, test-matlab.mat, etc... it wouldn't be much extra effort would it?