nipype / pydra

Pydra Dataflow Engine
https://nipype.github.io/pydra/
Other
120 stars 59 forks source link

`shell_task` decorator #655

Closed tclose closed 1 year ago

tclose commented 1 year ago

Types of changes

Summary

Extends PR #653 to include @shell_task decorator, which streamlines the code required to define a shell command task. See discussion https://github.com/nipype/pydra/discussions/647 for rationale behind this approach

Checklist

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 96.33% and project coverage change: +0.30 :tada:

Comparison is base (426564e) 81.77% compared to head (ac3c436) 82.07%.

:exclamation: Current head ac3c436 differs from pull request most recent head 1b3f72e. Consider uploading reports for the commit 1b3f72e to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #655 +/- ## ========================================== + Coverage 81.77% 82.07% +0.30% ========================================== Files 20 21 +1 Lines 4400 4508 +108 Branches 1264 0 -1264 ========================================== + Hits 3598 3700 +102 - Misses 798 808 +10 + Partials 4 0 -4 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `82.07% <96.33%> (+0.30%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nipype#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://app.codecov.io/gh/nipype/pydra/pull/655?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nipype) | Coverage Δ | | |---|---|---| | [pydra/mark/shell.py](https://app.codecov.io/gh/nipype/pydra/pull/655?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nipype#diff-cHlkcmEvbWFyay9zaGVsbC5weQ==) | `96.26% <96.26%> (ø)` | | | [pydra/mark/\_\_init\_\_.py](https://app.codecov.io/gh/nipype/pydra/pull/655?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nipype#diff-cHlkcmEvbWFyay9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | | ... and [4 files with indirect coverage changes](https://app.codecov.io/gh/nipype/pydra/pull/655/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nipype)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

tclose commented 1 year ago

I have now implemented the @shell_task decorator, which can be used as per the discussion thread like


# Split out common arguments into separate attrs classes
@attrs.define(kw_only=True, slots=False)
class RecurseOption:
    recurse: bool = shell_arg(
        help_string="Recursively descend the directory tree",
        argstr="-R",
    )

# Define an interface for the `ls` command
@shell_task
class Ls:
    executable = "ls"

    class Inputs(RecurseOption):
        directory: os.PathLike = shell_arg(
            help_string="the directory to list the contents of",
            argstr="",
            mandatory=True,
            position=-1,
        )
        hidden: bool = shell_arg(
            help_string="display hidden FS objects",
            argstr="-a",
            default=False,
        )

    class Outputs:
        entries: list = shell_out(
            help_string="list of entries returned by ls command",
            callable=list_entries,
        )

or alternatively as a function in order to dynamically generate interfaces

Ls = shell_task(
    "Ls",
    executable="ls",
    input_fields={
        "directory": {
            "type": os.PathLike,
            "help_string": "the directory to list the contents of",
            "argstr": "",
            "mandatory": True,
            "position": -1,
        },
        "hidden": {
            "type": bool,
            "help_string": "display hidden FS objects",
            "argstr": "-a",
        },
    output_fields={
        "entries": {
            "type": list,
            "help_string": "list of entries returned by ls command",
            "callable": list_entries,
        }
    },
    inputs_bases=[RecursiveOptions],
)

The decorator/function automatically generates the input_spec and output_spec SpecInfor class attributes, but I think they could be redundant now, as the attrs-generated classes for inputs and outputs are generated at MyTask.Inputs and MyTask.Outputs, respectively, with output_file_template generated outputs automatically inserted.

If this syntax looks alright I can update the docs to match it.

If there is interest, I could also look at refreshing the function task decorator so that it can be used in the same way for dynamic classes and so it also generates Inputs/Outputs classes so we can do without input_spec and output_spec all together.

tclose commented 1 year ago

closing to see where the #692 discussion goes first