search

nipype / pydra

Pydra Dataflow Engine
https://nipype.github.io/pydra/
Other
119 stars 57 forks source link

WIP: Static checking with mypy #621

Open ghisvail opened 1 year ago

ghisvail commented 1 year ago

Work in progress

See methodology here.

ghisvail commented 1 year ago

First run:

$ python -m mypy pydra
pydra/utils/messenger.py:185: error: Skipping analyzing "pyld": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/helpers.py:5: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/core.py:13: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/core.py:161: error: Argument 1 to "Path" has incompatible type "Union[str, File]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/core.py:162: error: Argument 1 to "Path" has incompatible type "Union[str, File]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/core.py:169: error: Argument after ** must be a mapping, not "Union[str, File, Dict[Any, Any], None]"  [arg-type]
pydra/engine/core.py:174: error: Need type annotation for "inp_lf" (hint: "inp_lf: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:179: error: Need type annotation for "_inner_cont_dim" (hint: "_inner_cont_dim: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:180: error: Need type annotation for "_output" (hint: "_output: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:181: error: Need type annotation for "_result" (hint: "_result: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/core.py:185: error: Incompatible types in assignment (expression has type "Dict[<nothing>, <nothing>]", variable has type "None")  [assignment]
pydra/engine/core.py:903: error: Argument 1 to "_sanitize_input_spec" has incompatible type "Union[List[str], SpecInfo, None]"; expected "Union[SpecInfo, List[str]]"  [arg-type]
pydra/engine/core.py:926: error: Need type annotation for "name2obj" (hint: "name2obj: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/engine/workers.py:860: error: Cannot find implementation or library stub for module named "dask.distributed"  [import]
pydra/engine/task.py:44: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/task.py:141: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, Any]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/task.py:141: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/task.py:141: note: Consider using "Sequence" instead, which is covariant
pydra/engine/task.py:143: error: Item "BaseSpec" of "Union[SpecInfo, BaseSpec]" has no attribute "fields"  [union-attr]
pydra/engine/task.py:184: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, Any]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/task.py:184: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/task.py:184: note: Consider using "Sequence" instead, which is covariant
pydra/engine/task.py:297: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/task.py:594: error: Need type annotation for "bindings" (hint: "bindings: Dict[<type>, <type>] = ...")  [var-annotated]
pydra/__init__.py:26: error: Skipping analyzing "etelemetry": module is installed, but missing library stubs or py.typed marker  [import]
pydra/__init__.py:26: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
pydra/utils/tests/test_messenger.py:52: error: Skipping analyzing "pyld": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/boutiques.py:95: error: Skipping analyzing "boutiques.searcher": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/utils.py:16: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]
pydra/engine/tests/utils.py:164: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/utils.py:164: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/utils.py:197: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:199: error: Incompatible return value type (got "Path", expected "File")  [return-value]
pydra/engine/tests/utils.py:199: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:205: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:207: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:217: error: Argument 1 to "open" has incompatible type "Union[str, File, Path]"; expected "Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]]"  [arg-type]
pydra/engine/tests/utils.py:219: error: Argument 1 to "Path" has incompatible type "Union[str, File, Path]"; expected "Union[str, PathLike[str]]"  [arg-type]
pydra/engine/tests/utils.py:221: error: Incompatible types in assignment (expression has type "int", target has type "Path")  [assignment]
pydra/engine/tests/utils.py:227: error: No overload variant of "open" matches argument type "File"  [call-overload]
pydra/engine/tests/utils.py:227: note: Possible overload variants:
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['r+', '+r', 'rt+', 'r+t', '+rt', 'tr+', 't+r', '+tr', 'w+', '+w', 'wt+', 'w+t', '+wt', 'tw+', 't+w', '+tw', 'a+', '+a', 'at+', 'a+t', '+at', 'ta+', 't+a', '+ta', 'x+', '+x', 'xt+', 'x+t', '+xt', 'tx+', 't+x', '+tx'], Literal['w', 'wt', 'tw', 'a', 'at', 'ta', 'x', 'xt', 'tx'], Literal['r', 'rt', 'tr', 'U', 'rU', 'Ur', 'rtU', 'rUt', 'Urt', 'trU', 'tUr', 'Utr']] = ..., buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> TextIOWrapper
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: Literal[0], encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> FileIO
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedRandom
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedWriter
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedReader
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: int = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BinaryIO
pydra/engine/tests/utils.py:227: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: str, buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> IO[Any]
pydra/engine/tests/utils.py:236: error: No overload variant of "open" matches argument type "File"  [call-overload]
pydra/engine/tests/utils.py:236: note: Possible overload variants:
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['r+', '+r', 'rt+', 'r+t', '+rt', 'tr+', 't+r', '+tr', 'w+', '+w', 'wt+', 'w+t', '+wt', 'tw+', 't+w', '+tw', 'a+', '+a', 'at+', 'a+t', '+at', 'ta+', 't+a', '+ta', 'x+', '+x', 'xt+', 'x+t', '+xt', 'tx+', 't+x', '+tx'], Literal['w', 'wt', 'tw', 'a', 'at', 'ta', 'x', 'xt', 'tx'], Literal['r', 'rt', 'tr', 'U', 'rU', 'Ur', 'rtU', 'rUt', 'Urt', 'trU', 'tUr', 'Utr']] = ..., buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> TextIOWrapper
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: Literal[0], encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> FileIO
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedRandom
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedWriter
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], buffering: Literal[-1, 1] = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BufferedReader
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: Union[Literal['rb+', 'r+b', '+rb', 'br+', 'b+r', '+br', 'wb+', 'w+b', '+wb', 'bw+', 'b+w', '+bw', 'ab+', 'a+b', '+ab', 'ba+', 'b+a', '+ba', 'xb+', 'x+b', '+xb', 'bx+', 'b+x', '+bx'], Literal['rb', 'br', 'rbU', 'rUb', 'Urb', 'brU', 'bUr', 'Ubr'], Literal['wb', 'bw', 'ab', 'ba', 'xb', 'bx']], buffering: int = ..., encoding: None = ..., errors: None = ..., newline: None = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> BinaryIO
pydra/engine/tests/utils.py:236: note:     def open(file: Union[int, Union[str, bytes, PathLike[str], PathLike[bytes]]], mode: str, buffering: int = ..., encoding: Optional[str] = ..., errors: Optional[str] = ..., newline: Optional[str] = ..., closefd: bool = ..., opener: Optional[Any] = ...) -> IO[Any]
pydra/engine/tests/test_tasks_files.py:20: error: No overload variant of "listdir" matches argument type "Directory"  [call-overload]
pydra/engine/tests/test_tasks_files.py:20: note: Possible overload variants:
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: Optional[Union[str, PathLike[str]]] = ...) -> List[str]
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: Union[bytes, PathLike[bytes]]) -> List[bytes]
pydra/engine/tests/test_tasks_files.py:20: note:     def listdir(path: int) -> List[str]
pydra/engine/tests/test_tasks_files.py:45: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/test_tasks_files.py:45: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/test_tasks_files.py:46: error: Argument 1 to "load" has incompatible type "File"; expected "Union[str, bytes, PathLike[Any], _SupportsReadSeek[bytes]]"  [arg-type]
pydra/engine/tests/test_tasks_files.py:56: error: Invalid type comment or annotation  [valid-type]
pydra/engine/tests/test_tasks_files.py:56: note: Suggestion: use ty.NamedTuple[...] instead of ty.NamedTuple(...)
pydra/engine/tests/test_tasks_files.py:57: error: Argument 1 to "load" has incompatible type "File"; expected "Union[str, bytes, PathLike[Any], _SupportsReadSeek[bytes]]"  [arg-type]
pydra/engine/tests/test_singularity.py:13: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]
pydra/engine/tests/test_profiles.py:6: error: Skipping analyzing "pympler": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_helpers_file.py:258: error: Need type annotation for "MOUNT_OUTPUTS"  [var-annotated]
pydra/engine/tests/test_task.py:5: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_task.py:326: error: No overload variant of "sum" matches argument type "Union[float, int]"  [call-overload]
pydra/engine/tests/test_task.py:326: note: Possible overload variants:
pydra/engine/tests/test_task.py:326: note:     def sum(Iterable[bool], /, start: int = ...) -> int
pydra/engine/tests/test_task.py:326: note:     def [_SupportsSumNoDefaultT <: _SupportsSumWithNoDefaultGiven] sum(Iterable[_SupportsSumNoDefaultT], /) -> Union[_SupportsSumNoDefaultT, Literal[0]]
pydra/engine/tests/test_task.py:326: note:     def [_AddableT1 <: SupportsAdd[Any, Any], _AddableT2 <: SupportsAdd[Any, Any]] sum(Iterable[_AddableT1], /, start: _AddableT2) -> Union[_AddableT1, _AddableT2]
pydra/engine/tests/test_task.py:339: error: Argument 1 to "sum" has incompatible type "Union[str, int]"; expected "Iterable[bool]"  [arg-type]
pydra/engine/tests/test_task.py:379: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_task.py:394: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_task.py:410: error: Argument 1 to "len" has incompatible type "MultiInputObj"; expected "Sized"  [arg-type]
pydra/engine/tests/test_shelltask_inputspec.py:2172: error: Argument "fields" to "SpecInfo" has incompatible type "List[Tuple[str, type, Dict[str, object]]]"; expected "List[Tuple[Any, ...]]"  [arg-type]
pydra/engine/tests/test_shelltask_inputspec.py:2172: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
pydra/engine/tests/test_shelltask_inputspec.py:2172: note: Consider using "Sequence" instead, which is covariant
pydra/engine/tests/test_shelltask_inputspec.py:2173: error: Need type annotation for "task_output_fields" (hint: "task_output_fields: List[<type>] = ...")  [var-annotated]
pydra/engine/tests/test_shelltask_inputspec.py:2175: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/tests/test_nipype1_convert.py:18: error: Argument "bases" to "SpecInfo" has incompatible type "Tuple[Type[ShellOutSpec]]"; expected "Sequence[Type[BaseSpec]]"  [arg-type]
pydra/engine/tests/test_helpers.py:8: error: Skipping analyzing "cloudpickle": module is installed, but missing library stubs or py.typed marker  [import]
pydra/engine/tests/test_boutiques.py:15: error: Argument 1 to "__call__" of "_SkipifMarkDecorator" has incompatible type "int"; expected "Union[str, bool]"  [arg-type]
codecov[bot] commented 1 year ago

Codecov Report

Base: 81.17% // Head: 80.49% // Decreases project coverage by -0.69% :warning:

Coverage data is based on head (9bcd0dc) compared to base (b5fe4c0). Patch has no changes to coverable lines.

:exclamation: Current head 9bcd0dc differs from pull request most recent head d4dc196. Consider uploading reports for the commit d4dc196 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #621 +/- ## ========================================== - Coverage 81.17% 80.49% -0.69% ========================================== Files 20 20 Lines 4393 4393 Branches 1264 0 -1264 ========================================== - Hits 3566 3536 -30 - Misses 823 857 +34 + Partials 4 0 -4 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `80.49% <ø> (-0.69%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [pydra/engine/task.py](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-cHlkcmEvZW5naW5lL3Rhc2sucHk=) | `88.07% <0.00%> (-5.51%)` | :arrow_down: | | [pydra/engine/helpers.py](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-cHlkcmEvZW5naW5lL2hlbHBlcnMucHk=) | `84.39% <0.00%> (-1.66%)` | :arrow_down: | | [pydra/engine/helpers\_file.py](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-cHlkcmEvZW5naW5lL2hlbHBlcnNfZmlsZS5weQ==) | `84.89% <0.00%> (-1.21%)` | :arrow_down: | | [pydra/engine/core.py](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-cHlkcmEvZW5naW5lL2NvcmUucHk=) | `92.96% <0.00%> (-0.16%)` | :arrow_down: | | [pydra/engine/workers.py](https://codecov.io/gh/nipype/pydra/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-cHlkcmEvZW5naW5lL3dvcmtlcnMucHk=) | `18.76% <0.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.