pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.71k stars 17.93k forks source link

Missing Imports #28133

Open simonjayhawkins opened 5 years ago

simonjayhawkins commented 5 years ago

in setup.cfg we currently have

[mypy]
ignore_missing_imports=True

mypy states "We recommend using this approach only as a last resort: it’s equivalent to adding a # type: ignore to all unresolved imports in your codebase." https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports

the number of instances of missing imports resolving to Any is already significant..

$ mypy pandas --disallow-any-unimported
pandas\util\_test_decorators.py:106: error: Return type becomes "Any" due to an unfollowed import
pandas\util\_test_decorators.py:120: error: Return type becomes "Any" due to an unfollowed import
pandas\_typing.py:19: error: Constraint 5 becomes "Any" due to an unfollowed import
pandas\_typing.py:22: error: Constraint 2 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 1 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 2 becomes "Any" due to an unfollowed import
pandas\_typing.py:23: error: Constraint 3 becomes "Any" due to an unfollowed import
pandas\io\formats\printing.py:252: error: Base type BaseFormatter becomes "Any" due to an unfollowed import
pandas\core\dtypes\dtypes.py:653: error: Type of variable becomes "Type[Any]" due to an unfollowed import
pandas\core\dtypes\dtypes.py:812: error: Type of variable becomes "Type[Any]" due to an unfollowed import
pandas\core\dtypes\common.py:144: error: Return type becomes "Any" due to an unfollowed import
pandas\core\dtypes\common.py:144: error: Argument 1 to "ensure_int_or_float" becomes "Any" due to an unfollowed import
pandas\core\dtypes\common.py:181: error: Argument 1 to "ensure_python_int" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\indexers.py:39: error: Argument 2 to "is_empty_indexer" becomes "Any" due to an unfollowed import
pandas\core\indexers.py:115: error: Argument 1 to "validate_indices" becomes "Any" due to an unfollowed import
pandas\core\dtypes\cast.py:765: error: Argument 1 to "maybe_convert_objects" becomes "Any" due to an unfollowed import
pandas\core\dtypes\cast.py:814: error: Argument 1 to "soft_convert_objects" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:194: error: Return type becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:194: error: Argument 1 to "_maybe_get_mask" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:194: error: Argument 3 to "_maybe_get_mask" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:237: error: Return type becomes "Tuple[Any, Optional[Any], Any, Any, Any]" due to an unfollowed import
pandas\core\nanops.py:237: error: Argument 1 to "_get_values" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:237: error: Argument 5 to "_get_values" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:634: error: Return type becomes "Tuple[Union[int, Any], Union[int, Any]]" due to an unfollowed import
pandas\core\nanops.py:634: error: Argument 2 to "_get_counts_nanvar" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:672: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1117: error: Return type becomes "Union[Any, int]" due to an unfollowed import
pandas\core\nanops.py:1117: error: Argument 1 to "_maybe_arg_null_out" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1117: error: Argument 3 to "_maybe_arg_null_out" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:1141: error: Return type becomes "Union[int, Any]" due to an unfollowed import
pandas\core\nanops.py:1141: error: Argument 2 to "_get_counts" becomes "Optional[Any]" due to an unfollowed import
pandas\core\nanops.py:1185: error: Return type becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1185: error: Argument 1 to "_maybe_null_out" becomes "Any" due to an unfollowed import
pandas\core\nanops.py:1185: error: Argument 3 to "_maybe_null_out" becomes "Optional[Any]" due to an unfollowed import
pandas\core\construction.py:51: error: Return type becomes "Any" due to an unfollowed import
pandas\core\construction.py:51: error: Argument 2 to "array" becomes "Union[str, Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\tools\datetimes.py:71: error: Argument 1 to "should_cache" becomes "Union[List[Any], Tuple[Any, ...], Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:161: error: Return type becomes "Any" due to an unfollowed import
pandas\core\tools\datetimes.py:161: error: Argument 1 to "_box_as_indexlike" becomes "Any" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Return type becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Argument 1 to "_convert_and_box_cache" becomes "Union[Any, List[Any], Tuple[Any, ...], Any, Any]" due to an unfollowed import
pandas\core\tools\datetimes.py:191: error: Argument 2 to "_convert_and_box_cache" becomes "Any" due to an unfollowed import
pandas\tseries\offsets.py:157: error: Base type BaseOffset becomes "Any" due to an unfollowed import
pandas\tseries\offsets.py:2513: error: Base type _Tick becomes "Any" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Return type becomes "Tuple[Any, str]" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Argument 1 to "generate_regular_range" becomes "Any" due to an unfollowed import
pandas\core\arrays\_ranges.py:15: error: Argument 2 to "generate_regular_range" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:398: error: Argument 1 to "should_extension_dispatch" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:1152: error: Argument 1 to "maybe_dispatch_ufunc_to_dunder_op" becomes "Any" due to an unfollowed import
pandas\core\ops\__init__.py:1152: error: Argument 4 to "maybe_dispatch_ufunc_to_dunder_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:230: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:230: error: Argument 2 to "_from_sequence" becomes "Sequence[Optional[Any]]" due to an unfollowed import
pandas\core\arrays\period.py:301: error: Argument 2 to "_unbox_scalar" becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\arrays\period.py:313: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:655: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\period.py:655: error: Argument 2 to "_addsub_int_array" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\period.py:833: error: Argument 1 to "period_array" becomes "Sequence[Optional[Any]]" due to an unfollowed import
pandas\core\arrays\datetimes.py:330: error: Type of variable becomes "Union[Any, DatetimeTZDtype]" due to an unfollowed import
pandas\core\arrays\datetimes.py:604: error: Return type becomes "Union[Any, DatetimeTZDtype]" due to an unfollowed import
pandas\core\arrays\base.py:267: error: Argument 2 to "__setitem__" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\arrays\base.py:392: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:414: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:525: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:637: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:661: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:764: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:855: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:865: error: Return type becomes "Union[Any, Any]" due to an unfollowed import
pandas\core\arrays\base.py:936: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:956: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\base.py:956: error: Argument 2 to "_concat_same_type" becomes "Sequence[Any]" due to an unfollowed import
pandas\core\arrays\base.py:980: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\numpy_.py:89: error: Base type NDArrayOperatorsMixin becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:58: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:65: error: Return type becomes "Type[Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:74: error: Return type becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:96: error: Argument 2 to "_unbox_scalar" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:115: error: Argument 2 to "_check_compatible_with" becomes "Union[Any, Any, Any, Any]" due to an unfollowed import
pandas\core\arrays\datetimelike.py:348: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\datetimelike.py:453: error: Argument 3 to "__setitem__" becomes "Union[Any, Any, Sequence[Any]]" due to an unfollowed import
pandas\core\base.py:990: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\frozen.py:123: error: Base type ndarray becomes "Any" due to an unfollowed import
pandas\core\indexes\base.py:3871: error: Return type becomes "Union[ExtensionArray, Any, Any]" due to an unfollowed import
pandas\core\arrays\sparse.py:109: error: Argument 2 to "__init__" becomes "Union[str, Any, ExtensionDtype]" due to an unfollowed import
pandas\core\arrays\sparse.py:393: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:393: error: Argument 1 to "_get_fill" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:417: error: Argument 1 to "_sparse_array_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:417: error: Argument 2 to "_sparse_array_op" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Argument 2 to "_simple_new" becomes "Any" due to an unfollowed import
pandas\core\arrays\sparse.py:706: error: Argument 3 to "_simple_new" becomes "Any" due to an unfollowed import
pandas\core\arrays\interval.py:143: error: Base type IntervalMixin becomes "Any" due to an unfollowed import
pandas\core\arrays\integer.py:519: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\integer.py:576: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:479: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:502: error: Argument 2 to "astype" becomes "Union[str, Any, ExtensionDtype]" due to an unfollowed import
pandas\core\arrays\categorical.py:502: error: Return type becomes "Any" due to an unfollowed import
pandas\core\arrays\categorical.py:2652: error: Argument 1 to "_recode_for_categories" becomes "Any" due to an unfollowed import
pandas\core\internals\blocks.py:1858: error: Argument 3 to "shift" becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:64: error: Base type BaseMultiIndexCodesEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:64: error: Base type UInt64Engine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:103: error: Base type BaseMultiIndexCodesEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\multi.py:103: error: Base type ObjectEngine becomes "Any" due to an unfollowed import
pandas\core\indexes\category.py:525: error: Argument 2 to "get_value" becomes "Any" due to an unfollowed import
pandas\core\indexes\range.py:79: error: Type of variable becomes "Any" due to an unfollowed import
pandas\core\indexes\range.py:363: error: Argument 2 to "__contains__" becomes "Union[int, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:201: error: Base type IntervalMixin becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:629: error: Argument 2 to "_can_reindex" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:790: error: Return type becomes "Union[int, slice, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:861: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:861: error: Argument 2 to "get_indexer" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:931: error: Return type becomes "Tuple[Any, Any]" due to an unfollowed import
pandas\core\indexes\interval.py:931: error: Argument 2 to "get_indexer_non_unique" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:972: error: Return type becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:972: error: Argument 2 to "get_indexer_for" becomes "Any" due to an unfollowed import
pandas\core\indexes\interval.py:989: error: Argument 2 to "get_value" becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:123: error: Base type ConversionInterface becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:150: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:193: error: Base type DateConverter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:252: error: Base type DateConverter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:325: error: Base type AutoDateFormatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:330: error: Base type AutoDateLocator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:355: error: Base type DateLocator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:936: error: Base type Locator becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:1030: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\plotting\_matplotlib\converter.py:1098: error: Base type Formatter becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1294: error: Return type becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1400: error: Argument 2 to "__init__" becomes "Union[Any, Series, DatetimeIndex, DatetimeArray]" due to an unfollowed import
pandas\io\formats\format.py:1457: error: Argument 1 to "format_percentiles" becomes "Union[Any, List[Union[int, float]], List[float], List[Union[str, float]]]" due to an unfollowed import
pandas\io\formats\format.py:1527: error: Argument 1 to "_is_dates_only" becomes "Union[Any, DatetimeArray, Index, DatetimeIndex]" due to an unfollowed import
pandas\io\formats\format.py:1548: error: Argument 1 to "_format_datetime64" becomes "Union[Any, Any]" due to an unfollowed import
pandas\io\formats\format.py:1548: error: Argument 2 to "_format_datetime64" becomes "Union[Any, tzutc, None]" due to an unfollowed import
pandas\io\formats\format.py:1565: error: Argument 1 to "_format_datetime64_dateonly" becomes "Union[Any, Any]" due to an unfollowed import
pandas\io\formats\format.py:1592: error: Argument 1 to "_get_format_datetime64_from_values" becomes "Union[Any, DatetimeArray, DatetimeIndex]" due to an unfollowed import
pandas\io\formats\format.py:1623: error: Argument 2 to "__init__" becomes "Union[Any, TimedeltaIndex]" due to an unfollowed import
pandas\io\formats\format.py:1641: error: Argument 1 to "_get_format_timedelta64" becomes "Union[Any, TimedeltaIndex, TimedeltaArray]" due to an unfollowed import
pandas\io\formats\format.py:1719: error: Argument 1 to "_trim_zeros_complex" becomes "Any" due to an unfollowed import
pandas\io\formats\format.py:1730: error: Argument 1 to "_trim_zeros_float" becomes "Union[Any, List[str]]" due to an unfollowed import
pandas\io\formats\format.py:1872: error: Argument 1 to "_binify" becomes "List[Any]" due to an unfollowed import
pandas\io\formats\format.py:1872: error: Argument 2 to "_binify" becomes "Union[Any, int]" due to an unfollowed import
pandas\core\indexing.py:102: error: Base type _NDFrameIndexerBase becomes "Any" due to an unfollowed import
pandas\core\indexing.py:650: error: Argument 3 to "_align_series" becomes "Any" due to an unfollowed import
pandas\core\indexing.py:742: error: Argument 3 to "_align_frame" becomes "Any" due to an unfollowed import
pandas\core\indexing.py:2162: error: Base type _NDFrameIndexerBase becomes "Any" due to an unfollowed import
pandas\core\indexing.py:2371: error: Return type becomes "Any" due to an unfollowed import
pandas\core\internals\managers.py:1874: error: Return type becomes "Union[Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\generic.py:164: error: Argument 5 to "__init__" becomes "Union[str, Any, ExtensionDtype, None]" due to an unfollowed import
pandas\core\frame.py:6535: error: Return type becomes "Union[Series, Any]" due to an unfollowed import
pandas\core\frame.py:6535: error: Argument 4 to "_gotitem" becomes "Union[Series, Any, None]" due to an unfollowed import
pandas\core\groupby\groupby.py:1087: error: Return type becomes "Tuple[Any, Type[Any]]" due to an unfollowed import
pandas\core\groupby\groupby.py:1087: error: Argument 1 to "objs_to_bool" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1095: error: Return type becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1095: error: Argument 1 to "result_to_bool" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1879: error: Return type becomes "Tuple[Any, Optional[Type[Any]]]" due to an unfollowed import
pandas\core\groupby\groupby.py:1879: error: Argument 1 to "pre_processor" becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1894: error: Return type becomes "Any" due to an unfollowed import
pandas\core\groupby\groupby.py:1894: error: Argument 1 to "post_processor" becomes "Any" due to an unfollowed import
pandas\io\packers.py:800: error: Base type _Packer becomes "Any" due to an unfollowed import
pandas\io\packers.py:820: error: Base type _Unpacker becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:209: error: Return type becomes "Optional[Any]" due to an unfollowed import
pandas\core\window\rolling.py:222: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:222: error: Argument 2 to "_prep_values" becomes "Optional[Any]" due to an unfollowed import
pandas\core\window\rolling.py:335: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:352: error: Argument 4 to "_get_roll_func" becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:788: error: Return type becomes "Any" due to an unfollowed import
pandas\core\window\rolling.py:844: error: Argument 4 to "_get_roll_func" becomes "Any" due to an unfollowed import

Ideas for keeping this manageable?

cc @WillAyd

WillAyd commented 5 years ago

Yea I started along this path in #26645 . The hang up on that was some of the internal imports which (maybe?) require stubs. Never went far beyond that so if you want to dive deeper I think still worth exploring

simonjayhawkins commented 5 years ago

I think we should create stubs for pyx files.

It would be nice if this could be done automatically (@mrocklin wrote http://matthewrocklin.com/blog/work/2017/11/05/cython-everywhere which mentioned use of cython in regular .py files instead of creating .pyx files)

if we create the stubs manually, i suspect there would not be an issue keeping them in sync, since changes to the code for .pyx file changes would likely cause mypy to report an error if the stub wasn't updated as well.

WillAyd commented 5 years ago

Cool yea I agree with everything you've mentioned above. Article is certainly interesting as well - not sure what limitations there are to that approach but worth exploring if you've got time and interest I think

rhshadrach commented 2 days ago

I believe we will never be able to set ignore_missing_imports=False unless all dependencies (optional or not) publish stubs or add a py.typed marker. If this happens - great, but otherwise, I think there is nothing we can do here. Marking as a closing candidate.