search

unionai-oss / pandera

A light-weight, flexible, and expressive statistical data testing library
https://www.union.ai/pandera
MIT License
3.27k stars 305 forks source link

version 0.14.x breaks mypy compatibility #1121

Closed PetitLepton closed 1 year ago

PetitLepton commented 1 year ago

Describe the bug It seems that versions 0.14.x break the compatibility with mypy.

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

pyproject.toml

[tool.mypy]
plugins = ["pandera.mypy"]

test.py

import pandas
import pandera
pip install mypy pandera[mypy]==0.14.3 && mypy test.py

leads to

[...]/lib/python3.10/site-packages/pandera/api/pandas/model.py:563: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 1.1.1
[...]/lib/python3.10/site-packages/pandera/api/pandas/model.py:563: : note: please use --show-traceback to print a traceback when reporting a bug

but

pip install mypy pandera[mypy]==0.13.4 && mypy test.py

is OK

Success: no issues found in 1 source file

Expected behavior

Consistent behaviour with version 0.13.4.

Desktop (please complete the following information):

cosmicBboy commented 1 year ago

hi @PetitLepton I'm unable to reproduce the error. I'm installing the same versions of mypy, pandas, and pandas-stubs on my virtual env and I get:

❯ mypy test.py
Success: no issues found in 1 source file

Can you provide more details on what OS and Python version you're on?

PetitLepton commented 1 year ago

hi @PetitLepton I'm unable to reproduce the error. I'm installing the same versions of mypy, pandas, and pandas-stubs on my virtual env and I get:

❯ mypy test.py
Success: no issues found in 1 source file

Can you provide more details on what OS and Python version you're on?

Hi @cosmicBboy , thanks for the quick update. I am on Ubuntu 22.10. I used conda/mamba to create the virtual environment as shown above for python 3.10.9. Then pip for the rest.

cosmicBboy commented 1 year ago

You may have to debug this on your side. It looks like the offending line is https://github.com/unionai-oss/pandera/blob/main/pandera/api/pandas/model.py#L563

Can you try noodling around with this to see what happens? Some ideas:

PetitLepton commented 1 year ago

Here is the report

mypy test.py --show-traceback
/home/flavien/.conda/envs/test-mypy/lib/python3.10/site-packages/pandera/api/pandas/model.py:563: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.1.1
Traceback (most recent call last):
  File "mypy/semanal.py", line 6261, in accept
  File "mypy/nodes.py", line 1123, in accept
  File "mypy/semanal.py", line 1561, in visit_class_def
  File "mypy/semanal.py", line 1641, in analyze_class
  File "mypy/semanal.py", line 1668, in analyze_class_body_common
  File "mypy/semanal.py", line 1751, in apply_class_plugin_hooks
  File "/home/flavien/.conda/envs/test-mypy/lib/python3.10/site-packages/pandera/mypy.py", line 93, in _pandera_model_class_maker_callback
    transformer.transform()
  File "/home/flavien/.conda/envs/test-mypy/lib/python3.10/site-packages/pandera/mypy.py", line 125, in transform
    self.erase_field_type_arg()
  File "/home/flavien/.conda/envs/test-mypy/lib/python3.10/site-packages/pandera/mypy.py", line 140, in erase_field_type_arg
    type_ = def_.type
AttributeError: 'ExpressionStmt' object has no attribute 'type'
/home/flavien/.conda/envs/test-mypy/lib/python3.10/site-packages/pandera/api/pandas/model.py:563: : note: use --pdb to drop into pdb
PetitLepton commented 1 year ago

The SchemaModel = DataFrameModel removes the error.

PetitLepton commented 1 year ago

To be "sure," I performed the same set operations in a Docker container using python:3.10-slim-buster and the result is the same.

cosmicBboy commented 1 year ago

are you using any other mypy configuration settings?

PetitLepton commented 1 year ago

Not that I know of. And that should not come into play when using the Docker container if I am not mistaken.

cosmicBboy commented 1 year ago

I noticed you're using pandas-stubs==1.4.3.220807. Any particular reason you're using this version? can you try pandas-stubs==1.5.2.221213

PetitLepton commented 1 year ago

It doesn't seem to change the result.

Here is the Dockerfile that reproduces the issue on my machine.

FROM python:3.10-slim-buster

RUN pip install mypy pandera[mypy]==0.14.3 
WORKDIR /tmp

RUN echo '[tool.mypy]\nplugins = ["pandera.mypy"]' > pyproject.toml 
RUN echo 'import pandera\nimport pandas' > test.py 

CMD ["mypy", "--show-traceback", "/tmp/test.py"]

This

docker build -t test-mypy:0.1 . && docker run -i test-mypy:0.1

gives me the issue.

cosmicBboy commented 1 year ago

https://github.com/unionai-oss/pandera/pull/1123 should address this issue.

I was able to repro this with your dockerfile, but it's concerning that I couldn't repro on my mac 🤷‍♂️. I'd prefer not to be flying in the dark on this, but then again mypy support for pandera is still experimental and it's a bumpy road if you want to use pandas-stubs and pandera mypy type-linting :)

Can you confirm that this works for you?

FROM python:3.10-slim-buster

RUN apt-get update && apt-get install -y git

RUN pip install git+https://github.com/unionai-oss/pandera@bugfix/1121 mypy pandas-stubs==1.5.2.221213
WORKDIR /tmp

RUN echo '[tool.mypy]\nplugins = ["pandera.mypy"]' > pyproject.toml 
RUN echo 'import pandera\nimport pandas' > test.py 

CMD ["mypy", "--show-traceback", "/tmp/test.py"]
PetitLepton commented 1 year ago

@cosmicBboy I am away from laptop right now. Let me confirm that tomorrow morning, Paris time. In the meantime, thanks for the effort and the library.