search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.26k stars 17.8k forks source link

BUG: "TypeError: Cannot interpret 'string[pyarrow]' as a data type" when reading csv with pyarrow dtypes #51876

Closed weimenglee closed 1 year ago

weimenglee commented 1 year ago

Pandas version checks

Reproducible Example

pd.set_option("mode.dtype_backend", "pyarrow")
df = pd.read_csv('flights.csv', usecols=[7,8], use_nullable_dtypes=True)
df

Issue Description

I am using the 2015 flights delay dataset: Source: https://www.kaggle.com/datasets/usdot/flight-delays.

When loading columns 7 and 8, which contains mixed types (numeric and strings), loading it using pyarrow as the backend gives this error:

TypeError: Cannot interpret 'string[pyarrow]' as a data type

Using Pandas as the backend have no issues (albeit a type warning).

Expected Behavior

It should load the two columns as string type.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 1a2e300170efc08cb509a0b4ff6248f8d55ae777 python : 3.10.6.final.0 python-bits : 64 OS : Darwin OS-release : 21.6.0 Version : Darwin Kernel Version 21.6.0: Mon Dec 19 20:43:09 PST 2022; root:xnu-8020.240.18~2/RELEASE_ARM64_T6000 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8 pandas : 2.0.0rc0 numpy : 1.23.5 pytz : 2022.2.1 dateutil : 2.8.2 setuptools : 65.2.0 pip : 22.2.2 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.0.0 IPython : 8.4.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : 1.3.7 brotli : fastparquet : None fsspec : 2023.3.0 gcsfs : None matplotlib : 3.3.2 numba : 0.56.4 numexpr : 2.8.4 odfpy : None openpyxl : 3.0.10 pandas_gbq : None pyarrow : 9.0.0 pyreadstat : None pyxlsb : None s3fs : 2023.3.0 scipy : 1.9.1 snappy : None sqlalchemy : 1.4.46 tables : None tabulate : None xarray : None xlrd : 2.0.1 zstandard : None tzdata : 2022.7 qtpy : None pyqt5 : None
zmunro commented 1 year ago

Here is a full stack trace for people's reference:

>>> pd.read_csv('flights.csv', usecols=[7,8], use_nullable_dtypes=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/zach/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 918, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 589, in _read
    return parser.read(nrows)
           ^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1721, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 233, in read
    data = _concatenate_chunks(chunks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 387, in _concatenate_chunks
    common_type = np.find_common_type(
                  ^^^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/numpy/core/numerictypes.py", line 649, in find_common_type
    array_types = [dtype(x) for x in array_types]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zach/.local/lib/python3.11/site-packages/numpy/core/numerictypes.py", line 649, in <listcomp>
    array_types = [dtype(x) for x in array_types]
                   ^^^^^^^^
TypeError: Cannot interpret 'string[pyarrow]' as a data type
MarcoGorelli commented 1 year ago

thanks for the report - to expedite resolution, could you make a minimal reproducible example please?

weimenglee commented 1 year ago

Hi! Here is the code:

pd.set_option("mode.dtype_backend", "pyarrow")df = pd.read_csv('flights.csv', usecols=[7,8], use_nullable_dtypes=True)df

Thanks!

On Fri, 10 Mar 2023 at 6:17 PM, Marco Edward Gorelli < @.***> wrote:

thanks for the report - to expedite resolution, could you make a minimal reproducible example please?

— Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/51876#issuecomment-1463582498, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGHP3NAXDJRSNI53RRZF5TW3L5STANCNFSM6AAAAAAVVZWK5I . You are receiving this because you authored the thread.Message ID: @.***>

-- Wei-Meng Lee, Technologist & Founder, Developer Learning Solutions http://www.Learn2Develop.net Tel: (65)-9-692-4065 Twitter: @weimenglee

Author of:

mroeschke commented 1 year ago

Is there a StringIO equivalent reproducer of this behavior? I can't access flights.csv without a Kaggle account

weimenglee commented 1 year ago

Hi: I have a copy of the flights.csv at: https://bit.ly/4048vMR

Hope this helps!

Thanks!

Wei-Meng Lee, Technologist & Founder, Developer Learning Solutions http://www.Learn2Develop.net Tel: (65)-9-692-4065 Twitter: @weimenglee

Author of:

On Tue, Mar 14, 2023 at 3:43 AM Matthew Roeschke @.***> wrote:

Is there a StringIO equivalent reproducer of this behavior? I can't access flights.csv without a Kaggle account

— Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/51876#issuecomment-1466852253, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGHP3MFAJ2T3NRXGJDBL5TW352HDANCNFSM6AAAAAAVVZWK5I . You are receiving this because you were mentioned.Message ID: @.***>

mroeschke commented 1 year ago

Thanks. For now, looks like reading the entire CSV in with the pyarrow engine and selecting the 7th and 8th column seems to work

# dtype_backend is new syntax coming in 2.0rc2
In [6]: pd.read_csv("flights.csv", dtype_backend="pyarrow", engine="pyarrow").iloc[:, [7, 8]]
Out[6]:
        ORIGIN_AIRPORT DESTINATION_AIRPORT
0                  ANC                 SEA
1                  LAX                 PBI
2                  SFO                 CLT
3                  LAX                 MIA
4                  SEA                 ANC
...                ...                 ...
5819074            LAX                 BOS
5819075            JFK                 PSE
5819076            JFK                 SJU
5819077            MCO                 SJU
5819078            JFK                 BQN

[5819079 rows x 2 columns]
phofl commented 1 year ago

This needs a minimal reproducible example, I don't see the error locally on main. Please read https://matthewrocklin.com/minimal-bug-reports

phofl commented 1 year ago

Hm weird now I am getting the error as well, but not all the time. Can you create a reproducible example anyway?