BUG: `read_csv()` with `engine="pyarrow"` converts numeric string even when `dtype=str` is specified

LawrenceLiu023 commented 5 months ago

Pandas version checks

[ ] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

demo_df: pd.DataFrame = pd.DataFrame(
    {
        "sample_name": ["sample_1", "sample_2"],
        "case_id": ["00001", "00002"],
        "sample_id": ["00001001", "00001002"],
        "sample_type": ["T", "T"],
    }
)
demo_df.to_csv("bug_demo.csv", header=True, index=False, sep=",", encoding="utf-8")
pyarrow_df: pd.DataFrame = pd.read_csv(
    "bug_demo.csv",
    header=0,
    index_col=None,
    sep=",",
    encoding="utf-8",
    engine="pyarrow",
    dtype=str,
)
print("This is read by pyarrow engine.")
print(pyarrow_df)
c_df: pd.DataFrame = pd.read_csv(
    "bug_demo.csv",
    header=0,
    index_col=None,
    sep=",",
    encoding="utf-8",
    engine="c",
    dtype=str,
)
print("This is read by c engine.")
print(c_df)
python_df: pd.DataFrame = pd.read_csv(
    "bug_demo.csv",
    header=0,
    index_col=None,
    sep=",",
    encoding="utf-8",
    engine="python",
    dtype=str,
)
print("This is read by python engine.")
print(python_df)

Issue Description

I have encountered an issue with the read_csv() function in pandas when using the pyarrow engine. Even when specifying dtype=str, pure numeric strings are being converted to numeric type. Additionally, pure numeric strings starting with multiple zeros lose the leading zeros in the resulting DataFrame. This behavior is unexpected as I would like to preserve the original format of the numeric strings as text.

Expected Behavior

The example demonstrates reading a CSV file with different engines. It is expected that pyarrow engine should get the same DataFrame as c engine and python engine when using dtype=str. It should output the following texts.

# This is read by pyarrow engine.
  sample_name case_id sample_id sample_type
0    sample_1       1      1001           T
1    sample_2       2      1002           T
# This is read by c engine.
  sample_name case_id sample_id sample_type
0    sample_1   00001  00001001           T
1    sample_2   00002  00001002           T
# This is read by python engine.
  sample_name case_id sample_id sample_type
0    sample_1   00001  00001001           T
1    sample_2   00002  00001002           T

Installed Versions

INSTALLED VERSIONS ------------------ commit : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140 python : 3.12.2.final.0 python-bits : 64 OS : Linux OS-release : 6.5.0-26-generic Version : #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : None LOCALE : en_US.UTF-8 pandas : 2.2.2 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0 setuptools : 69.5.1 pip : 24.0 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 5.1.0 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.3 IPython : 8.22.2 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : 1.3.8 dataframe-api-compat : None fastparquet : None fsspec : None gcsfs : None matplotlib : None numba : None numexpr : 2.9.0 odfpy : None openpyxl : None pandas_gbq : None pyarrow : 11.0.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.12.0 sqlalchemy : None tables : None tabulate : 0.9.0 xarray : None xlrd : None zstandard : 0.22.0 tzdata : 2024.1 qtpy : None pyqt5 : None

rohanjain101 commented 5 months ago

eshaready commented 5 months ago

take

mroeschke commented 2 months ago

Closing as a duplicate of https://github.com/pandas-dev/pandas/issues/57666

pandas-dev / pandas