pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.79k stars 17.98k forks source link

BUG: skiprows ignored in read_csv(engine="pyarrow") #58048

Open rohanjain101 opened 7 months ago

rohanjain101 commented 7 months ago

Pandas version checks

Reproducible Example

>>> data = """a\nb\nc"""
>>> pd.read_csv(StringIO(data), engine="pyarrow")
   a
0  b
1  c
>>> pd.read_csv(StringIO(data), engine="pyarrow", skiprows=1)
   a
0  b
1  c
>>>

Issue Description

skiprows seems to be ignored

Expected Behavior

The first row should be skipped, and the returned df should contain same data as default csv engine.

Installed Versions

INSTALLED VERSIONS ------------------ commit : bdc79c146c2e32f2cab629be240f01658cfb6cc2 python : 3.11.8.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.22631 machine : AMD64 processor : Intel64 Family 6 Model 151 Stepping 2, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_United States.1252 pandas : 2.2.1 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.8.2 setuptools : 69.0.3 pip : 23.3.2 Cython : None pytest : None hypothesis : None sphinx : 4.2.0 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.3 IPython : 8.21.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : 2024.2.0 gcsfs : None matplotlib : 3.8.3 numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 15.0.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.12.0 sqlalchemy : None tables : None tabulate : 0.9.0 xarray : None xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None >>>
ashorkey-umich commented 7 months ago

take

kymcglyn commented 7 months ago

take

rohanjain101 commented 7 months ago

From looking at the code, it seems skiprows is only honored if header=None for pyarrow types, but this is different from the default csv engine.