pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
42.59k stars 17.57k forks source link

BUG: Pandas does not validate some parameters properly when reading CSVs and it causes segmentation faults #59059

Closed BergLucas closed 5 days ago

BergLucas commented 1 week ago

Pandas version checks

Reproducible Example

#######################################################################
# Example 1
#######################################################################

import pandas
import io

string_i_o_0 = io.StringIO('"To98odN"\n-40542\n-5\n-4\n')

pandas.read_csv(string_i_o_0, encoding_errors=string_i_o_0)

#######################################################################
# Example 2
#######################################################################

import pandas
import io

string_i_o_0 = io.StringIO('"To98odN"\n-40542\n-5\n-4\n')
pandas.read_table(string_i_o_0, encoding_errors=string_i_o_0)

#######################################################################
# Example 3
#######################################################################

import io
import pandas

string_i_o_0 = io.StringIO('"H45"\n6\n-1\n8\n')
var_0 = pandas.read_csv(string_i_o_0)
var_1 = pandas.read_csv(
    string_i_o_0,
    skipinitialspace=var_0,
)

Issue Description

I was working on Pynguin, an automated unit test generation tool for Python, and the tool found that Pandas does not validate some parameters properly and the generated tests produce the following output:

Segmentation fault (core dumped)

First, there is the encoding_errors parameter in the pandas.read_csv and pandas.read_table functions. I don't know why it happens for this one.

Second, there is the skipinitialspace parameter in the pandas.read_csv function. I think that this one comes from the fact that Pandas does not support that an exception is raised in the __bool__ method of the var_0 dataframe.

Traceback (most recent call last):
  File "/home/lucas/Documents/GitHub/pynguin-for-ML-libraries/err13.py", line 7, in <module>
    print(bool(var_0))
  File "/home/lucas/.conda/envs/pynguin-for-ML-libraries/lib/python3.10/site-packages/pandas/core/generic.py", line 1495, in __nonzero__
    raise ValueError(
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Expected Behavior

Pandas should raise an exception instead of causing a segmentation fault when a wrong value is passed as an argument.

Installed Versions

commit : c46fb76afaf98153b9eef97fc9bbe9077229e7cd python : 3.10.14.final.0 python-bits : 64 OS : Linux OS-release : 6.8.11-200.fc39.x86_64 Version : #1 SMP PREEMPT_DYNAMIC Sun May 26 20:05:41 UTC 2024 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+1125.gc46fb76afa numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 68.2.2 pip : 23.3.1 Cython : None pytest : 8.2.0 hypothesis : 6.103.2 sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.3 IPython : None pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None bottleneck : None fastparquet : None fsspec : 2024.2.0 gcsfs : None matplotlib : 3.8.4 numba : None numexpr : None odfpy : None openpyxl : None pyarrow : 10.0.1 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.13.0 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None