pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.91k stars 18.03k forks source link

ENH: read_html(): large colspan values cause unreasonable memory allocations #55036

Open guseggert opened 1 year ago

guseggert commented 1 year ago

Pandas version checks

Reproducible Example

import pandas as pd
pd.read_html('<table><tr><td>a</td></tr><tr><td colspan="1000000000">b</td></tr></table>')

Issue Description

I am parsing dataframes from varied untrusted HTML sources. Occasionally I'll encounter an HTML table that has a large colspan such as in the example, which causes pandas to allocate memory until the process OOMs.

Related: https://github.com/pandas-dev/pandas/issues/17054

Expected Behavior

At the very least, it would be nice to have a column limit to prevent pandas from crashing the program on some non-sensical HTML such as this.

It's possible of course to parse the HTML first and look for colspan attributes that are too big, but this is expensive since the HTML is parsed twice, since I can't pass a parsed tree into read_html(), and it also requires deep knowledge of how pandas is parsing the HTML.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : 3.10.12.final.0 python-bits : 64 OS : Linux OS-release : 6.4.11-200.fc38.x86_64 Version : #1 SMP PREEMPT_DYNAMIC Wed Aug 16 17:42:12 UTC 2023 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 1.5.3 numpy : 1.25.2 pytz : 2023.3 dateutil : 2.8.2 setuptools : 68.0.0 pip : 23.2.1 Cython : None pytest : 7.4.0 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.3 html5lib : 1.1 pymysql : None psycopg2 : 2.9.6 jinja2 : 3.1.2 IPython : 8.14.0 pandas_datareader: None bs4 : 4.12.2 bottleneck : None brotli : None fastparquet : None fsspec : 2023.9.0 gcsfs : 2023.9.0 matplotlib : 3.7.2 numba : None numexpr : None odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : 12.0.1 pyreadstat : None pyxlsb : 1.0.10 s3fs : 2023.9.0 scipy : 1.11.1 snappy : None sqlalchemy : None tables : None tabulate : 0.9.0 xarray : None xlrd : 2.0.1 xlwt : None zstandard : None tzdata : None
guseggert commented 1 year ago

related: https://github.com/pandas-dev/pandas/issues/17054