BUG: rowspan in read_html failed

[duongkstn]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# html string
s = '<table>\n<caption>Đơn vị tính: Việt Nam đồng</caption>\n<tr>\n<th rowspan="2">Tổng giới hạn bồi thường / năm / người</th>\n<th>Chương trình 1</th>\n<th>Chương trình 2</th>\n<th>Chương trình 3</th>\n</tr>\n<tr>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Tử vong do tai nạn</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Thương tật toàn bộ vĩnh viễn/ năm</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Thương tật bộ phận vĩnh viễn/ năm</td>\n<td>50,000,000</td>\n<td>100,000,000</td>\n<td>200,000,000</td>\n</tr>\n<tr>\n<td>Chi phí y tế điều trị thương tật do tai nạn/ năm</td>\n<td>10,000,000</td>\n<td>20,000,000</td>\n<td>30,000,000</td>\n</tr>\n</table>'

print(pd.read_html(s))

Issue Description

I have a HTML string (with rowspan attribute), here is how it look when rendering

Then I convert to dataframe by using pd.read_html command (pandas 2.2.3), here is how it looks:

Totally wrong, the row of 50000000, 100000000 and 200000000 should be aligned to the right.

Expected Behavior

the row of 50000000, 100000000 and 200000000 should be aligned to the right

Installed Versions

INSTALLED VERSIONS

commit : 0691c5cf90477d3503834d983f69350f250a6ff7 python : 3.10.9 python-bits : 64 OS : Linux OS-release : 5.15.0-124-generic Version : #134~20.04.1-Ubuntu SMP Tue Oct 1 15:27:33 UTC 2024 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.3 numpy : 1.26.4 pytz : 2022.7 dateutil : 2.8.2 pip : 24.3.1 Cython : None sphinx : 5.0.2 IPython : 8.10.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None blosc : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : 2024.3.1 html5lib : None hypothesis : None gcsfs : None jinja2 : 3.1.2 lxml.etree : 4.9.1 matplotlib : None numba : N/A numexpr : 2.8.4 odfpy : None openpyxl : 3.1.3 pandas_gbq : None psycopg2 : 2.9.9 pymysql : None pyarrow : 16.1.0 pyreadstat : None pytest : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.10.1 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : 2.0.1 xlsxwriter : 3.2.0 zstandard : 0.19.0 tzdata : 2024.1 qtpy : None pyqt5 : None

[simonjayhawkins]

Thanks @duongkstn for the report.

For the reproducible example, please follow this guide on how to provide a minimal, copy-pastable example.

I understand this requires additional effort, but it significantly increases the likelihood that a member of the volunteer community will address the issue.

Some preliminary thoughts:

Does the behavior persist without the caption tags?

Does the issue occur with just two rows of data or two columns of data?

Can this be reproduced with regular text?

Additionally, it is worth noting that a warning is issued about passing literal HTML to read_html. In version 3.0.0 (and currently on the development branch), this code sample will result in a FileNotFoundError.

It's possible that some reported issues have already been resolved. Therefore, I suggest updating the code sample to ensure it runs on the latest development version.

If this issue is indeed a bug in pandas and has not been previously reported, any PR with a fix will also need a test. Creating a Minimal Complete Verifiable Example that works on 'main' can help streamline this process, as the test will already be written.

[rhshadrach]

Minimal example:

s = '<table><tr><th rowspan="2">A</th><th>B</th></tr><tr><td>1</td></tr><tr><td>C</td><td>2</td></tr></table>'
buf = io.StringIO(s)
print(pd.read_html(buf)[0])
#    A                  B
#    A Unnamed: 1_level_1
# 0  1                NaN

# Expected:
#    A  B
# 0  A  1
# 1  C  2

Switching the top row from th to td produces the expected results (except that the column labels are now the first row, as they should be). It appears pandas does not handle rowspan="2" correctly in a th.

Browsers do handle this case in the expected manner:

[simonjayhawkins]

Thanks @rhshadrach.

Your expected output is consistent with the given string, but the rendered HTML and actual output aren't quite right.

To confirm, the behavior is the same on the main branch and also back in version 1.5.3.

This seems like a corner case where we need to explicitly define the expected behavior. I understand why we get the current output and could potentially justify it if needed. Conversely, how would you justify that your expected output is correct? The HTML has a header spanning two rows, so why would you expect to get back a DataFrame without a multilevel Index?

[rhshadrach]

The HTML has a header spanning two rows, so why would you expect to get back a DataFrame without a multilevel Index?

In the browser-rendered output, the value A is shared in the top and 2nd row, while the 2nd row is not bold. I take that as an indication that the value "A" belongs to the 2nd row as well as the first, and that the 2nd row is not a header.

[duongkstn]

any updates :) ?

[rhshadrach]

@duongkstn - PRs to fix this issue are welcome.