Open sasan00 opened 5 years ago
Can you write the exact expected output?
I have updated the issue with the requested information. Couldn't find a way to remove the "Needs Info" label.
Thanks. Can you check if the HTML parsing libraries (lxml, bs4) typically convert p tags to newlines? Do they provide options to do that?
That wouldn't help as the below example shows:
import pandas as pd
html = """
<html>
<body>
<table>
<tr>
<td>
Field 1
Field 2
</td>
<td>
Value 1
Value 2
</td>
</tr>
</table>
</body>
</html>
"""
tables = pd.read_html(html)
print(tables[0].iat[0, 0])
Still returns "Field 1 Field 2"
I'm just wondering if our behavior matches the expected behavior of the underlying parsing libraries, and whether they have ways of dealing with it. Presumably they've had requests for similar features around whitespace normalization.
On Thu, Jan 17, 2019 at 10:38 AM sasan00 notifications@github.com wrote:
That wouldn't help as the below example shows:
import pandas as pd
html = """
"""
Field 1 Field 2 Value 1 Value 2 tables = pd.read_html(html)print(tables[0].iat[0, 0])
Still returns "Field 1 Field 2"
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/24766#issuecomment-455240169, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHInwNpzOb1pvbnIjUgP387yVDxBzwks5vEKbvgaJpZM4Z-i-t .
lxml respects whitespaces.
import pandas as pd
from lxml.etree import fromstring
from lxml.html import HTMLParser
html = """
<html>
<body>
<table>
<tr>
<td>
Field 1
Field 2
</td>
<td>
Value 1
Value 2
</td>
</tr>
</table>
</body>
</html>
"""
tables = pd.read_html(html)
print(tables[0].iat[0, 0])
parser = HTMLParser()
root = fromstring(html, parser)
for elem in root.iter('td'):
print(repr(elem.text))
Result:
Field 1 Field 2 '\n Field 1\n \n Field 2\n ' '\n Value 1\n \n Value 2\n '
Thanks. Can you check if pandas explicitly strips / normalizes whitespace in read_html then? If so, this would be a good parameter to add to read_html.
On Thu, Jan 17, 2019 at 10:49 AM sasan00 notifications@github.com wrote:
lxml respects whitespaces.
import pandas as pdfrom lxml.etree import fromstringfrom lxml.html import HTMLParser
html = """
"""
Field 1 Field 2 Value 1 Value 2 tables = pd.read_html(html)print(tables[0].iat[0, 0]) parser = HTMLParser() root = fromstring(html, parser)for elem in root.iter('td'): print(repr(elem.text))
Result:
Field 1 Field 2 '\n Field 1\n \n Field 2\n ' '\n Value 1\n \n Value 2\n '
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pandas-dev/pandas/issues/24766#issuecomment-455244452, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQHIt86UF1iJOARwLdx1oJBZzcjMMWWks5vEKmjgaJpZM4Z-i-t .
Yes. In _parse_raw_data
, calls are made to _remove_whitespace
for each column in each row using the default value of the regex
argument which is _RE_WHITESPACE
whose value is re.compile(r'[\r\n]+|\s{2,}')
.
I think whether whitespaces are "cleaned up" (i.e., replaced with a single space character) should be an optional functionality.
Thanks for investigating. I think an option to disable that behavior makes sense.
You've given two examples now, one with newlines in the text, and one with <p>
tags. Do you expect to normalize the <p>
tags to newlines, so that the two would give the same output? Do we have any prior art to copy here?
I think adding an extra argument as a function that takes the raw text of a cell, and returns the "cleaned up" version would work best. Its default value would be _remove_whitespace
to ensure backwards compatibility.
Hi, wondering if this issue was ever resolved? In my case, I have a <ul>
inside the HTML table and all the elements of each list are squished together after the table is parsed by read_html
.
still encountering this bug.
@Derekt2 this is open you are welcome to submit a pull request to patch
Hi. I'm experiencing the same bug with newlines in cells. I see that there are some existing contributions, will they be merged?
same
take
Code Sample, a copy-pastable example if possible
Problem description
In the current implementation, the p tags are ignored, and therefore it's not possible to infer that field 1 has value 1 and field 2 has value 2.
Expected Output
Output of
pd.show_versions()