Open dscience7 opened 6 years ago
Hmm interesting. Looks like this is still an issue on master even specifying the encoding to be used:
>>> pd.read_html('https://en.wikipedia.org/wiki/2013–14_Premier_League', encoding='utf-8')
UnicodeEncodeError: 'ascii' codec can't encode character '\u2013' in position 14: ordinal not in range(128)
Investigation and PRs are always welcome
https://stackoverflow.com/questions/39229439/encoding-error-when-reading-url-with-urllib
As seen in this similar issue, urllib only works with ASCII requests. To remedy, I used the Requests library (http://docs.python-requests.org/en/master/).
FWIW the sample call works fine under Python 2.7.15 but not Python 3.6.5. Choice of engine doesn't matter, however.
I used the following solution:
import requests url = "https://ru.wikipedia.org/wiki/Города_России_с_населением_более_500_тысяч_человек" r = requests.get(url, auth=('user', 'pass')) website = r.text
import pandas as pd tables = pd.read_html( website, encoding="UTF-8")
City_pop = tables[4]
@StepanSushko 's solution works for me.
I investigate this error, and I personally think it should be fixed. Let me take it and send PR. In urllib's library, all URL characters should be ASCII, so what you need is just pass the escaped URL if you have multibyte. also, urllib is the Python's standard library.
import urllib
import pandas
url = "https://en.wikipedia.org/wiki/2013–14_Premier_League"
url_escaped = urllib.parse.quote_plus(url, "/:?=&") # need this line
print(f"url:{url}")
print(f"escaped:{url_escaped}")
tables = pd.read_html(url_escaped, encoding="UTF-8") # non-error
tables = pd.read_html(url, encoding="UTF-8") # error
$ url:https://en.wikipedia.org/wiki/2013–14_Premier_League
$ escaped:https://en.wikipedia.org/wiki/2013%E2%80%9314_Premier_League
url = urllib.parse.quote_plus(url, "/:?=&") # need this line
tables = pd.read_html(url, encoding="UTF-8") # non-error
so, we can just use urllib instead of requests.
I'm gonna try to implement the Unicode handling feature, but because of Python's RFC design, This is treated as the spec rather than the bug.
details are written here. https://github.com/pandas-dev/pandas/pull/50259
I am trying to read the tables from a Wikipedia page using the following code:
Doing that generates the following error:
I have tried
But still get the same error. The following works:
What I want to know is how to get pd.read_html() to work directly on the url without requests. What is it that I don't understand about encoding or is this a problem with Pandas?
I am running an Anaconda distribution of Pandas 0.21.1 and Python 3.5.4. Thanks for any help.