fix chardet support in Python 3

[GoogleCodeExporter]

feedparser-5.1.3 introduced 3515 new test errors with Python 3 when chardet is 
installed for Python 3.

When chardet is not installed for Python 3.2, then feedparser-5.1.2 has 762 
errors with Python 3.2 (total number of tests: 4367), while feedparser-5.1.3 
has 765 errors with Python 3.2 (total number of tests: 4377).

When chardet is installed for Python 3.2, then feedparser-5.1.2 has 773 errors 
with Python 3.2 (total number of tests: 4375), while feedparser-5.1.3 has 4291 
errors with Python 3.2 (total number of tests: 4385).

All new test errors seem to be identical. Example:

======================================================================
ERROR: test_lxml_etree_bug (__main__.TestLxmlBug)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "feedparsertest.py", line 689, in test_lxml_etree_bug
    feedparser.parse(feedparser._StringIO(doc))
  File "../build/lib/feedparser.py", line 3966, in parse
    data, result['encoding'], error = convert_to_utf8(http_headers, data)
  File "../build/lib/feedparser.py", line 3768, in convert_to_utf8
    chardet_encoding = str(chardet.detect(data)['encoding'] or '', 'ascii', 'ignore')
TypeError: decoding str is not supported

----------------------------------------------------------------------

Original issue reported on code.google.com by Arfrever...@gmail.com on 17 Dec 2012 at 5:29

[GoogleCodeExporter]

Confirmed, 5.1.2 works perfectly for me, 5.1.3 throws the mentioned exception.

Original comment by TomWallr...@gmail.com on 17 Dec 2012 at 4:13

[GoogleCodeExporter]

This issue was closed by revision 20a32910f4a5.

Original comment by kurtmckee on 17 Dec 2012 at 4:40

• Changed state: Fixed

[GoogleCodeExporter]

Fixed. Thanks!

Original comment by kurtmckee on 17 Dec 2012 at 4:42

• Changed title: fix chardet support in Python 3

[GoogleCodeExporter]

I'm getting the same errors in year 2014. Feedparser 5.1.3. still seems to be 
the version published on PyPI. Any plans to release a new version with this fix?

Original comment by kermit666 on 6 Feb 2014 at 9:19

[GoogleCodeExporter]

Just installed 5.1.3, running python 3.2. Got the same error
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    d = feedparser.parse('http://www.reddit.com/r/python/.rss')
  File "C:\Python32\lib\site-packages\feedparser-5.1.3-py3.2.egg\feedparser.py", line 3966, in parse
    data, result['encoding'], error = convert_to_utf8(http_headers, data)
  File "C:\Python32\lib\site-packages\feedparser-5.1.3-py3.2.egg\feedparser.py", line 3768, in convert_to_utf8
    chardet_encoding = str(chardet.detect(data)['encoding'] or '', 'ascii', 'ignore')
TypeError: decoding str is not supported

Original comment by HotGrin...@gmail.com on 30 Apr 2014 at 12:01

[GoogleCodeExporter]

Are you sure that this is fixed?

INFO 2014-05-07T21:28:11 supybot load called in private by "Mikaela!mkaysi@unaffiliated/mikaela". ERROR 2014-05-07T21:28:11 supybot Uncaught exception in ['load']. Traceback (most recent call last): File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/plugins/RSS/plugi n.py", line 292, in getFeed results = feedparser.parse(url) File "/home/users/mkaysi/.pyenv/versions/3.3.5/lib/python3.3/site-packages/feedparser .py", line 3966, in parse data, result['encoding'], error = convert_to_utf8(http_headers, data) File "/home/users/mkaysi/.pyenv/versions/3.3.5/lib/python3.3/site-packages/feedparser .py", line 3768, in convert_to_utf8 chardet_encoding = str(chardet.detect(data)['encoding'] or '', 'ascii', 'ignore') TypeError: decoding str is not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/callbacks.py", line 1266, in _callCommand self.callCommand(command, irc, msg, _args, _kwargs) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/utils/python.py", line 91, in g f(self, _args, _kwargs) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/callbacks.py", line 1247, in callCommand method(irc, msg, _args, _kwargs) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/commands.py", line 1076, in newf f(self, irc, msg, args, _state.args, _state.kwargs) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/plugins/Owner/plu gin.py", line 445, in load cb = plugin.loadPluginClass(irc, module) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/plugin.py", line 87, in loadPluginClass cb = module.Class(irc) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/utils/python.py", line 103, in init originalinit(self, _args, *_kwargs) File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/plugins/RSS/plugi n.py", line 80, in init self.getFeed(url) # So announced feeds don't announce on startup. File "/home/users/mkaysi/.local/lib/python3.3/site-packages/supybot/plugins/RSS/plugi n.py", line 295, in getFeed except feedparser.sgmllib.SGMLParseError: AttributeError: type object 'sgmllib' has no attribute 'SGMLParseError' ERROR 2014-05-07T21:28:11 supybot Exception id: 0xa4e05

This output is from Limnoria. My pip freeze:

Django==1.6.4 SQLAlchemy==0.9.4 beautifulsoup4==4.3.2 charade==1.0.3 chardet==2.2.1 feedparser==5.1.3 limnoria==2014.05.07 lxml==3.3.5 oauthlib==0.6.1 pysandbox==1.5.1 python-dateutil==2.2 python-gnupg==0.3.6 python-twitter==1.3.1 pytz==2014.2 requests==2.2.1 requests-oauthlib==0.4.0 simplejson==3.4.1 six==1.6.1

Original comment by S.Mik...@gmail.com on 7 May 2014 at 6:47

[GoogleCodeExporter]

It is fixed. You need feedparser trunk or 5.1.4, not 5.1.3.

Original comment by Arfrever...@gmail.com on 7 May 2014 at 9:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

pip3 install feedparser==5.1.4 doesn't work
(err: could not find a version that satisfies the requirement feedparser==5.1.4)
seems to stop after 5.1.3
am i missing something ?

cheers

Original comment by eric.del...@aequamcapital.com on 25 Jun 2014 at 3:04

[GoogleCodeExporter]

Learn how to use `git clone`.

Original comment by Arfrever...@gmail.com on 25 Jun 2014 at 3:42

[GoogleCodeExporter]

> Learn how to use `git clone`.

If git is installed, Pip can install without git clone.

When will be 5.1.4 be send to pypi? This issue has soon been open for two years 
without being fixed in pypi where most of people are probably installing this 
and I don't think any distribution has 5.1.4.

Original comment by S.Mik...@gmail.com on 25 Jun 2014 at 3:59

[GoogleCodeExporter]

>> Learn how to use `git clone`.

>If git is installed, Pip can install without git clone.

pip install git+https://code.google.com/p/feedparser/

Original comment by S.Mik...@gmail.com on 25 Jun 2014 at 4:00

[GoogleCodeExporter]

It seems that the git version identifies as 5.1.3 too.

Original comment by S.Mik...@gmail.com on 25 Jun 2014 at 4:03

[GoogleCodeExporter]

anyway like it or not, pip is by far the mainstream way to install for python 
ecosystem now, agree with S.Mik(...) pypi release should be up-to-date.

Original comment by eric.del...@aequamcapital.com on 25 Jun 2014 at 7:22

[GoogleCodeExporter]

I think its not fixed. Where can i download feedparser 5.1.4? 

Original comment by andre.sc...@googlemail.com on 3 Jul 2014 at 5:36

[GoogleCodeExporter]

5.1.4 wasn't released yet. install feedparser from git.

Original comment by raf...@rafaelmartins.eng.br on 3 Jul 2014 at 6:05

[GoogleCodeExporter]

I don't have any experience with git.
Is this the rigth repository? https://github.com/Unode/python-feedparser

Original comment by andre.sc...@googlemail.com on 3 Jul 2014 at 6:21

[GoogleCodeExporter]

No. Go to "Source" tab at the top of this page.

Original comment by Arfrever...@gmail.com on 3 Jul 2014 at 7:34

[GoogleCodeExporter]

Did anyone ever read comments #12 and #13 ? If not, I would like to correct #12 
and repeat #13:

#12: As root: 
pip install git+https://code.google.com/p/feedparser/ --upgrade

With sudo:
sudo pip install git+https://code.google.com/p/feedparser/ --upgrade

Without root:
pip install git+https://code.google.com/p/feedparser/ --upgrade --user

And #13: When I checked the git version called itself as feedparser 5.1.3 and I 
have the same issue using it. I am now checking again and I am using that 
command which I told earlier and checking pip freeze.

% pip install git+https://code.google.com/p/feedparser/ --upgrade --user

Downloading/unpacking git+https://code.google.com/p/feedparser/ Cloning https://code.google.com/p/feedparser/ to /tmp/pip-oldfsf-build Running setup.py (path:/tmp/pip-oldfsf-build/setup.py) egg_info for package from git+https://code.google.com/p/feedparser/

Installing collected packages: feedparser Found existing installation: feedparser 5.1.3 Uninstalling feedparser: Successfully uninstalled feedparser Running setup.py install for feedparser Fixing build/lib/feedparser.py Skipping implicit fixer: buffer Skipping implicit fixer: idioms Skipping implicit fixer: set_literal Skipping implicit fixer: ws_comma Fixing build/lib/feedparser.py Skipping implicit fixer: buffer Skipping implicit fixer: idioms Skipping implicit fixer: set_literal Skipping implicit fixer: ws_comma

Successfully installed feedparser Cleaning up... mkaysi@lakka ~ % mkaysi@lakka ~ % pip freeze|grep feedparser feedparser==5.1.3

Original comment by S.Mik...@gmail.com on 4 Jul 2014 at 10:56

[GoogleCodeExporter]

Also, why is this in status "closed" before stable release which fixes this is 
released?

Original comment by S.Mik...@gmail.com on 4 Jul 2014 at 10:57

[GoogleCodeExporter]

Hey guys, sorry for the delay in response.

I believe this is indeed fixed but haven't tested in a while. The bug was 
auto-closed by Google Code's commit-sniffing feature -- I mentioned in the 
commit message that the bug was fixed and Google Code auto-closed the report 
(see comment #2).

I have a lot of catching up to do with bug reports and pull requests but for 
the time being please continue to use git master.

Original comment by kurtmckee on 10 Jul 2014 at 4:52

[GoogleCodeExporter]

Issue 403 has been merged into this issue.

Original comment by kurtmckee on 10 Jul 2014 at 4:53

[GoogleCodeExporter]

Issue 405 has been merged into this issue.

Original comment by kurtmckee on 10 Jul 2014 at 4:54

[GoogleCodeExporter]

Hey guys, Is this issue fixed?
After upgrade feedParser with the command:
    sudo pip install git+https://code.google.com/p/feedparser/ --upgrade
Error still exists:
TypeError: decoding str is not supported.

Original comment by liuchaoz...@gmail.com on 5 Sep 2014 at 7:29

[GoogleCodeExporter]

#24: Are you using Python 2 or 3? If you're using Python 3 but Python 2 is 
still the default version on your system then you might be accidentally 
upgrading the Python 2 installation of feedparser. Try the following:

    sudo pip3 install git+https://code.google.com/p/feedparser/ --upgrade

note the use of pip3 instead of pip. That fixed the issue for me.

Original comment by jswad...@gmail.com on 16 Nov 2014 at 12:58