TwitterStream.user() fails on production server, works in local development environment

[lmaloney]

OK, this is a hard one.... when I've gotten this error before, it typically meant I lost the connection to twitter, and a quick restart fixes everything, but not anymore...

Here is the stack trace:

auth = twitter.Oauth( ... keys )

Code:

from twitter import * stream = twitter.stream.TwitterStream(auth=auth, domain='userstream.twitter.com') stream.user()

Trace:

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/twitter/api.py", line 308, in call return self._handle_response(req, uri, arg_data, _timeout) File "/usr/local/lib/python2.7/dist-packages/twitter/stream.py", line 284, in _handle_response _timeout or timeout, heartbeat_timeout) File "/usr/local/lib/python2.7/dist-packages/twitter/stream.py", line 209, in handle_stream_response raise TwitterHTTPError(e, uri, 'json', arg_data) File "/usr/local/lib/python2.7/dist-packages/twitter/api.py", line 72, in init data = json.loads(data.decode('utf8')) File "/usr/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

=========

This code is from my program, which works just FINE in my local development environment.

Local Dev =

Ubuntu 14 Python 2.7 WingIDE VirtualEnvirnment with depenedencies installed.

However, on my server, hosted on EC2 it fails:

Ubuntu 14 Python 2.7 Virtual Environment Run from command line with python manage.py shell , then load program with execfile('mybot.py')

The same code has been working, but stopped suddenly with the above JSON error.

If I run the same code in DEV on my local system, it works, however running on production server, it fails.

If compared the environments between both systems, and they SEEM to be the same...

On the production server I CAN do OTHER streaming functionality like SENDING DM's to my twitter account.... So SENDING works, but listening with stream.user() FAILS with this JSON error.

I would appreciate ANY suggestions. :)

Thank you,

PS: Love this tool, I think it's better than the other Python Twitter options out there.

[RouxRC]

Hi, can you check with pip freeze what version of the library and its dependencies you have in both environments?

[lmaloney]

I'm doing that now.... one thing that seems to stand out is urllib3... urllib3 is installed on my server, but NOT on my localenvironment...

Here is my working local dev:

=== \ Note: Not all of these are used... ChatterBot==0.2.0 CherryPy==3.6.0 Cython==0.22 Django==1.6.5 Flask==0.10.1 Jinja2==2.7.3 MarkupSafe==0.23 MySQL-python==1.2.5 PyYAML==3.10 WebOb==1.4.1 Werkzeug==0.10.4 argparse==1.2.1 cchardet==0.3.5 chardet==2.3.0 django-dashing==0.2.5 django-unfriendly==0.3 django-uuidfield==0.5.0 dnspython==1.11.1 dnsq==1.1.3 expiringdict==1.1.3 flanker==0.4.28 fuzzywuzzy==0.5.0 glob2==0.4.1 httplib2==0.9 itsdangerous==0.24 jasmine==2.2.0 jasmine-core==2.2.0 jsondatabase==0.0.6 lxml==3.4.0 mixpanel-py==3.1.3 mixpanel-py-async==0.0.3 nltk==3.0.3 numpy==1.9.2 oauthlib==0.6.3 ordereddict==1.1 parse-rest==0.2.20141004 phonenumbers==7.0.0 pyRFC3339==0.2 pycrypto==2.6.1

pydns==2.3.6

Server:

cffi (1.0.2) cloud-init (0.7.5) colorama (0.2.5) cryptography (0.9) distribute (0.7.3) Django (1.8.2) django-dashing (0.2.6) django-uuidfield (0.5.0) enum34 (1.0.4) html5lib (0.999) idna (2.0) ipaddress (1.0.7) mixpanel (4.0.2) MySQL-python (1.2.5) ndg-httpsclient (0.4.0) oauthlib (0.7.2) parse-rest (0.2.20141004) phonenumbers (7.0.4) pip (7.0.3) pyasn1 (0.1.7) pycparser (2.13) pydns (2.3.6) pyinotify (0.9.4) PyJWT (1.1.0) pyOpenSSL (0.15.1) pyRFC3339 (0.2) pyshorteners (0.3) python-openid (2.2.5) python-social-auth (0.2.9) pytz (2015.4) requests (2.7.0) requests-oauthlib (0.5.0) setuptools (3.3) six (1.9.0) ssh-import-id (3.21) twitter (1.17.0) twython (3.2.0) urllib3 (1.10.4) validate-email (1.3) virtualenv (1.11.4)

wsgi-intercept (0.9.1)

I'm going to try removing urllib3, but that might break something else...

[RouxRC]

looks like the local list got cut after pydns, I don't see twitter listed in :)

[lmaloney]

Let me try again...

Verified Local pip freeze: ChatterBot==0.2.0 CherryPy==3.6.0 Cython==0.22 Django==1.6.5 Flask==0.10.1 Jinja2==2.7.3 MarkupSafe==0.23 MySQL-python==1.2.5 PyYAML==3.10 WebOb==1.4.1 Werkzeug==0.10.4 argparse==1.2.1 cchardet==0.3.5 chardet==2.3.0 django-dashing==0.2.5 django-unfriendly==0.3 django-uuidfield==0.5.0 dnspython==1.11.1 dnsq==1.1.3 expiringdict==1.1.3 flanker==0.4.28 fuzzywuzzy==0.5.0 glob2==0.4.1 httplib2==0.9 itsdangerous==0.24 jasmine==2.2.0 jasmine-core==2.2.0 jsondatabase==0.0.6 lxml==3.4.0 mixpanel-py==3.1.3 mixpanel-py-async==0.0.3 nltk==3.0.3 numpy==1.9.2 oauthlib==0.6.3 ordereddict==1.1 parse-rest==0.2.20141004 phonenumbers==7.0.0 pyRFC3339==0.2 pycrypto==2.6.1 pydns==2.3.6 pyexchange==0.4.1 pyicloud==0.5.2 pyshorteners==0.3 python-ntlm==1.1.0 python-openid==2.2.5 python-social-auth==0.1.26 pytz==2014.4 redis==2.10.3 regex==2015.05.07 repoze.lru==0.6 requests==2.7.0 requests-oauthlib==0.5.0 selenium==2.45.0 silpa-common==0.3 simplejson==3.5.2 six==1.7.3 soundex==1.1.3 transitions==0.2.5 twilio==3.6.12 twitter==1.17.0 twython==3.2.0 urbanairship==0.6.3 validate-email==1.3 wsgiref==0.1.2

Verified SERVER pip freeze: cffi==1.0.2 cloud-init==0.7.5 colorama==0.2.5 cryptography==0.9 Django==1.8.2 django-dashing==0.2.6 django-uuidfield==0.5.0 enum34==1.0.4 html5lib==0.999 idna==2.0 ipaddress==1.0.7 mixpanel==4.0.2 MySQL-python==1.2.5 ndg-httpsclient==0.4.0 oauthlib==0.7.2 parse-rest==0.2.20141004 phonenumbers==7.0.4 pyasn1==0.1.7 pycparser==2.13 pydns==2.3.6 pyinotify==0.9.4 PyJWT==1.1.0 pyOpenSSL==0.15.1 pyRFC3339==0.2 pyshorteners==0.3 python-openid==2.2.5 python-social-auth==0.2.9 pytz==2015.4 requests==2.7.0 requests-oauthlib==0.5.0 six==1.9.0 ssh-import-id==3.21 twitter==1.17.0 twython==3.2.0 urllib3==1.10.4 validate-email==1.3 virtualenv==1.11.4 wsgi-intercept==0.9.1

[RouxRC]

This is really just try and error, but maybe pyOpenSSL could be the culprit, it has changed much recently following the various security issues raised and some of its system requirements were updated a lot recently, you might want to try to remove and reinstall it maybe.

[lmaloney]

Ah, that would NOT surprise me at all either.... I've seen plenty of pyOpenSSL warnings about certs in my server logs. I will try that, and keep you posted. :)

[RouxRC]

if you do uninstall reinstall, you mgiht want to include in the batch cffi and urllib3 which are linked with it too

[lmaloney]

Update:

OK, I tried this on another server and had success,, .user() worked fine on our 2nd production server:

stream = twitter.stream.TwitterStream(auth=auth, domain='userstream.twitter.com') stream.user() <generator object iter at 0x7ff73c8b5280>

My pip freeze on 2nd production server: argparse==1.2.1 cloud-init==0.7.5 colorama==0.2.5 html5lib==0.999 oauthlib==0.7.2 pydns==2.3.6 pyinotify==0.9.4 requests==2.2.1 requests-oauthlib==0.5.0 ssh-import-id==3.21 twitter==1.17.0 twython==3.2.0 urllib3==1.7.1 validate-email==1.3 virtualenv==1.11.4 wsgi-intercept==0.9.1 wsgiref==0.1.2

urllib3 is installed on the 2nd production server.... I haven't tried re-installing pyopenssl yet, but I'll try that next.

Funny, I just granted root to someone else on the team, and asked them if they had changed the environment, and they swore that hadn't. :/

[lmaloney]

Update:

Removed pyOpenSSL with pip uninstall pyOpenSSL

Ran code stream.user() , failed with same error No JSON Object could be decoded.

Next: Re-install and try again.

[lmaloney]

Installed pyopenssl with:

ip install pyopenssl Collecting pyopenssl Using cached pyOpenSSL-0.15.1-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/dist-packages (from pyopenssl) Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /usr/local/lib/python2.7/dist-packages (from pyopenssl) Requirement already satisfied (use --upgrade to upgrade): idna in /usr/local/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/local/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/local/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): cffi>=0.8 in /usr/local/lib/python2.7/dist-packages (from cryptography>=0.7->pyopenssl) Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=0.8->cryptography>=0.7->pyopenssl) Installing collected packages: pyopenssl Successfully installed pyopenssl-0.15.1

[lmaloney]

After re-install, code has same JSON error... :(

Doing more research...

[lmaloney]

No joy..

I uninstalled, and re-installed: cffi, pyopenssl, urllib3 and even twitter

Still getting json error...

I may have to create a new virtualenv that is empty and start from there...

[RouxRC]

this is quite weird, not sure what to do here... @sixohsix an idea?

[lmaloney]

Quick Summary Review:

stream.user() works in localdev stream.user() works on 2nd production environment stream.user() fails on 1st production environement

All running twitter 0.17.0

Each environment is slightly different, but mostly the same.

Going to try from fresh virtualenv, and try to ignore global environment on 1st production server.

[lmaloney]

Starting with:

(testenv) # pip freeze argparse==1.2.1 twitter==1.17.0 wsgiref==0.1.2

Then run my code:

(testenv) # python Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import twitter from twitter import *

auth = OAuth( ... consumer_key='redacted', ... consumer_secret='redacted', ... token='redacted', ... token_secret='redacted' ... ) stream = twitter.stream.TwitterStream(auth=auth, domain='userstream.twitter.com') stream.user() Traceback (most recent call last): File "", line 1, in File "/opt/genee/testenv/local/lib/python2.7/site-packages/twitter/api.py", line 308, in call return self._handle_response(req, uri, arg_data, _timeout) File "/opt/genee/testenv/local/lib/python2.7/site-packages/twitter/stream.py", line 284, in _handle_response _timeout or timeout, heartbeat_timeout) File "/opt/genee/testenv/local/lib/python2.7/site-packages/twitter/stream.py", line 209, in handle_stream_response raise TwitterHTTPError(e, uri, 'json', arg_data) File "/opt/genee/testenv/local/lib/python2.7/site-packages/twitter/api.py", line 72, in init data = json.loads(data.decode('utf8')) File "/usr/lib/python2.7/json/init.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

Ugh!

[lmaloney]

Working server:

production-env)ubuntu@prod:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twitter
>>> import json
>>> from twitter import *
>>> auth = OAuth(
...     consumer_key='redactedw',
...     consumer_secret='redacted',
...     token='redacted',
...     token_secret='redacted'
... )

>>> stream = twitter.stream.TwitterStream(auth=auth, domain='userstream.twitter.com')
>>> stream.user()
<generator object __iter__ at 0x7ff73c8b5280> 
>>> quit()

(genee-env)ubuntu@prod:~$ pip freeze
argparse==1.2.1
cloud-init==0.7.5
colorama==0.2.5
html5lib==0.999
oauthlib==0.7.2
pydns==2.3.6
pyinotify==0.9.4
requests==2.2.1
requests-oauthlib==0.5.0
ssh-import-id==3.21
twitter==1.17.0
twython==3.2.0
urllib3==1.7.1
validate-email==1.3
virtualenv==1.11.4
wsgi-intercept==0.9.1
wsgiref==0.1.2
(genee-env)ubuntu@prod:~$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

[lmaloney]

PS: I've installed twitter with: "pip install twitter"

Are there any different install directions? Seems to have worked everywhere else.

[lmaloney]

Update: Still not sure what is happening... I am updating and upgrading the OS to see if that helps. Doubtful.

[lmaloney]

WOW! That worked!!

I'll try to provide a list of updates...

I basically updated ubuntu with apt-get update and apt-get upgrade, and rebooted the machine.

Ran the same code, and it worked! No JSON error!

I don't know if the restart fixed it, or some update in the OS fixed it.

My pip freeze is: $ pip freeze argparse==1.2.1 oauthlib==0.7.2 requests==2.7.0 twitter==1.17.0 wsgiref==0.1.2

[RouxRC]

definitely strange... But I would bet on ssl issues, a lot has been updated recently and I guess the python lib used relies on some of these dependencie.

[lmaloney]

OK... that makes sense. I just want to be thorough for all to utilize... libssl was updated..

here is what updated on my system: cat /var/log/dpkg.log | grep "\ upgrade\ " 2015-06-19 11:15:53 upgrade libtsan0:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:53 upgrade libgomp1:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:53 upgrade libitm1:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:53 upgrade libatomic1:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:53 upgrade libasan0:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:54 upgrade libquadmath0:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:54 upgrade g++-4.8:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:56 upgrade gcc-4.8:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:57 upgrade cpp-4.8:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:15:58 upgrade binutils:amd64 2.24-5ubuntu3.1 2.24-5ubuntu13 2015-06-19 11:15:59 upgrade libstdc++-4.8-dev:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:16:00 upgrade libgcc-4.8-dev:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:16:01 upgrade gcc-4.8-base:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:16:02 upgrade libstdc++6:amd64 4.8.2-19ubuntu1 4.8.4-2ubuntu1~14.04 2015-06-19 11:16:03 upgrade libapt-pkg4.12:amd64 1.0.1ubuntu2.7 1.0.1ubuntu2.8 2015-06-19 11:16:03 upgrade apt:amd64 1.0.1ubuntu2.7 1.0.1ubuntu2.8 2015-06-19 11:16:06 upgrade libapt-inst1.5:amd64 1.0.1ubuntu2.7 1.0.1ubuntu2.8 2015-06-19 11:16:06 upgrade libssl-doc:all 1.0.1f-1ubuntu2.11 1.0.1f-1ubuntu2.15 2015-06-19 11:16:07 upgrade libssl-dev:amd64 1.0.1f-1ubuntu2.11 1.0.1f-1ubuntu2.15 2015-06-19 11:16:07 upgrade libssl1.0.0:amd64 1.0.1f-1ubuntu2.11 1.0.1f-1ubuntu2.15 2015-06-19 11:16:08 upgrade sysvinit-utils:amd64 2.88dsf-41ubuntu6.1 2.88dsf-41ubuntu6.2 2015-06-19 11:16:08 upgrade sysv-rc:all 2.88dsf-41ubuntu6.1 2.88dsf-41ubuntu6.2 2015-06-19 11:16:17 upgrade initscripts:amd64 2.88dsf-41ubuntu6.1 2.88dsf-41ubuntu6.2 2015-06-19 11:16:18 upgrade udev:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:18 upgrade libudev1:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:18 upgrade libldap2-dev:amd64 2.4.31-1+nmu2ubuntu8 2.4.31-1+nmu2ubuntu8.1 2015-06-19 11:16:19 upgrade libldap-2.4-2:amd64 2.4.31-1+nmu2ubuntu8 2.4.31-1+nmu2ubuntu8.1 2015-06-19 11:16:19 upgrade libnuma1:amd64 2.0.9~rc5-1ubuntu3 2.0.9~rc5-1ubuntu3.14.04.1 2015-06-19 11:16:19 upgrade libpam-systemd:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:20 upgrade systemd-services:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:20 upgrade libsystemd-daemon0:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:20 upgrade libsystemd-login0:amd64 204-5ubuntu20.11 204-5ubuntu20.12 2015-06-19 11:16:20 upgrade apt-utils:amd64 1.0.1ubuntu2.7 1.0.1ubuntu2.8 2015-06-19 11:16:21 upgrade isc-dhcp-client:amd64 4.2.4-7ubuntu12.1 4.2.4-7ubuntu12.2 2015-06-19 11:16:21 upgrade isc-dhcp-common:amd64 4.2.4-7ubuntu12.1 4.2.4-7ubuntu12.2 2015-06-19 11:16:22 upgrade libapparmor1:amd64 2.8.95~2430-0ubuntu5.1 2.8.95~2430-0ubuntu5.2 2015-06-19 11:16:22 upgrade libapparmor-perl:amd64 2.8.95~2430-0ubuntu5.1 2.8.95~2430-0ubuntu5.2 2015-06-19 11:16:22 upgrade apparmor:amd64 2.8.95~2430-0ubuntu5.1 2.8.95~2430-0ubuntu5.2 2015-06-19 11:16:22 upgrade apt-transport-https:amd64 1.0.1ubuntu2.7 1.0.1ubuntu2.8 2015-06-19 11:16:23 upgrade openssl:amd64 1.0.1f-1ubuntu2.11 1.0.1f-1ubuntu2.15 2015-06-19 11:16:23 upgrade grub-pc:amd64 2.02~beta2-9ubuntu1.1 2.02~beta2-9ubuntu1.2 2015-06-19 11:16:23 upgrade grub-pc-bin:amd64 2.02~beta2-9ubuntu1.1 2.02~beta2-9ubuntu1.2 2015-06-19 11:16:24 upgrade grub2-common:amd64 2.02~beta2-9ubuntu1.1 2.02~beta2-9ubuntu1.2 2015-06-19 11:16:24 upgrade grub-common:amd64 2.02~beta2-9ubuntu1.1 2.02~beta2-9ubuntu1.2 2015-06-19 11:16:25 upgrade linux-libc-dev:amd64 3.13.0-53.89 3.13.0-55.92 2015-06-19 11:16:26 upgrade cloud-init:all 0.7.5-0ubuntu1.5 0.7.5-0ubuntu1.6 2015-06-19 11:16:26 upgrade grub-legacy-ec2:all 0.7.5-0ubuntu1.5 0.7.5-0ubuntu1.6 2015-06-19 11:16:27 upgrade overlayroot:all 0.25ubuntu1 0.25ubuntu1.14.04.1

[lmaloney]

I'm closing this issue. Hope anyone else who has this problem is helped.

\ Remember, double check your openssl packages first before anything else as rouxc advises.

Thank you Benjamin!

/Larry

[RouxRC]

no pb, thank you for the extensive and comprehensive bug report :)