AttributeError: 'NoneType' object has no attribute 'fill'

[72squared]

We send out about 2.4 million APNS requests every day. Anywhere from 30 to 40 run into an exception with this backtrace:

Traceback (most recent call last): -- File /srv/sites/marcopolo/notify/push_apns.py, line 154, 
    in push_apns conn.send_notification(token_hex=push_token, notification=notification, topic=topic, expiration=expiry)
    File /opt/venv-marcopolo/src/apns2/apns2/client.py, line 38, in send_notification resp = self.__connection.get_response(stream_id)
    File /opt/venv-marcopolo/local/lib/python2.7/site-packages/hyper/http20/connection.py, line 293, in get_response return HTTP20Response(stream.getheaders(), stream)
    File /opt/venv-marcopolo/local/lib/python2.7/site-packages/hyper/http20/stream.py, line 223, in getheaders self._recv_cb(stream_id=self.stream_id)
    File /opt/venv-marcopolo/local/lib/python2.7/site-packages/hyper/http20/connection.py, line 744, in _recv_cb self._single_read()
    /opt/venv-marcopolo/local/lib/python2.7/site-packages/hyper/http20/connection.py, line 644, in _single_read self._sock.fill()
    AttributeError: 'NoneType' object has no attribute 'fill'

Not sure what causes it. Maybe some sort of race condition? This happens in a background job system which does use threads, so maybe there is a race condition in the library where it removes the self._sock objects from the connection in one thread and tries to use it in another? I'm not totally convinced that's the problem though because it happens in short bursts across multiple webservers. Maybe some sort of network disruption event causes it?

[Pr0Ger]

Since this is actually a hyper issue I'm closing this issue

[72squared]

Since it bubbles up into the application in weird ways, you might want to try/catch around the call to hyper and raising a more meaningful error until it gets fixed? but yeah it's definitely a problem that should be fixed in hyper.