I'm not sure yet what is causing this. We're seeing it in production about once a day at this point. I had previously been running aioamqp 0.8.2 on Python 3.5.2 on Ubuntu LTS 16.04.1 LTS and didn't have the problem. I recently moved to aioamqp 0.9.0 on Python 3.5.2 on Alpine Linux 3.5 and started getting this exception:
AttributeError: 'NoneType' object has no attribute 'readexactly'
File "aioamqp/protocol.py", line 304, in run
yield from self.dispatch_frame()
File "aioamqp/protocol.py", line 251, in dispatch_frame
frame = yield from self.get_frame()
File "aioamqp/protocol.py", line 235, in get_frame
yield from frame.read_frame()
File "aioamqp/frame.py", line 412, in read_frame
data = yield from self.reader.readexactly(7)
The error causes a tight loop. I don't know yet how to reproduce it. I've attempted forcibly killing RabbitMQ while connected, gracefully shutting down, etc. In an attempt to mask the error I created a child class to attempt to convert the error:
import aioamqp
import aioamqp.exceptions
import aioamqp.protocol
LOGGER = logging.getLogger(__name__)
class ProtocolSafe(aioamqp.protocol.AmqpProtocol):
"""
This class exists to sidestep an issue
https://github.com/Polyconseil/aioamqp/issues/115
which we are plagued with in production
"""
@asyncio.coroutine
def read_frame(self):
try:
super().read_frame(self)
except AttributeError:
LOGGER.warning("Translating '%s' into a closed connection")
raise aioamqp.exceptions.AmqpClosedConnection('Inferred from AttributeError')
...
yield from aioamqp.connect(..., protocol_factory=ProtocolSafe)
After deploying that I still get the problem. Something about my class isn't working correctly because while sentry indicates that the protocol instance is a ProtocolSafe the stack trace does not show my version of read_frame. I'm going to keep working this angle and report back here.
I'm not sure yet what is causing this. We're seeing it in production about once a day at this point. I had previously been running aioamqp 0.8.2 on Python 3.5.2 on Ubuntu LTS 16.04.1 LTS and didn't have the problem. I recently moved to aioamqp 0.9.0 on Python 3.5.2 on Alpine Linux 3.5 and started getting this exception:
The error causes a tight loop. I don't know yet how to reproduce it. I've attempted forcibly killing RabbitMQ while connected, gracefully shutting down, etc. In an attempt to mask the error I created a child class to attempt to convert the error:
After deploying that I still get the problem. Something about my class isn't working correctly because while sentry indicates that the protocol instance is a
ProtocolSafe
the stack trace does not show my version ofread_frame
. I'm going to keep working this angle and report back here.