Polyconseil / aioamqp

AMQP implementation using asyncio
Other
280 stars 88 forks source link

AttributeError: 'NoneType' object has no attribute 'readexactly' #139

Closed EliRibble closed 7 years ago

EliRibble commented 7 years ago

I'm not sure yet what is causing this. We're seeing it in production about once a day at this point. I had previously been running aioamqp 0.8.2 on Python 3.5.2 on Ubuntu LTS 16.04.1 LTS and didn't have the problem. I recently moved to aioamqp 0.9.0 on Python 3.5.2 on Alpine Linux 3.5 and started getting this exception:

AttributeError: 'NoneType' object has no attribute 'readexactly'
  File "aioamqp/protocol.py", line 304, in run
    yield from self.dispatch_frame()
  File "aioamqp/protocol.py", line 251, in dispatch_frame
    frame = yield from self.get_frame()
  File "aioamqp/protocol.py", line 235, in get_frame
    yield from frame.read_frame()
  File "aioamqp/frame.py", line 412, in read_frame
    data = yield from self.reader.readexactly(7)

The error causes a tight loop. I don't know yet how to reproduce it. I've attempted forcibly killing RabbitMQ while connected, gracefully shutting down, etc. In an attempt to mask the error I created a child class to attempt to convert the error:

import aioamqp
import aioamqp.exceptions
import aioamqp.protocol

LOGGER = logging.getLogger(__name__)

class ProtocolSafe(aioamqp.protocol.AmqpProtocol):
    """
    This class exists to sidestep an issue
    https://github.com/Polyconseil/aioamqp/issues/115
    which we are plagued with in production
    """
    @asyncio.coroutine
    def read_frame(self):
        try:
            super().read_frame(self)
        except AttributeError:
            LOGGER.warning("Translating '%s' into a closed connection")
            raise aioamqp.exceptions.AmqpClosedConnection('Inferred from AttributeError')

...
yield from aioamqp.connect(..., protocol_factory=ProtocolSafe)

After deploying that I still get the problem. Something about my class isn't working correctly because while sentry indicates that the protocol instance is a ProtocolSafe the stack trace does not show my version of read_frame. I'm going to keep working this angle and report back here.

EliRibble commented 7 years ago

screen shot 2017-04-19 at 8 28 35 am

EliRibble commented 7 years ago

Looks like this may be a duplicate of https://github.com/Polyconseil/aioamqp/issues/115