jeremycline
commented
5 years ago So after a lot of staring at the backtrace and the code, I believe what's happening is publish is being called in the composer consumer, but no exceptions are being handled so when the connection error occurs this is bubbled up into the consumer loop and crashes the consumer.
The publish API is raising the proper "ConnectionException" in response to "pika.exceptions.StreamLostError" so this isn't exactly a bug in fedora-messaing, but what it's not doing is automatically retrying before giving up because it's entering the generic AMQPError handler. We should fix this, especially because the blocking connection times out regularly as it cannot heartbeat. This will cause users to hit it less frequently, but they still need to handle exceptions where the automatic retry doesn't work.
bowlofeggs
In Fedora's staging environment, I have observed this problem in both OpenShift and bodhi-backend01.stg today:
In the case of Bodhi's OpenShift consumer, it ends up crashing the process, systemd restarts it, the message is tried again, and it was successful.
However, in Bodhi's composer, the composer itself caught the exception (it has a catch all so we can log and mark the compose as failed), which means that the message was ultimately ACK'd and the process did not halt and the compose was left in the failed state.
Should it be the responsibility of the application (i.e., Bodhi) to catch this kind of error, or should fedora-messaging catch and handle it for me? I'm not sure fedora-messaging could catch it in the case of Bodhi's composer…