Livepeer Studio is your home for building, broadcasting, and publishing video on the open internet with the Livepeer Network. Effortlessly manage livestreams, video uploads, API keys, network usage, billing, and more.
We have had a couple incidents where the API just stops sending messages to
RabbitMQ and we don't even know about it, since the library we use abstracts
the underlying connection and failures. I believe it just accumulates messages in
memory and keeps trying to reconnect forever, hoping it will eventually succeed.
Update: This is even weirder now, from the docs, since they claim that if a callback
is not sent, the returned promise will only be fulfilled when the publish actually happens.
So this needs further investigation. My immediate suspicion is that the lib is not actually
checking broker confirmations (like a reverse ACK) and we need to enable that somehow.
Shih-Yu
commented
1 year ago
We have had a couple incidents where the API just stops sending messages to RabbitMQ and we don't even know about it, since the library we use abstracts the underlying connection and failures. I believe it just accumulates messages in memory and keeps trying to reconnect forever, hoping it will eventually succeed.
There is a way to send a callback to the publish function to make sure we only return a success to the users if we do have a success publishing the message though: https://www.npmjs.com/package/amqp-connection-manager#channelwrapperpublish-and-channelwrappersendtoqueue
Update: This is even weirder now, from the docs, since they claim that if a callback is not sent, the returned promise will only be fulfilled when the publish actually happens. So this needs further investigation. My immediate suspicion is that the lib is not actually checking broker confirmations (like a reverse ACK) and we need to enable that somehow.