calpaterson / quarchive

Traditional "Web 2.0" social bookmarking, with small improvements

https://quarchive.com/

GNU Affero General Public License v3.0

99 stars 6 forks source link

When messages pile up, things go terribly wrong in the bg-worker #31

Closed calpaterson closed 3 years ago

calpaterson commented 3 years ago

Description

Things go terribly wrong in the bg worker when multiple (4+) messages stack up. Something rotten in Missive.

Steps to reproduce

Queue a number of messages for the bg-worker (can just publish when it's stopped)
Then consume them all at once

Expected result

Messages processed one after another

Actual result

Processor crashes.

Additional details

Looks like there are a few different problems occurring:

Wrong messages being acked by RabbitMQAdapter.ack
Database not being rolled back between handlers
Perhaps other problems?
Runaway channel creation on the publisher side

calpaterson commented 3 years ago

Steps to a robust resolution:

[X] Stop sharing an AMQP connection between consuming and publishing
[X] Missive fix to message acking
[x] Rollback database between message handles (Missive change required)
[ ] Log message ids for correlation
[ ] Improve __str__ method of missive.Message

calpaterson commented 3 years ago

Relates directly to https://github.com/calpaterson/missive/issues/1 and should probably fix https://github.com/calpaterson/missive/issues/2 at the same time

calpaterson commented 3 years ago

And requires https://github.com/calpaterson/missive/issues/3