search

alphagov / govuk_crawler_worker

A worker that will consume GOV.UK URLs from a message queue and crawl them, saving the output to disk
MIT License
18 stars 8 forks source link

Retry when publisher confirms fail #7

Open KushalP opened 10 years ago

KushalP commented 10 years ago

We're currently monitoring publisher confirms as part of the QueueConnection object but aren't doing anything in the event that a given publish fails. You can see an example of how this can be done from the author of the amqp library we're using here: https://github.com/streadway/amqp/issues/104#issuecomment-46810166

mattbostock commented 10 years ago

Noting this here because I think it's relevant:

We set the mandatory and immediate flags to false when publishing a message.

This could lead to circumstances in which messages are silently dropped.

We may need to look at implementing a listener with Channel.NotifyReturn to handle any undeliverable messages when calling publish as per if we enable either of these flags: http://godoc.org/github.com/streadway/amqp#Channel.Publish

KushalP commented 10 years ago

Agreed. This is all related to the possibility that the AMQP server tells us it couldn't confirm receipt of a message.