search

sstone / amqp-client

[THIS PROJECT HAS BEEN ARCHIVED AND IS NO LONGER MAINTAINED] Simple fault-tolerant AMQP client written in Scala and based on Akka and the RabbitMQ java client
MIT License
161 stars 88 forks source link

Waiting for all messages to be sent #72

Closed krvss closed 8 years ago

krvss commented 8 years ago

Recently I had a task to send more than a million messages to RabbitMQ (exporting processed data from the one system to another).

Imagine a very basic flow: read the message from S3, send it to rabbit, repeat.

I had no problems with the export implementation using ampq-client, but noticed, that publishing goes quite fast, but I need to wait for Akka system to actually send all messages to rabbit - if I call system.shutdown prematurely it will interrupt the sending process, and just 10% of messages will actually go to rabbit.

I guess I need to listen for some event from Akka, but because I had to finish things quickly I just moved to Java client for synchronous sending. If you know how to get a notification that all the messages were processed, I suggest to add it to the examples.

sstone commented 8 years ago

It might be a good use case for publisher confirms (https://www.rabbitmq.com/confirms.html). I'm working on something else right now but I'll try and find some time to write an example soon.

sstone commented 8 years ago

Hi. Sorry it took so long, here's a simple gist that demonstrates how to you publisher confirms to wait till all messages have been published. I'll add it to the examples.

sstone commented 8 years ago

No new is good news so I'm closing this issue.