Queues not durable

[tuwid]

I was expecting messages to pile up with :durable using the following code:

my $connection = $n.connect.result;

react {
    whenever $n.open-channel(1) -> $channel {
        $channel.qos(0,1);
        whenever $channel.declare-queue("task_queue", :durable) -> $q {
            $q.consume(:ack);
            whenever $q.message-supply -> $message {
                say $message.body.decode;
                sleep (^10).pick;
                $channel.ack($message.delivery-tag);
            }
        }
    }
}

await $connection;

but the previous messages were gone. So basically I'm expecting that if i have no workers consuming the jobs, such jobs would pile up and wait for the next consumer but such thing its not happening.

Any ideas ?

[jonathanstowe]

Hi, Thanks for this. I'm assuming you are using the code from https://github.com/jonathanstowe/Perl6-RMQ-Examples/tree/master/tutorial-002 (which is nice to know as I hadn't really promoted it much :)

The :durable flag on the queue declaration only applies to the queue itself (that is it will persist between broker restarts,) and not to the messages published to the queue, for which you should set the :persistent flag on the publish (which in turn sets the delivery mode of the message to 2, i.e. persistent,) the sender example from the above does this,.

Now on the face of it the code in Net::AMQP appears to do the right thing and running the sender without the consumer running the messages do appear to be queued: you can determine this with e.g.

  sudo rabbitmqctl list_queues name messages_ready

Or looking at http://localhost:15672/#/queues/%2F/task_queue or whatever. So at this point all seems to be golden.

Now starting the receiver it appears that all the queued messages are consumed almost immediately yet only (in my test anyway,) one is actually handled, which is patently incorrect. At this stage it's not entirely clear to me what is happening to the queued messages and whether the fault is in the example code or in Net::AMQP::Queue.message-supply but I'll dig into it later as one of them obviously needs fixing.

Thanks for pointing this out, if nothing else it points to a gap in the tests. I'll get back when I have something more concrete to report.

[jonathanstowe]

Actually I think you may have discovered two infelicities for the price of one :) It seems that my assertion that the persistence works is only true if the first unread message in the queue was created as a persistent message by some other means (in my test above I created it with the Web UI,) now not only does this suggest a fault in Net::AMQP it also confuses me because I'm not sure why the presence of a persistent message in a queue should influence the others (but that could be me failing to understand the way RabbitMQ works.)

Anyhow, I've determined that the tests don't actually cover this behaviour as in every case they start the consumer before sending the messages. This is entirely my fault as I wrote those tests.

[jonathanstowe]

Hi, I managed to take a look at that this morning. It transpires that the particular set of defaults that are used for the message properties cause the messages not to be very persistent at all (specifically the expiration and the priority,) I have removed all the defaults for the publish and this has fixed the behaviours described above. I'm still not sure why the messages were staying queued only when there was an existing one in there, but I think that is a problem (perhaps,) with RabbitMQ rather than this module. The reason that I was seeing all the messages getting consumed off the queue and only one being actually handled was that when the existing properly persistent message was popped off the queue all the other ones (created by Net::AMQP,) suddenly expired and disappeared rather than being consumed by the receiver program.

Anyhow I've fixed that all in a branch and will merge and release when I have fixed up some tests to catch this behaviour. Should be this evening.

[tuwid]

Hi @jonathanstowe , thanks for all the effort. I was just playing around with perl6 for fun ( as an old perl 5 user ) but since I had to finish my microservice quickly i fell back to perl 5 to have a prototype ready. Will test this and will investigate it a bit as I noticed that rabbit has other more conventional ways of working that the one we're using in the example.

Thanks again, Artur

[jonathanstowe]

That should be all good now,