Support a mirror queue for testing

[nhoughto]

We are using ElasticMQ as local SQS substitute for testing, one problem we have is in a full integrate test we are testing both the producers and consuming components and asserting a message was created is racey as it is often consumed before the assertion runs.

Lots of ways to tackle this, one would be for ElasticMQ to mirror all messages received on a specific queue to a mirror queue that would only be consumed by the test/assertions themselves. Is such a thing possible / desirable? Doing it in ElasticMQ rather than the client code seems cleaner.

[adamw]

That's not currently supported, but adding an "audit" queue to verify that all messages have been properly handled could make much sense. It shouldn't be very hard to implement, if you'd like to try :)

[nhoughto]

Does the “delay” setting try to achieve the same goal? If I enable a delay on the queues does that give me a window to assert again the message being in the queue before it’s read?

I tried delay but it didn’t seem to do anything, messages were just read straight away.

[adamw]

I think the original purpose of delay is different, to deliver messages after a certain period of time, but I suppose it should be possible to implement what you are after using this as well.

If I recall correctly, a global queue-delay should apply to all messages, unless they have a different delay explicitly specified. Is that not the behavior you are seeing?

[nhoughto]

I've got it configured like this

queues {
  transactions {
    delay = 10 seconds
  }
}

And with DEBUG level i'm expecting to see a Sent.. event, and no Receive.. event until 10+seconds afterwards. But i see the Receive.. immediately afterwards. I couldn't see anywhere to specify a global delay (which i would prefer), how do I do that?

[nhoughto]

Either way, as your mentioned delay doesn't achieve what i want, it would just make the messages unavailable to both the consumer and the test assertion. The mirror / audit queue is a better solution, any tips for how to PR it in? My scala is weak.

[adamw]

There's no way to create a global delay; and I think the way you configured the queue should work fine; maybe that's a separate bug ;)

Anyway, you are right that using the delay (even if it worked properly) wouldn't do the trick here, as the messages would be unavailable to both consumer.

I think the way to go here would be to add an auditQueue, similarly to how a deadLettersQueue is specified. This would require changes in the configuration, server, all the way up to the core actors. The Scala isn't very sophisticated, but it would be probably a couple of hours of work :)

[nhoughto]

We went another direction with this (doing assertions further downstream), so I won't have time to do the work. Still seems like a good idea though =)

[adamw]

Ok, I'll keep this open to implement this one day (maybe :) )

[sbinq]

Hi guys, one concern regarding this feature for testing:

I think it would be beneficial to have not only "copy each message to another queue" ability (i.e. what is called "audit queue" here) - but also smth like "move/redirect each message to another queue" (thus removing message from original queue).

The difference here is that component handling that queued message might be able (or not able - depending on "lucky"/"unlucky" timing) to read message and process or partially process it (thus performing any arbitrary side effects/updating databases/interacting with other services/doing anything else) - which could lead to difficulties in tests (assuming in scope of the given test we want to check only "message was sent", not "message was sent and all the processing happened").

What do you guys think about it? @adamw ?

[adamw]

@sbinq if people would find it useful for testing, why not. I suppose that could take the form of options on a queue, copy-to = [other queue name] (audit), move-to = [other queue name]

[sbinq]

Hi @adamw, could you please take a look at the pull request referenced above if you have some time for this?

That's my first attempt implementing functionality as discussed here - please let me know if you'd like smth to be done in different way/any concerns on this/etc. Thanks!

[v1-wizard]

Hi @adamw many thanks for merged request @sbinq. I think you could close this issue.