Messages always returned in reverse order, intentional?

[owst]

When receiving multiple messages from elasticmq, the messages are always returned in reverse-insertion order - is this a conscious design decision?

This behaviour is allowed by the SQS FAQs (see "Does Amazon SQS provide message ordering?"), as standard SQS queues only "provide a loose-FIFO capability that attempts to preserve the order of messages" and therefore elasticmq is not breaking any "specification". But it seems somewhat strange to actively return the messages in reverse order, rather than making any attempt at preserving insertion order (which is trivial to do, see below).

Example

Consider the following simple test script that relies on the aws cli and jq and assumes elasticmq is available at http://localhost:9324:

#!/usr/bin/env bash
set -euo pipefail

queue_url=$(aws --endpoint-url http://localhost:9324 sqs create-queue --queue-name test-queue | jq -r .QueueUrl)

function sqs() {
    aws --endpoint-url http://localhost:9324 sqs $@ --queue-url $queue_url
}

function send() {
    message_id=$(sqs send-message --message-body $1 | jq .MessageId)
    echo "Sent message $message_id with body $1"
}

function receive() {
    echo "Receiving at most $1 messages..."
    sqs receive-message --max-number-of-messages $1 | jq '.Messages | .[] | .Body'
}

send 1-1
send 1-2
send 1-3
send 1-4

receive 1
receive 1
receive 1
receive 1

send 2-1
send 2-2
send 2-3
send 2-4

receive 2
receive 2

send 4-1
send 4-2
send 4-3
send 4-4

receive 4

sqs delete-queue --queue-url $queue_url

Here, we send 4 messages three times, receiving one at a time, then two at a time and finally four at a time; an example output as per ca10395bb9e7f39f3a687174f2c63d9908943bcd is:

Sent message "5d8a80f5-a60a-4a90-94d1-de050437426c" with body 1-1
Sent message "2b63922d-9e41-41fd-b581-87101e6440f4" with body 1-2
Sent message "9824ae60-c02f-4b9f-a115-70181bf23c6d" with body 1-3
Sent message "6a9699d1-a658-4381-8159-3bc1a6992e30" with body 1-4
Receiving at most 1 messages...
"1-1"
Receiving at most 1 messages...
"1-2"
Receiving at most 1 messages...
"1-3"
Receiving at most 1 messages...
"1-4"
Sent message "b65f55a2-2ca8-4042-9382-4d97236310a2" with body 2-1
Sent message "6e8ee5fd-1ff6-4611-98c7-c508d5fa20f8" with body 2-2
Sent message "49343746-2dca-4c69-a418-207e28161bdd" with body 2-3
Sent message "bec16f5d-f761-4a85-9f21-2b9c856d1ea6" with body 2-4
Receiving at most 2 messages...
"2-2"
"2-1"
Receiving at most 2 messages...
"2-4"
"2-3"
Sent message "d59cf9a7-0294-478c-a53f-6d1575e26f4a" with body 4-1
Sent message "13d3b8a7-c1d0-4781-918f-4eaa279572ae" with body 4-2
Sent message "b1d468fb-74c9-4560-adba-800278512500" with body 4-3
Sent message "4a2cd3c0-a7f4-46ec-aabd-b8b710a068c9" with body 4-4
Receiving at most 4 messages...
"4-4"
"4-3"
"4-2"
"4-1"

Notice that the n-oldest messages are returned in each request (e.g. 2-2 and 2-1 are returned before 2-4 or 2-3), but when more than one message is returned in a batch, the message list is returned in reverse order.

The "fix" is simple: change this line such that we call .reverse on the resulting List[MessageData]. The list is built in reverse order as messages are repeatedly dequeued from the internal Deque and cons'd onto the (front of the) accumulation list.

Indeed, with this simple diff

diff --git a/core/src/main/scala/org/elasticmq/actor/queue/QueueActorMessageOps.scala b/core/src/main/scala/org/elasticmq/actor/queue/QueueActorMessageOps.scala
index 92767df..12e41f9 100644
--- a/core/src/main/scala/org/elasticmq/actor/queue/QueueActorMessageOps.scala
+++ b/core/src/main/scala/org/elasticmq/actor/queue/QueueActorMessageOps.scala
@@ -78,7 +78,7 @@ trait QueueActorMessageOps extends Logging {
       }
     }

-    doReceiveMessages(count, Nil)
+    doReceiveMessages(count, Nil).reverse
   }

   @tailrec

an example output from the above test script is:

Sent message "6c822d52-881e-49e1-9022-6eacc1a8500c" with body 1-1
Sent message "792ec322-d0a3-48ff-9e1d-9dbe8fbc5093" with body 1-2
Sent message "d163e5d9-5846-4570-ad8d-4a1a0d104367" with body 1-3
Sent message "2b8ef991-acbf-4b0c-b869-3e8177af0a83" with body 1-4
Receiving at most 1 messages...
"1-1"
Receiving at most 1 messages...
"1-2"
Receiving at most 1 messages...
"1-3"
Receiving at most 1 messages...
"1-4"
Sent message "0cae4455-7a6e-4dcb-936a-fdaeaef8a050" with body 2-1
Sent message "29cbb437-89cf-4028-8a41-e1fe8f1e0ffd" with body 2-2
Sent message "010a61b1-3a70-4abd-b742-14ba0aef24f4" with body 2-3
Sent message "3fa90e6e-8f6f-4787-8fe6-f09e98196553" with body 2-4
Receiving at most 2 messages...
"2-1"
"2-2"
Receiving at most 2 messages...
"2-3"
"2-4"
Sent message "515551b1-133c-43cd-9c59-a92a20c6ff94" with body 4-1
Sent message "7191e04b-3290-44a1-88d7-f5d2c0f37600" with body 4-2
Sent message "85019883-b0f2-470f-84bb-d59d6b411e75" with body 4-3
Sent message "3f979e76-7fe0-40c1-9299-baf27faabe75" with body 4-4
Receiving at most 4 messages...
"4-1"
"4-2"
"4-3"
"4-4"

Notice now that the messages are always returned in insertion order.

I suppose this change may lead to a false impression of SQS behaviour, but it seems less surprising than always returning in exactly reverse order? If you agree this is a sensible change, I can open a PR!

Thanks, Owen.

[adamw]

Thanks for the detailed investigation! As the goal is to emulate SQS, then if there's a difference, we should definitely adjust if possible :) A PR would be great!

[owst]

Great, I'll try and get a PR sorted today