SequenceNumber missing from Message Attributes (SQS fifo)

[terozio]

When doing a receive-message for a fifo queue message attributes should contain attribute called SequenceNumber (to indicate the ordering within a message group in a fifo queue). https://docs.aws.amazon.com/cli/latest/reference/sqs/receive-message.html#output

Steps to replicate:

1. docker run -p 9324:9324 -p 9325:9325 softwaremill/elasticmq-native:1.2.0
2. aws sqs create-queue --queue-name test-queue.fifo --attributes FifoQueue=true --endpoint-url http://localhost:9324"
3. aws sqs send-message --queue-url http://localhost:9324/queue/test-queue.fifo --message-body "msg body" --message-group-id "my-group" --message-deduplication-id "1234" --endpoint-url http://localhost:9324
4. aws sqs receive-message --queue-url http://localhost:9324/queue/test-queue.fifo --attribute-names All --endpoint-url http://localhost:9324
5. Observe that the response does not contain a field called SequenceNumber inside attributes

Our usecase is testing Lambda functions locally, we use SequenceNumber for idempotency purposes when the lamba stores the last message that has been successfully processed.

Not sure if this is a bug or that the fifo functionality is just partially implemented, didn't notice any explicit statement on the documentation.

[pierscin]

It appears to be an issue in elastic.

According to sqs docs:

SequenceNumber -> (string)

This parameter applies only to FIFO (first-in-first-out) queues. The large, non-consecutive number that Amazon SQS assigns to each message. The length of SequenceNumber is 128 bits. SequenceNumber continues to increase for a particular MessageGroupId.

[adamw]

Fixed in 1.2.1

[peebles]

I am still not seeing this attribute, nor am I seeing the FifoQueue attribute.

I am using ELQ version 1.2.2: 21:28:26.792 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (1.2.2) ...

My queue looks like: $ aws --endpoint-url http://localhost:9324 sqs get-queue-attributes --queue-url "http://localhost:9324/queue/zc-ingest-local.fifo" --attribute-names All { "Attributes": { "ApproximateNumberOfMessagesNotVisible": "0", "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:elasticmq:000000000000:zc-ingest-local-dlq.fifo\",\"maxReceiveCount\":1}", "ContentBasedDeduplication": "true", "ApproximateNumberOfMessagesDelayed": "0", "CreatedTimestamp": "1633210155", "ApproximateNumberOfMessages": "0", "ReceiveMessageWaitTimeSeconds": "5", "DelaySeconds": "0", "FifoQueue": "true", "VisibilityTimeout": "10", "LastModifiedTimestamp": "1633210155", "QueueArn": "arn:aws:sqs:elasticmq:000000000000:zc-ingest-local.fifo" } }

But the attributes when I dequeue a message look like

{ "SentTimestamp": "1633210803056", "ApproximateReceiveCount": "1", "ApproximateFirstReceiveTimestamp": "1633210803056", "SenderId": "127.0.0.1", "MessageDeduplicationId": "c9f005404172eaf9ca2cea9a86be9cc47b75d948313ad7efa086e4d1dddcc04e", "MessageGroupId": "IMa2eeuCPv" }

[pierscin]

Thanks for your feedback - I must have confused message attributes with attributes at some point.

I've made some changes and tested with:

val sendReq = new SendMessageRequest(qurl, s"Message 1")
      .withMessageGroupId("1")
      .withMessageDeduplicationId("test")

val sendRes = client.sendMessage(sendReq)

println(sendRes.getSequenceNumber)

val msg = client
  .receiveMessage(new ReceiveMessageRequest(qurl)
    .withWaitTimeSeconds(2)
    .withAttributeNames("All")
    .withMessageAttributeNames("All"))
  .getMessages
  .asScala
  .headOption
  .orNull

println(msg.getAttributes.get("SequenceNumber"))

[adamw]

@peebles can you try 1.2.3? Should be fixed

[peebles]

I see SequenceNumber now, but I don't see FifoQueue. In fact, quite a few others seem to be missing? From the AWS doc:

var params = {
  QueueUrl: 'STRING_VALUE', /* required */
  AttributeNames: [
    All | Policy | VisibilityTimeout | MaximumMessageSize | MessageRetentionPeriod | ApproximateNumberOfMessages | ApproximateNumberOfMessagesNotVisible | CreatedTimestamp | LastModifiedTimestamp | QueueArn | ApproximateNumberOfMessagesDelayed | DelaySeconds | ReceiveMessageWaitTimeSeconds | RedrivePolicy | FifoQueue | ContentBasedDeduplication | KmsMasterKeyId | KmsDataKeyReusePeriodSeconds | DeduplicationScope | FifoThroughputLimit | RedriveAllowPolicy,
    /* more items */
  ],
  MaxNumberOfMessages: 'NUMBER_VALUE',
  MessageAttributeNames: [
    'STRING_VALUE',
    /* more items */
  ],
  ReceiveRequestAttemptId: 'STRING_VALUE',
  VisibilityTimeout: 'NUMBER_VALUE',
  WaitTimeSeconds: 'NUMBER_VALUE'
};
sqs.receiveMessage(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});

I don't know how practical it is to implement all of those. I just wanted to point it out.

[adamw]

@peebles Yeah some of the attributes might be missing. Not all functionalities are supported in ElasticMQ, though (e.g. Kms*). If there's something that you think is missing and can be added right away, or a funcitonality is missing that you need, please open separate issues. Thanks :)