Cannot retreive MessageAttributes from Message

[jairovadillo]

I'm working with a very small code to publish and receive events. I'm adding MessageAttributes in the publisher side but I'm not receiving anything on the consumer side.

I'm working with python3.7 and boto3 and the same code works perfectly with the real AWS SNS/SQS.

I don't know if it's necessary to add code since it's pretty basic and simple: boto3.client('sns', **aws_settings).publish(arn, message, MessageAttributes={'foo': 'bar'})

Any thoughts?

[slang25]

It could be that this will fix it: https://github.com/p4tin/goaws/pull/178

[jairovadillo]

@p4tin here you have how to reproduce the error: https://github.com/21Buttons/molange/pull/10

If you ran the end-to-end tests (explained on the readme) you'll see that the message received from SQS misses the MessageAttributes key.

If you ran this same tests changing the env vars to AWS everything works :( Let me know if I can help with something

[p4tin]

@jairovadillo I have done a test with the "aws cli":

Created an 'attribute.json' file and then run a send message with those attributes and then a receive message and all the attributes are fine, see run below...

You might have a boto configuration - i'm not sure at the moment without boto3 investigation....

Did you mean "Attributes" and not "MessageAttributes"??

➜  aws more attributes.json
{
  "City": {
    "DataType": "String",
    "StringValue": "Any City"
  },
  "Greeting": {
    "DataType": "Binary",
    "BinaryValue": "Hello, World!"
  },
  "Population": {
    "DataType": "Number",
    "StringValue": "1250800"
  }
}

➜  aws aws --endpoint-url http://localhost:4101 sqs send-message --queue-url http://localhost:4100/local-queue1 --message-body "this is a test of the GoAws Queue messaging" --message-attributes file://attributes.json
{
    "MD5OfMessageBody": "9d3f5eaac3b1b4dd509f39e71e25f954",
    "MD5OfMessageAttributes": "28eb0e573cf8e8a77e349a2f968eac4a",
    "SequenceNumber": "",
    "MessageId": "c385091f-5acc-46b8-825d-4f3f5e7343a5"
}
➜  aws aws --endpoint-url http://localhost:4100 sqs receive-message --queue-url http://localhost:4100/local-queue1
{
    "Messages": [
        {
            "Body": "this is a test of the GoAws Queue messaging",
            "ReceiptHandle": "c385091f-5acc-46b8-825d-4f3f5e7343a5#0b6aac87-f24e-496b-bb01-26f5efb8102f",
            "MD5OfBody": "9d3f5eaac3b1b4dd509f39e71e25f954",
            "MD5OfMessageAttributes": "28eb0e573cf8e8a77e349a2f968eac4a",
            "MessageId": "c385091f-5acc-46b8-825d-4f3f5e7343a5",
            "MessageAttributes": {
                "City": {
                    "DataType": "String",
                    "StringValue": "Any City"
                },
                "Greeting": {
                    "DataType": "Binary",
                    "BinaryValue": "SGVsbG8sIFdvcmxkIQ=="
                },
                "Population": {
                    "DataType": "Number"
                }
            }
        }
    ]
}

[jairovadillo]

@p4tin hey I've created the same example but with SNS + SQS and still the same, no MessageAttributes:

Using same JSON and this conf:

Local:
  Host: goaws                   # hostname of the goaws system  (for docker-compose this is the tag name of the container)
  Port: 4100                        # port to listen on.
  Region: local-01
  LogMessages: true                 # Log messages (true/false)
  LogFile: ./goaws_messages.log  # Log filename (for message logging
  QueueAttributeDefaults:           # default attributes for all queues
    VisibilityTimeout: 30              # message visibility timeout
    ReceiveMessageWaitTimeSeconds: 0   # receive message max wait time
  Queues:                           # List of queues to create at startup
  - Name: local-queue1                # Queue name
    ReceiveMessageWaitTimeSeconds: 20 # Queue receive message max wait time
  Topics:                           # List of topic to create at startup
  - Name: local-topic1            # Topic name - with some Subscriptions
    Subscriptions:                # List of Subscriptions to create for this topic (queues will be created as required)
    - QueueName: local-queue1   # Queue name
      Raw: false                 # Raw message delivery (true/false)

Then I've executed:

$ docker-compose up -d

$ aws --endpoint-url http://localhost:4100 sns publish --topic-arn local-topic1 --message "this is a test of the GoAws Queue messaging" --message-attributes file://attributes.json

$ aws --endpoint-url http://localhost:4100 sqs receive-message --queue-url http://localhost:4100/local-queue1
{
    "Messages": [
        {
            "Body": "{\"Type\":\"Notification\",\"MessageId\":\"bb642a2c-7a87-42dc-80ce-4cf9c8f673b7\",\"TopicArn\":\"local-topic1\",\"Subject\":\"\",\"Message\":\"this is a test of the GoAws Queue messaging\",\"Timestamp\":\"2018-12-27T19:27:39.012Z\"}",
            "MD5OfMessageAttributes": "b095c6d16871105acb75d59332513337",
            "ReceiptHandle": "fc5068d7-b79b-4c05-b84c-b3e8a93f2ade#449d67ec-2366-4bcc-9bd7-42b3df57f9ee",
            "MD5OfBody": "121bc0ebfe9e0df6836b27074e437d90",
            "MessageId": "fc5068d7-b79b-4c05-b84c-b3e8a93f2ade"
        }
    ]
}

And this is the result with 'real' AWS that has a MessageAttributes key inside $messages[0].body:

$ aws sns publish --topic-arn arn:aws:sns:eu-west-1:373715944720:dev-molange --message "this is a test of the GoAws Queue messaging" --message-attributes file://attributes.json
$ aws sqs receive-message --queue-url https://sqs.eu-west-1.amazonaws.com/373715944720/dev-molange
{
    "Messages": [
        {
            "Body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"510c8b6a-8a7b-5701-b5fc-dff5a1ad7c4b\",\n  \"TopicArn\" : \"arn:aws:sns:eu-west-1:373715944720:dev-molange\",\n  \"Message\" : \"this is a test of the GoAws Queue messaging\",\n  \"Timestamp\" : \"2018-12-27T19:35:50.431Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"jMSvJ2zs+lhY0JiX0MyHDKvXB+J+wru5GzY/blx+iTkv6Y2Lp2EPawjCuQAIA9ryz295lPSeQA55OoIbiQAZKPxh8l95QJ/YKSysA9d0Zh+bJC43qVPu7kjgrMTYh49+9++LuYr4WcTRuxG2HhiFPmyYn8gqaCDKsvf2hsdNHNfezmsfDLPUHg40itFKeUJ/8b6wxeAjk72EjKmGWMA/Sf7IS4ZkhfOI1uLUPrLvmZ+F6UsTXPDuM9LuJ5N+VK1of9/NLNbY/7dfbSaEr609cX6dJoACwQpS1pmEWw36s8gxG2/kimng2NqoiSF19tYQAbaUKNS8gnAwyTqPukCyDg==\",\n  \"SigningCertURL\" : \"https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f9598aa1d9b.pem\",\n  \"UnsubscribeURL\" : \"https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:373715944720:dev-molange:d91ca429-bd08-49f8-8c1e-769616cff72d\",\n  \"MessageAttributes\" : {\n    \"Greeting\" : {\"Type\":\"Binary\",\"Value\":\"SGVsbG8sIFdvcmxkIQ==\"},\n    \"Population\" : {\"Type\":\"Number\",\"Value\":\"1250800\"},\n    \"City\" : {\"Type\":\"String\",\"Value\":\"Any City\"}\n  }\n}",
            "ReceiptHandle": "AQEB8xQNiuad6BHeGWEuM1qNOvbsZNdNgxLWRtw3YaFaBWK60Nj/jOe5lImIwdzQ3MnCVzCEaos2NHBe47IEDbt7Hv07ic0UTcwf5SnDdMh+TivjrI+aOM1h2RGLxZTVNnxEcculcgtvKrXo3rE4m32xtJtjEnfpGzddFlBw6WWDcRyF/Ccr0of8E1itzI58Brsz7QXZxdIeLc3qAYUuz/l7k7AXHQ6zs9i/z8LB+385nhN06pnKfrFSofXgTKtMOOFuh5felLRZIE1mcbfBcIbKd7ygmFpT9i2QVNjBR+5zG0/U3Vgq00tMTsJy0AgrpEISa8fyexOv2Mj7UVaYRZOdPJeLfxDnxgz7Q6dBJ73A86r7AgKHqUpC0oedv9QzUbfl",
            "MD5OfBody": "d126e3c3b440153e722bcf0558f8199a",
            "MessageId": "fa150a4a-00de-416f-8662-f717323542f4"
        }
    ]
}

[p4tin]

@jairovadillo I believe that is because I have not updated the docker file in dockerhub in a very long time. I will do that early tomorrow (my time EST) and then you can try again...

[jairovadillo]

@p4tin I've built the image from master and the result is the same. It looks that AWS is doing something weird when delivering the messages from SNS to SQS and you're not simulating this behaviour.

I'll have a look a your code but I'm not fluent with go... :(

[jairovadillo]

Hey @p4tin, I think it's here: https://github.com/p4tin/goaws/blob/master/app/gosns/gosns.go#L517

When subscription is not raw you are not building a message body with the MessageAttributes keys. :D

[p4tin]

@jairovadillo - ok I found the issue with you help and have a fix now i need to fix all the unit tests but master should have a fix by the end of today.

[jairovadillo]

YEAH! I've docker-built the PR #183 and I know some unit tests are failing but my "integration test" it's working awesome :)

Thanks so much!

[p4tin]

@jairovadillo Sorry it closed on it's own after the merge to master - can you git pull build your docker image again and see if the issue is resolved?

[jairovadillo]

I closed the issue because I saw that the PR was working. I've build the code from your branch and it was ok so I think it's gonna be fine from master.

Anyway I'm rebuilding it from master and I'll share the results with you.

[jairovadillo]

Great! Master works just fine now!