How to include MessageAttributes with sns/publish ?

[keeth]

Hello, I have an SQS queue suscribed to an SNS, and am using fink-nottle at both ends.

In order to use the sqs.tagged decoding facility I need to add the fink-nottle-tag=json message attribute to the SNS message, which will get passed through to SQS. I have been able to get this working by explicitly setting the MessageAttributes.member.1.* KV pairs, just wondering if there's an easier way or if support could be added to the sns publish function for an attrs param map.

I got it working doing the following...

(fink-nottle.sns/publish-topic!! creds topic message-body
{"MessageAttributes.member.1.Name" "fink-nottle-tag"
  "MessageAttributes.member.1.Value.DataType" "String"
  "MessageAttributes.member.1.Value.StringValue" "json"})

Thanks!

Also I built a utility on top of fink-nottle to help provision and connect SNS and SQS queues, see https://github.com/sprightco/chard

[moea]

Keith,

Neat! Chard looks super-helpful - I'll put a link in the fink-nottle README. What kind of stuff are you transmitting, out of curiosity?

So, all the evidence points to me having completely blanked on the SNS message attribute facility. It's documented incorrectly (as "MessageAttributes.entry..." - though who knows, maybe that works, too).

I pushed fink-nottle 0.4.4-SNAPSHOT, which would allow your invocation above to become:

(fink-nottle.sns/publish-topic!! creds topic message
 {:attrs {:fink-nottle-tag "json"}})

Let me know if it works out for you, and I'll promote the release. One thing which tripped me up during testing: even if the subscription has raw message delivery enabled, unless the message type is raw/not-json (i.e. you pass a string for "message", above, instead of {:sqs "string"}), the attributes won't be delivered over SQS. So, e.g. if you wanted to publish a single message with different bodies (i.e. one SQS-specific, one for HTTP) but identical attributes, that doesn't seem possible - at least as far as receiving the attributes via SQS. You probably already know that, it just slowed me down a bit.

Do you think it's worth including a convenience function for publishing tagged SNS messages, so people don't have to burrow to figure out what to do? Like

(sns/publish-tagged!! creds topic tag string-message)

?

[keeth]

Thanks for your reply @moea,

I'll try the snapshot. Yes, it is interesting that the message attributes do not propagate for json-structured messages.. I did not run into that as I am not using that feature. Yes I've found Amazon's SNS/SQS docs to be quite confusing while debugging/testing these issues.

I am playing with Slack bots, using a microservice arch with SNS and SQS for inter-service communication. So the payloads are small JSON-formatted Slack events and messages.

Re: convenience function, I think ideal would be to document how the 'tagged' concept works. It took me a while to figure out how that worked, and that message attributes were being used to pass the message format. Also valuable might be an example showing how to send a message to a queue via SNS, using fink-nottle at both ends, since there are several things that could trip people up there. A convenience method might be useful too, to help ppl understand what's going on. For my purposes you adding support for :attrs does the trick.

[keeth]

Confirmed that the snapshot works, thanks!

[moea]

0.4.4 pushed. Will improve the documentation.