SNS MessageAttribute are mixed up

[matthias-Q]

Hi, I am trying to publish a SNS message with four MessageAttributes. The key/value pairs in the list that I provide to sns$publish differs from the one received from the topic.

Here is a minimal example:

send_message <- function(topic_arn, dest, s3paths, bucket, prefix, message_type) {

  msg_attributes = list(
    files=list(DataType="String.Array", StringValue=paste0(s3paths, collapse = ',')),
    parquet_schema=list(DataType="String", StringValue=paste0("s3://", bucket, "/schemas/", prefix, ".json")),
    message_type=list(DataType="String", StringValue=message_type),
    platform=list(DataType="String", StringValue=dest)
  )

  sns <- paws::sns()
  sns$publish(
    Message = "my message",
    TopicArn = topic_arn,
    MessageStructure="string",
    MessageAttributes=msg_attributes
  )
  return(msg_attributes)
}

bucket = "mybucket"
s3paths = c(
    "a", "b", "c"
)

topic_arn = "arn:aws:sns:eu-central-1:12345678:my_topic"

prefix = "my_prefix"
message_type = "my_message_type"

send_message(
    topic_arn,
    "platform_A",
    s3paths,
    bucket,
    prefix,
    message_type
)

This is the message that I receive:

{
  "Type" : "Notification",
  "MessageId" : "...",
  "TopicArn" : "...",
  "Message" : "my message",
  "Timestamp" : "2023-11-10T12:38:19.337Z",
  "SignatureVersion" : "1",
  "Signature" : "...",
  "SigningCertURL" : "...",
  "UnsubscribeURL" : "...",
  "MessageAttributes" : {
    "files" : {"Type":"String.Array","Value":"/path/a,/path/b,/path/c"},
    "parquet_schema" : {"Type":"String","Value":"my_message_type"},
    "message_type" : {"Type":"String","Value":"s3://mybucket/schemas/my_prefix.json"},
    "platform" : {"Type":"String","Value":"south"}
  }
}

As you can see parquet_schema and message_type have swapped their values.

[DyfanJones]

Ah sorry about that, I will look into it shortly.

[DyfanJones]

@matthias-Q what version of paws and paws.common are you using please? :)

[matthias-Q]

packageVersion("paws") [1] ‘0.4.0’ packageVersion("paws.common") [1] ‘0.6.3’

[DyfanJones]

@matthias-Q I think I have found a bug in query_parse it seems to want to order stuff. This is causing the bug. I will see how easy of a fix it is :D

[DyfanJones]

OK I have a fix :)

remotes::install_github("dyfanjones/paws/paws.common", ref = "fix_query_mapping")

send_message <- function(topic_arn, dest, s3paths, bucket, prefix, message_type) {

  msg_attributes = list(
    files=list(DataType="String.Array", StringValue=paste0(s3paths, collapse = ',')),
    parquet_schema=list(DataType="String", StringValue=paste0("s3://", bucket, "/schemas/", prefix, ".json")),
    message_type=list(DataType="String", StringValue=message_type),
    platform=list(DataType="String", StringValue=dest)
  )

  sns <- paws::sns()
  sns$publish(
    Message = "my message",
    TopicArn = topic_arn,
    MessageStructure="string",
    MessageAttributes=msg_attributes
  )
  return(msg_attributes)
}

bucket = "mybucket"
s3paths = c(
  "a", "b", "c"
)

topic_arn = "arn:aws:sns:eu-central-1:12345678:my_topic"

prefix = "my_prefix"
message_type = "my_message_type"

# options(paws.log_level = 3)
resp <- send_message(
  topic_arn,
  "platform_A",
  s3paths,
  bucket,
  prefix,
  message_type
)

resp |> jsonlite::toJSON(pretty = T, auto_unbox = T)
#> {
#>   "files": {
#>     "DataType": "String.Array",
#>     "StringValue": "a,b,c"
#>   },
#>   "parquet_schema": {
#>     "DataType": "String",
#>     "StringValue": "s3://mybucket/schemas/my_prefix.json"
#>   },
#>   "message_type": {
#>     "DataType": "String",
#>     "StringValue": "my_message_type"
#>   },
#>   "platform": {
#>     "DataType": "String",
#>     "StringValue": "platform_A"
#>   }
#> }

client <- paws::s3()
message <- client$get_object(
  Bucket = bucket, Key = sprintf("schemas/%s.json", prefix)
)$Body |> rawToChar() |> jsonlite::fromJSON(simplifyDataFrame = F)

message$Records[[1]]$Sns$MessageAttributes$parquet_schema$Value
#> [1] "s3://mybucket/schemas/my_prefix.json"

^{Created on 2023-11-10 with reprex v2.0.2}

Note: for test purposes my SNS just outputted the message the AWS S3 using a basic AWS Lambda.

Sadly this has just missed the paws.common 0.6.4 cran release. @matthias-Q are you ok to use r-universe/ dev version of now until paws.common 0.6.5 is released?

[matthias-Q]

Wow, thanks for the quick fix. How often are these cran releases? We could easily wait for a week. I will test it right away on my machine, but on the production system we would like to have a release version.

[Update]: tested the dev version and I confirm that it now works as expected. Thanks!

[DyfanJones]

Usually I don't want to pester the cran guys with weekly releases.

I have requested paws.common-0.6.4 cran to be cancelled so that I can get this fix into the latest paws.common release (over the weekend hopefully). I will give an update as soon as I hear back from the cran

[DyfanJones]

Closing this ticket as paws.common 0.6.4 has been released to cran.