Cannot Parse SNS Message from FIFO SNS -> FIFO SQS

marclyo commented 1 year ago

Describe the bug

When attempting to parse a message with: var snsMessage = AwsSnsMessage.ParseMessage(message.Body); https://docs.aws.amazon.com/sdkfornet/latest/apidocs/items/MSNSUtilMessageParseMessageStringNET45.html

This error is received: Value cannot be null. (Parameter 'uriString'): ArgumentNullException

This error only occurs when the message has come via FIFO SNS and FIFO SQS, like this:

Message publisher -> FIFO SNS Topic -> FIFO SQS Queue -> Message Consumer ParseMessage()

as the message payload looks like this:

{
    "Type": "Notification",
    "MessageId": "19848d6e-64b6-5fbb-906e-d588078ccac0",
    "TopicArn": "arn*****",
    "Subject": "Subject",
    "Message": "redacted to make it easier to read",
    "Timestamp": "2023-07-06T13:24:10.934Z",
    "UnsubscribeURL": "redacted to make it easier to read"
}

i.e The SigningCertURL entry is missing

When the non-FIFO Topic and Queue are used the ParseMessage() is successful.

Message publisher -> SNS -> SQS -> Message Consumer ParseMessage()

as the message payload looks like this (i.e includes SigningCertURL):

{
    "Type": "Notification",
    "MessageId": "19848d6e-64b6-5fbb-906e-d588078ccac0",
    "TopicArn": "arn*****",
    "Subject": "Subject",
    "Message": "redacted to make it easier to read",
    "Timestamp": "2023-07-06T13:24:10.934Z",
    "SignatureVersion": "1",
    "Signature": "redacted to make it easier to read",
    "SigningCertURL": "redacted to make it easier to read",
    "UnsubscribeURL": "redacted to make it easier to read"
}

Expected Behavior

The message is parsed successfully.

Current Behavior

This error is received: Value cannot be null. (Parameter 'uriString'): ArgumentNullException

Reproduction Steps

Set up:

a FIFO SNS Topic
a FIFO SQS Queue
Subscribe the FIFO SQS to the FIFO SNS Topic (with raw_delivery=false)

Steps to reproduce:

Send a message to the FIFO SNS Topic
Retrieve message using AmazonSQSClient.ReceiveMessage()
Parse message using Message.ParseMessag()
- https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SNS/MMessageParseMessageString.html

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.SimpleNotificationService 3.7.3.76

Targeted .NET Platform

.NET 6

Operating System and version

Linux

ashishdhingra commented 1 year ago

Hi @marclyo,

Good morning.

Thanks for reporting the issue. Could you please share sample code to reproduce the issue? Are you using Amazon.SimpleNotificationService.Util.Message.ParseMessage() to parse the message? I'm getting SignatureVersion is missing error.

For enabling SignatureVersion, I used AWS CLI to set this attribute (refer Verifying the signatures of Amazon SNS messages). Looks like the SignatureVersion attribute in message body JSON is only set for HTTP/HTTPS endpoint, refer How do I verify the authenticity of Amazon SNS messages that are sent to HTTP and HTTPS endpoints?. Whereas, FIFO SNS topic only allows subscriptions with Amazon SQS protocol. So just curious about your setup and reproduction code.

Thanks, Ashish

github-actions[bot] commented 1 year ago

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

marclyo commented 1 year ago

Hi @ashishdhingra

Sure, I can provide some sample code to reproduce the issue.

Please use this CloudFormation template to create the FIFO SNS -> FIFO SQS Resources:

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  SnsTopic:
    Type: AWS::SNS::Topic
    Properties:
      FifoTopic: True

  Queue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: True

  SqsQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Id: MyQueuePolicy
        Statement:
        - Sid: Allow-SNS-SendMessage
          Effect: Allow
          Principal: "*"
          Action:
            - sqs:SendMessage
          Resource: !GetAtt Queue.Arn
          Condition:
            ArnEquals:
              aws:SourceArn: !Ref SnsTopic
      Queues:
      - !Ref Queue

  SnsSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Protocol: sqs
      Endpoint: !GetAtt Queue.Arn
      TopicArn: !Ref SnsTopic

Outputs:
  SqsQueueArn:
    Value: !Ref Queue

Create the CloudFormtation stack: aws cloudformation create-stack --stack-name test-fifo-subscription --template-body file://template.yaml

After the stack has been created then Publish a sample message to the FIFO SNS Topic:

aws sns publish --topic-arn arn:aws:sns:eu-west-1:826949559676:test-fifo-subscription-SnsTopic-Z20uv52pltvj.fifo --message "test message" --message-group-id 123 --message-deduplication-id 456

The message will now be available in the FIFO SQS Queue (as it is subscribed to the FIFO SNS Topic)

Then, using the aws-sdk-net:

Receive the file from SQS Queue using

AmazonSQSClient.ReceiveMessage()

Parse the message, using Amazon.SimpleNotificationService.Util.Message.ParseMessage()

AwsSnsMessage.ParseMessage(message.Body)

You should receive this error

Value cannot be null. (Parameter 'uriString'): ArgumentNullException

This is because the SNS Message has a body like this:

{
    "Type": "Notification",
    "MessageId": "19848d6e-64b6-5fbb-906e-d588078ccac0",
    "TopicArn": "arn*****",
    "Subject": "Subject",
    "Message": "redacted to make it easier to read",
    "Timestamp": "2023-07-06T13:24:10.934Z",
    "UnsubscribeURL": "redacted to make it easier to read"
}

I.e. it doesn't have a SigningCertURL key and the aws-sdk-net always tries to validate this value - https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/SimpleNotificationService/Custom/Util/Message.cs#L79

ashishdhingra commented 1 year ago

May be FIFO queue has a different message format. Needs investigation.

ashishdhingra commented 1 year ago

After setting up SQS FIFO queue using customer's CloudFormation template, we receive message with the following JSON body:

{
  "Type" : "Notification",
  "MessageId" : "06192b98-eaa2-56d5-acb8-76c513edd03e",
  "SequenceNumber" : "10000000000000003000",
  "TopicArn" : "arn:aws:sns:us-east-2:<<REDACTED>>:test-fifo-subscription-SnsTopic-v5q7qbvXDlBy.fifo",
  "Message" : "test message",
  "Timestamp" : "2023-08-21T23:45:19.931Z",
  "UnsubscribeURL" : "https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:<<REDACTED>>:test-fifo-subscription-SnsTopic-v5q7qbvXDlBy.fifo:590f8d18-c48e-4c32-8355-40f262eaddef"
}

Executing Amazon.SimpleNotificationService.Util.Message.ParseMessage(message.Body) throws below error:

Amazon.Runtime.AmazonClientException: SignatureVersion is missing
   at Amazon.SimpleNotificationService.Util.Message.ValidateSignatureVersion(String signatureVersion)

Also notice that some fields in FIFO SNS message are different from ones bing parsed in Amazon.SimpleNotificationService.Util.Message.ParseMessage().

dinesh-chander commented 8 months ago

Looks like these fields are not available for SNS FIFO

SignatureVersion
Signature
SigningCertURL

Screenshot 2023-12-29 at 10 35 40 PM

Also, found this commit in one of the other projects where these fields were made optional.

https://github.com/aws-powertools/powertools-lambda-python/pull/1606 https://github.com/aws-powertools/powertools-lambda-python/releases/tag/v1.31.1

aws / aws-sdk-net