SQS/SNS confusion?

[eslesarchuk]

Hey everyone,

I wanted to clarify a few questions while setting up my S3 logs integration. The Readme says i need an S3 bucket, which publishes events to SNS topic, which publishes to SQS queue. The plugin is picking up from an SQS queue.

However, I'm getting errors regarding SNS:

ERROR [S3Subscriber] Could not read messages from SNS. This is most likely a misconfiguration of the plugin. Going into sleep loop and retrying.

This is clearly a plugin's message.

There's no SNS configuration for the plugin, and there are no SNS actions allowed in the IAM example. The question is: what am I missing in the configuration?

I should also note that S3 allows to publish events directly to SQS, so SNS isn't needed here at all, and this is the way I used to configure the pipeline.

[eslesarchuk]

OK, I did a small research. Here are the example messages:

This is a direct S3-SQS message

{
  "Records": [{
    "eventVersion": "2.0",
    "eventSource": "aws:s3",
    "awsRegion": "us-east-1",
    "eventTime": "2018-09-14T08:10:53.998Z",
    "eventName": "ObjectCreated:Put",
    "userIdentity": {
      "principalId": "AAAAAAAAAAAAAA"
    },
    "requestParameters": {
      "sourceIPAddress": "10.85.35.240"
    },
    "responseElements": {
      "x-amz-request-id": "BBBBBBBBBBBBBBBB",
      "x-amz-id-2": "..."
    },
    "s3": {
      "s3SchemaVersion": "1.0",
      "configurationId": "tf-s3-queue-20180913134036905600000001",
      "bucket": {
        "name": "project-cloudfront-logs",
        "ownerIdentity": {
          "principalId": "DDDDDDDDDDDDDD"
        },
        "arn": "arn:aws:s3:::project-cloudfront-logs"
      },
      "object": {
        "key": "cdn.project.com/EEEEEEEEEEEEEE.2018-09-14-08.34114b8c.gz",
        "size": 8503,
        "eTag": "4753e047c7ce14a9d5e1d333d3d3e083",
        "sequencer": "005B9B6D0DF383EFDF"
      }
    }
  }]
}

And this one is S3-SNS-SQS message (which is the designed way to interact with the graylog's plugin)

{
  "Type": "Notification",
  "MessageId": "6db19eea-e48f-5fbb-a354-cafcfb4b49e3",
  "TopicArn": "arn:aws:sns:us-east-1:000000000000:project_s3_logs_notifications",
  "Subject": "Amazon S3 Notification",
  "Message": "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2018-09-14T20:43:46.903Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AAAAAAAAAAAAAA\"},\"requestParameters\":{\"sourceIPAddress\":\"10.163.220.172\"},\"responseElements\":{\"x-amz-request-id\":\"BBBBBBBBBBBBBBBB\",\"x-amz-id-2\":\"...\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"...\",\"bucket\":{\"name\":\"project-cloudfront-logs\",\"ownerIdentity\":{\"principalId\":\"BBBBBBBBBBBBBBBB\"},\"arn\":\"arn:aws:s3:::project-cloudfront-logs\"},\"object\":{\"key\":\"cdn.project.com/EEEEEEEEEEEEEE.2018-09-14-20.cb6ab2d8.gz\",\"size\":719,\"eTag\":\"b64c2807eaa90c2f6fa91a7c9cb53599\",\"sequencer\":\"005B9C1D82D1299CD3\"}}}]}",
  "Timestamp": "2018-09-14T20:43:48.007Z",
  "SignatureVersion": "1",
  "Signature": "...",
  "SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d277f9598aa1d9b.pem",
  "UnsubscribeURL": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:000000000000:project_s3_logs_notifications:57954447-e125-4d6f-9cf7-d078be57f44f"
}

The only difference is that SNS wraps the message, the messages have the same structure. I think it's worth adding support for direct S3-SQS messaging, omitting the SNS configuration, which isn't really straightforward.