TestAccAWSS3BucketNotification fails due to a casing issue in FilterRules

[thrau]

Reproducible with the attached terraform config.

When it is applied to localstack, aws s3api get-bucket-notification-configuration --bucket tf-test-notification-queue the filter rules return: "FilterRules": [{"Name": "prefix", "Value": "tf-acc-test/"}, {"Name": "suffix", "Value": ".mp4"} ]

wheres when it is applied to AWS it returns: "FilterRules": [{"Name": "Prefix", "Value": "tf-acc-test/"}, {"Name": "Suffix", "Value": ".mp4"} ]

however, from the logs (run with TF_LOG=debug terraform apply) in both cases the following XML config is applied to the PUT request: <NotificationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><QueueConfiguration><Filter><S3Key><FilterRule><Name>prefix</Name><Value>tf-acc-test/</Value></FilterRule><FilterRule><Value>.mp4</Value><Name>suffix</Name></FilterRule></S3Key></Filter><Id>thomas-test-notification-queue</Id><Queue>arn:aws:sqs:us-east-1:SNIPPED:tf-test-notification-queue</Queue><Event>s3:ObjectCreated:*</Event><Event>s3:ObjectRemoved:Delete</Event></QueueConfiguration></NotificationConfiguration>

The only similar issue I could find was https://github.com/aws/aws-sdk/issues/389 However not sure how the issues could be related, since in both cases we're using, through terraform, the aws-sdk-go to put the config, and the aws cli to read the config.

Terraform config to reproduce manually:

provider "aws" {
  access_key                  = "test"
  secret_key                  = "test"
  region                      = "us-east-1"
  s3_force_path_style         = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    ec2            = "http://localhost:4566"
    es             = "http://localhost:4566"
    elasticache    = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    rds            = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    route53        = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

data "aws_partition" "current" {}

resource "aws_sqs_queue" "queue" {
  name = "tf-test-notification-queue"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:${data.aws_partition.current.partition}:sqs:*:*:tf-test-notification-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_s3_bucket.bucket.arn}"
        }
      }
    }
  ]
}
POLICY
}

resource "aws_s3_bucket" "bucket" {
  bucket = "tf-test-notification-bucket"
  acl    = "public-read"
}

resource "aws_s3_bucket_notification" "notification" {
  bucket = aws_s3_bucket.bucket.id

  queue {
    id        = "tf-test-notification-queue"
    queue_arn = aws_sqs_queue.queue.arn

    events = [
      "s3:ObjectCreated:*",
      "s3:ObjectRemoved:Delete",
    ]

    filter_prefix = "tf-acc-test/"
    filter_suffix = ".mp4"
  }
}

[thrau]

looks like they should always come out upper case. when trying to replicate the behavior by running aws cli commands against AWS, it's the same behavior.

here are some commands to test it:

aws s3 mb s3://tf-acc-test-bucket
aws sqs create-queue --queue-name tf-acc-test-queue

BUCKET_ARN="arn:aws:s3:*:*:tf-acc-test-bucket"
QUEUE_URL=$(aws sqs list-queues --queue-name-prefix "tf-acc-test-queue" | jq -j '.QueueUrls[0]')
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url $QUEUE_URL --attribute-names QueueArn | jq -j '.Attributes.QueueArn')

# set access policy of queue to allow S3 to send a message
printf -v POLICY '{"Policy": "{\"Version\": \"2012-10-17\", \"Id\": \"tf-acc-test-notification-policy\", \"Statement\": [{\"Sid\": \"example-statement-ID\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"s3.amazonaws.com\"}, \"Action\": [\"SQS:SendMessage\"], \"Resource\": \"%s\", \"Condition\": {\"ArnLike\": { \"aws:SourceArn\": \"%s\" } } } ] }"}' "$QUEUE_ARN" "$BUCKET_ARN"

aws sqs set-queue-attributes --queue-url ${QUEUE_URL} --attributes "$POLICY"

# notification configuration
printf -v NOTIFICATION_CONFIG '{"QueueConfigurations": [{"Id": "tf-acc-test-notification", "QueueArn": "%s", "Events": ["s3:ObjectCreated:*", "s3:ObjectRemoved:Delete"], "Filter": {"Key": {"FilterRules": [{"Name": "prefix", "Value": "tf-acc-test/"}, {"Name": "Suffix", "Value": ".mp4"} ] } } } ] }' "$QUEUE_ARN"

aws s3api put-bucket-notification-configuration --bucket tf-acc-test-bucket --notification-configuration "$NOTIFICATION_CONFIG"

aws s3api get-bucket-notification-configuration --bucket tf-acc-test-bucket

# cleanup
aws s3 rb s3://tf-acc-test-bucket
aws sqs delete-queue --queue-url "${QUEUE_URL}"

it comes out as:

{
    "QueueConfigurations": [
        {
            "Id": "thomas-tf-acc-test-notification",
            "QueueArn": "<QUEUE ARN>",
            "Events": [
                "s3:ObjectCreated:*",
                "s3:ObjectRemoved:Delete"
            ],
            "Filter": {
                "Key": {
                    "FilterRules": [
                        {
                            "Name": "Prefix",
                            "Value": "tf-acc-test/"
                        },
                        {
                            "Name": "Suffix",
                            "Value": ".mp4"
                        }
                    ]
                }
            }
        }
    ]
}