peteristhegreat commented 8 months ago

Terraform Core Version

1.6.6

AWS Provider Version

5.31.0

Affected Resource(s)

aws_s3_object or aws_s3_bucket_object doesn't work with the optional region ap-east-1

Expected Behavior

Accessing s3 buckets in an optional region should work.

Actual Behavior

It throws an error: An error occurred (IllegalLocationConstraintException) when calling the GetObject operation: The ap-east-1 location constraint is incompatible for the region specific endpoint this request was sent to.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

provider "aws" {
  region                  = "ap-east-1"
  profile                  = "default"
}

variable "sumologic_lambda_s3_bucket_from_region" {
  type = map(string)
  description = "Mapping for the Sumo Logic region string from specified region."
  default = {
    us-east-1      = "appdevzipfiles-us-east-1"
    us-east-2      = "appdevzipfiles-us-east-2"
    us-west-1      = "appdevzipfiles-us-west-1"
    us-west-2      = "appdevzipfiles-us-west-2"
    ap-northeast-1 = "appdevzipfiles-ap-northeast-1"
    ap-northeast-2 = "appdevzipfiles-ap-northeast-2"
    ap-east-1      = "appdevzipfiles-ap-east-1"
    ap-south-1     = "appdevzipfiles-ap-south-1"
    ap-southeast-1 = "appdevzipfiles-ap-southeast-1"
    ap-southeast-2 = "appdevzipfiles-ap-southeast-2"
    eu-north-1     = "appdevzipfiles-eu-north-1"
    eu-central-1   = "appdevzipfiles-eu-central-1"
    eu-south-1     = "appdevzipfiles-eu-south-1"
    eu-west-1      = "appdevzipfiles-eu-west-1"
    eu-west-2      = "appdevzipfiles-eu-west-2"
    eu-west-3      = "appdevzipfiles-eu-west-3"
    af-south-1     = "appdevzipfiles-af-south-1"
    ca-central-1   = "appdevzipfiles-ca-central-1"
    me-south-1     = "appdevzipfiles-me-south-1"
    sa-east-1      = "appdevzipfiles-sa-east-1"
  }
}

resource "aws_lambda_function" "sumo_cw_process_dlq_lambda" {
  count = var.sumologic_cwl_lambda ? 1 : 0
  depends_on    = [aws_iam_role.sumo_cwl_lambda_execution_role[0], aws_sqs_queue.sumo_cw_dead_letter_queue[0]]
  function_name = "SumoCWProcessDLQLambda"
  s3_bucket     = var.sumologic_lambda_s3_bucket_from_region[var.region]
  s3_key        = "cloudwatchlogs-with-dlq.zip"
  role          = aws_iam_role.sumo_cwl_lambda_execution_role[0].arn
  timeout       = 300
  handler       = "DLQProcessor.handler"
  runtime       = "nodejs14.x"
  memory_size   = 128

  dead_letter_config {
    target_arn = aws_sqs_queue.sumo_cw_dead_letter_queue[0].arn
  }

  environment {
    variables = {
      SOURCE_CATEGORY_OVERRIDE = "${var.environment}/${var.customer_abbr}/cwl"
      SUMO_ENDPOINT            = sumologic_http_source.http_source.url
      LOG_FORMAT               = var.sumologic_log_format
      INCLUDE_LOG_INFO         = var.sumologic_include_log_group_info
      LOG_STREAM_PREFIX        = var.sumologic_log_stream_prefix
      NUM_OF_WORKERS           = var.sumologic_num_workers
      TASK_QUEUE_URL           = [https://sqs.${var.region}.amazonaws.com/${data.aws_caller_identity.current.account_id}/${aws_sqs_queue.sumo_cw_dead_letter_queue[0].name}](https://sqs.$%7bvar.region%7d.amazonaws.com/$%7bdata.aws_caller_identity.current.account_id%7d/$%7baws_sqs_queue.sumo_cw_dead_letter_queue%5b0%5d.name%7d)
    }
  }
}

Steps to Reproduce

Even though these are awscli calls, they are relevant for terraform, since it produces the same error very concisely.

The easiest way to see this happen is to enable an optional region in both your IAM account and in a secondary account, then try some things from the CLI.

aws --profile account-with-ap-east-1-enabled --region ap-east-1 s3 cp s3://appdevzipfiles-ap-east-1s/cloudwatchlogs-with-dlq.zip .

The above works.

Try again without the region specifier (assuming your IAM account is not in ap-east-1)

aws --profile account-with-ap-east-1-enabled s3 cp s3://appdevzipfiles-ap-east-1s/cloudwatchlogs-with-dlq.zip .
download failed: s3://appdevzipfiles-ap-east-1s/cloudwatchlogs-with-dlq.zip to ./cloudwatchlogs-with-dlq.zip An error occurred (IllegalLocationConstraintException) when calling the GetObject operation: The ap-east-1 location constraint is incompatible for the region specific endpoint this request was sent to.

The lack of specifying the region for this s3 bucket breaks the call.

Debug Output

No response

Panic Output

╷
│ Error: creating Lambda Function (SumoCWProcessDLQLambda): operation error Lambda: CreateFunction, https response error StatusCode: 400, RequestID: REDACTED, InvalidParameterValueException: Error occurred while GetObject. S3 Error Code: IllegalLocationConstraintException. S3 Error Message: The us-west-2 location constraint is incompatible for the region specific endpoint this request was sent to.
│
│   with module.stack.aws_lambda_function.sumo_cw_process_dlq_lambda[0],
│   on .terraform/modules/stack/sumologic.tf line 168, in resource "aws_lambda_function" "sumo_cw_process_dlq_lambda":
│  168: resource "aws_lambda_function" "sumo_cw_process_dlq_lambda" {
│
╵

Important Factoids

This has been an odd week of trying to get stuff working in ap-east-1. I've set up a half a dozen other regions in the past and now when working with an "optional" region, and one in china, I am hitting corner cases left and right.

There should be an optional argument to specify the region for awss3* for the region, or have a flag that forces the region argument into the interaction for regions with weird, abnormal geofencing rules.

References

https://github.com/hashicorp/terraform-provider-aws/issues/13638

https://github.com/hashicorp/terraform-provider-aws/issues/31300

https://github.com/aws/aws-cli/issues/8413#issuecomment-1879079322

Would you like to implement a fix?

None

github-actions[bot] commented 8 months ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

ewbankkit commented 8 months ago

Although I can't find anything in the S3 documentation, it seems (through various hints) that if the bucket's Region is not the currently configured Region and is an opt-in Region then the Region needs to be specified in the AWS API call.