search

cloudposse / terraform-aws-tfstate-backend

Terraform module that provision an S3 bucket to store the `terraform.tfstate` file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption.
https://cloudposse.com/accelerate
Apache License 2.0
408 stars 177 forks source link

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. #54

Closed organicnz closed 3 years ago

organicnz commented 4 years ago

Describe the Bug

Getting an error Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again. when running terraform apply -auto-approve.

Also, it asks to Enter the value of a region for S3, however, it's already in vars. World be nice to automate this step as well :)

Environment:

Steps to Reproduce

terraform apply -auto-approve

var.region
  AWS Region the S3 bucket should reside in

  Enter a value: us-west-2     

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2

aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=terraform-state-lock]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=eg-test-terraform-state-lock]
data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
module.terraform_state_backend.data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
aws_s3_bucket.default: Creating...
module.terraform_state_backend.aws_s3_bucket.default: Creating...

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        status code: 409, request id: DAE8503E57F632E7, host id: LLcTL4YZN1mIOL8mJzBL9y5d4YJKs/tt7CHh5Ks63naqarYBD/RC8Nnqzs7FQ9mRaRMsdQUhmgs=

  on main.tf line 145, in resource "aws_s3_bucket" "default":
 145: resource "aws_s3_bucket" "default" {

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        status code: 409, request id: 6DF9DDE094778C9C, host id: 2m8a4gn4qbQ4xwZpNaU1/vmCQuHFM+pV1EQA58+45JSmJ7FVxixXIoFigKhg5KXIrOCVqb7L8+4=

  on .terraform/modules/terraform_state_backend/main.tf line 124, in resource "aws_s3_bucket" "default":
 124: resource "aws_s3_bucket" "default" {
jamengual commented 4 years ago

@organicnz is you did not destroy the bucket before creating this is going to happen. this is not a bug AFAIK

organicnz commented 4 years ago

@organicnz is you did not destroy the bucket before creating this is going to happen. this is not a bug AFAIK

Just tried it again and it spews the same error :)

terraform destroy -refresh=false
var.region
  AWS Region the S3 bucket should reside in

  Enter a value: us-west-2

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Destroy complete! Resources: 0 destroyed.
terraform apply -auto-approve   
var.region
  AWS Region the S3 bucket should reside in

  Enter a value: us-west-2

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2

data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
module.terraform_state_backend.data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Creating...
aws_dynamodb_table.with_server_side_encryption[0]: Creating...
module.terraform_state_backend.aws_s3_bucket.default: Creating...
aws_s3_bucket.default: Creating...
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Still creating... [10s elapsed]
aws_dynamodb_table.with_server_side_encryption[0]: Still creating... [10s elapsed]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Creation complete after 10s [id=eg-test-terraform-state-lock]
aws_dynamodb_table.with_server_side_encryption[0]: Creation complete after 12s [id=terraform-state-lock]

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        status code: 409, request id: 4A229001A6A1EA6C, host id: LlszF09zWpcTVFKP0IEWt52qEFMjHbFSwtfU8W0S7scUK53fjrG2ACyY7QDwGgFkvmMh6t9uuj8=

  on main.tf line 145, in resource "aws_s3_bucket" "default":
 145: resource "aws_s3_bucket" "default" {

Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        status code: 409, request id: 59E964BDAA48DCDD, host id: aKpwR55Nh0aRFwAIv6Y8QQkMcnmZWSUFOLuvL5nmO4Bh+eOSAayCmXE1zrIHT66Faah1/VfglMA=

  on .terraform/modules/terraform_state_backend/main.tf line 124, in resource "aws_s3_bucket" "default":
 124: resource "aws_s3_bucket" "default" {
jamengual commented 4 years ago

Hi @organicnz This is normal behavior with S3 buckets, when buckets have deleted the names takes some time to be released so that it can be reused, this is not a bug on the module or terraform, this is how the AWS S3 api works

organicnz commented 4 years ago

Hi @organicnz This is normal behavior with S3 buckets, when buckets have deleted the names takes some time to be released so that it can be reused, this is not a bug on the module or terraform, this is how the AWS S3 api works

Yeah, looks like it is, still cleaning up S3 buckets in region us-west-2, but when it passed that step in us-west-1 or us-east-1 it drops another error with coalescelist. Someone mentioned that it should rely on the conditional operator on a thread. 

terraform apply -auto-approve                                                   

var.region
  AWS Region the S3 bucket should reside in

  Enter a value: us-west-1

provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-1

module.terraform_state_backend.data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
data.aws_iam_policy_document.prevent_unencrypted_uploads[0]: Refreshing state...
aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=terraform-state-lock]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=eg-test-terraform-state-lock]

Error: Error in function call

  on main.tf line 255, in data "template_file" "terraform_backend_config":
 255:       coalescelist(
 256: 
 257: 
 258: 
    |----------------
    | aws_dynamodb_table.with_server_side_encryption is empty tuple
    | aws_dynamodb_table.without_server_side_encryption is empty tuple

Call to function "coalescelist" failed: no non-null arguments.

Error: Error in function call

  on .terraform/modules/terraform_state_backend/main.tf line 234, in data "template_file" "terraform_backend_config":
 234:       coalescelist(
 235: 
 236: 
 237: 
    |----------------
    | aws_dynamodb_table.with_server_side_encryption is empty tuple
    | aws_dynamodb_table.without_server_side_encryption is empty tuple

Call to function "coalescelist" failed: no non-null arguments.
jamengual commented 4 years ago

are you cleaning the .terraform dir and such when switching to another region? this is to create the bucket for the state to be stored so it should not exist in the state until is created.

I will assume you switched to another region because you do not have a state so in that case, you need to reinitialize the project which can be done by rm -rf .terraform and then terraform init and then apply or plan

organicnz commented 4 years ago

are you cleaning the .terraform dir and such when switching to another region? this is to create the bucket for the state to be stored so it should not exist in the state until is created.

I will assume you switched to another region because you do not have a state so in that case, you need to reinitialize the project which can be done by rm -rf .terraform and then terraform init and then apply or plan

Thank you, I haven't cleaned it up before you mentioned and my deep apologies that it's not a bug as it was labelled initially. That's so weird it shows empty S3 storage, but the issue doesn't disappear anywhere. Should this error be rather addressed to AWS? 

#Error: Error creating S3 bucket: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        status code: 409, request id: C83D6B33B66E5AD0, host id: nsXFCzCdd2+HJUyPrrHIQokq735AEYzTntQPdNUY7OOBF+GSd5WrljJKto7LyyThoCsBCIt3LIQ=

  on .terraform/modules/terraform_state_backend/main.tf line 124, in resource "aws_s3_bucket" "default":
 124: resource "aws_s3_bucket" "default" {
ngtuna commented 4 years ago

Any conclusion ? I got same weird BucketAlreadyExists error

organicnz commented 4 years ago

Any conclusion ? I got same weird BucketAlreadyExists error

Hi mate, have you tried to create a new unique S3 bucket? I tried on a different AWS account and it helped to resolve this issue :)

sev3ryn commented 3 years ago

I have same issue - I'm creating bucket that never existed before with just random name but receive Error creating S3 bucket: BucketAlreadyExists

abulava commented 3 years ago

@sev3ryn Note that S3 bucket names must be globally unique amongst all AWS customers.

sev3ryn commented 3 years ago

yes, thanks, thats was it. Interesting what is aws reasoning behind that if bucket is not even public :)

ardarm commented 3 years ago

To make your bucket name unique, you can add random string after attributes. In my case, i define it like this

resource "random_string" "random" {
  length  = 12
  upper   = false
  number  = false
  lower   = true
  special = false
}

module "terraform_state_backend" {
  source = "cloudposse/tfstate-backend/aws"
  # Cloud Posse recommends pinning every module to a specific version
  version    = "0.33.1"
  namespace  = "eg"
  stage      = "test"
  name       = "terraform"
  attributes = ["state-${random_string.random.result}"]

  terraform_backend_config_file_path = "."
  terraform_backend_config_file_name = "backend.tf"
  force_destroy                      = false
}

This is more easier than searching new unique name since S3 bucket name is globally unique.

nitrocode commented 3 years ago

This doesn't seem like an issue with this module as this module cannot ensure that the bucket name is unique globally

tusharrverma16 commented 3 years ago

@sev3ryn Note that S3 bucket names must be globally unique amongst all AWS customers.

thanks for reminding, it helped :) cheers!

dondeswetha commented 10 months ago

Error: creating S3 Bucket (swethabuckets7128834): operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: 08J2B14W7G3ZB0D0, HostID: CegKwKnfuzghXHKd7N0RI7VlOVzYUVNErrFDzhtg/+hzWMs8KUF3Jeq1m9PEydPJMx6nuSzQZsFYEjICOjWsJw==, BucketAlreadyOwnedByYou:

Getting above error while changing tfstate file to remote.