aws_s3_buckets datasource

[ThinkBriK]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

We have a missing datasource form AWS S3 buckets : you cannot get a list of your account buckets via terraform at the moment. The only existing datasource is _aws_s3bucket and it does only support a single bucket by name.

We would need a filter to retrieve a bucket arns list, especially usefull to harden S3 VPC endpoints and prevent data exfiltration (using a policy for our account bucket only for example).

New or Affected Resource(s)

• data aws_s3_buckets

Potential Terraform Configuration

data aws_s3_buckets "supersecret_project_bucket" {
  filter {
    name   = "tag:Project"
    values = ["supersecret"]
  } 
  account = 111111111111
}

data "aws_iam_policy_document" "s3_restrict_to_account_buckets" {
  statement {
    sid = "S3RestrictWriteToSupersecretProjectBuckets"
    principals {
      identifiers = ["*"]
      type        = "*"
    }
    actions = [
      "s3:PutObject",
    ]
    effect    = "Allow"
    resources = formatlist("%s/*", data.aws_s3_buckets.arns)
  }
}

resource "aws_vpc_endpoint" "s3-gateway" {
  service_name = "com.amazonaws.${data.aws_region.current.name}.s3"
  vpc_id       = aws_vpc.vpc.id
  policy = data.awsaws_iam_policy_document.s3_restrict_to_account_buckets.json
}

• 0000

[ryanpodonnell1]

This works as a workaround:

data "external" "s3_buckets" {
  program = ["${path.root}/get_buckets.sh"]
}

output "buckets" {
  value = jsondecode(data.external.s3_buckets.result.buckets).Buckets[*].Name
}

get_buckets.sh:

#!/bin/bash
set -e

buckets=$(aws s3api list-buckets | jq . -r )

jq -n --arg buckets "$buckets" '{"buckets":$buckets}'

This creates a string list of bucket names if you use the value from the output or can be assigned to local

[simonweil]

This data source could be very useful

[fotto1]

Same for me 👍

[rdhatt]

There are other issues that talk about how useful aws_s3_buckets would be, the last one includes a PR!

The dev got frustrated at the lack of interest from maintainers and closed the issue unfortunately.