Add support to google_storage_transfer_job in terraform for referencing existing secret from secret manager for aws_s3_data_source

[psdonas]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

Currently there is no support referencing an existing secret from secret manager for aws_s3_data_source in terraform. I see this supported in terraform only for azure_blob_storage_data_source - credentials_secret

New or Affected Resource(s)

• google_storage_transfer_job

Potential Terraform Configuration

data "google_secret_manager_secret_version" "aws_access_key" {
 secret   = "aws_access_key"
 project = var.project
}

data "google_secret_manager_secret_version" "aws_secret_key" {
 secret   = "aws_secret_key"
 project  = var.project

data "google_secret_manager_secret_version" "aws_secret_creds" {
 secret   = "aws_secret_creds"
 project  = var.project

resource "google_storage_transfer_job" "s3-bucket-nightly-backup" {
  description = "s3-bucket-nightly-backup"
  name        = "s3-bucket-nightly-backup"
  project     = var.project

  transfer_spec {
    aws_s3_data_source {
      bucket_name = "aws-storage-bucket"
      #either something like this here
      aws_access_key {
        access_key_id     = data.google_secret_manager_secret_version.aws_access_key
        secret_access_key = data.google_secret_manager_secret_version.aws_access_key
     #or 
        credentials_secret = data.google_secret_manager_secret_version.aws_secret_creds
      }
    }
    gcs_data_sink {
      bucket_name = "some-storage-bucket"
    }
  }

References

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_transfer_job

b/340252079

[SarahFrench]

Note from triage: this request corresponds to adding support for the credentialsSecret field in the API, which would correspond to a new transfer_spec.aws_s3_data_source.credentials_secret field in the resource.

From the description of that field in the API docs:

Optional. The Resource name of a secret in Secret Manager.

AWS credentials must be stored in Secret Manager in JSON format:

{ "accessKeyId": "ACCESS_KEY_ID", "secretAccessKey": "SECRET_ACCESS_KEY" }