credentials_secret not identified as a valid parameter in the Storage Transfer Storage resource for GCP

fathom-farhan commented 2 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.6.3 on linux_arm64

provider registry.terraform.io/hashicorp/google v5.25.0
provider registry.terraform.io/hashicorp/google-beta v5.25.0

Affected Resource(s)

google_storage_transfer_job

Terraform Configuration

"google_storage_transfer_job": {
     "azure-to-gcs-transfer-job": {
       "//": {
         "metadata": {
           "path": "redacted",
           "uniqueId": "redacted"
         }
       },
       "depends_on": [
         "redacted"
       ],
       "description": "Transfer data from [redacted]",
       "project": "redacted",
       "schedule": {
         "schedule_end_date": {
           "day": 19,
           "month": 7,
           "year": 2024
         },
         "schedule_start_date": {
           "day": 2,
           "month": 7,
           "year": 2024
         }
       },
       "transfer_spec": {
         "azure_blob_storage_data_source": {
           "container": "redacted",
           "credentials_secret": "${redacted}",
           "path": "",
           "storage_account": "redacted"
         },
         "gcs_data_sink": {
           "bucket_name": "${redacted}",
           "path": "historical/redacted"
         },
         "transfer_options": {
           "delete_objects_from_source_after_transfer": false,
           "overwrite_when": "DIFFERENT"
         }
       }
     }
   }

"required_providers": {
     "google": {
       "source": "google",
       "version": "5.25.0"
     },
     "google-beta": {
       "source": "google-beta",
       "version": "5.25.0"
     }
   }

Debug Output

No response

Expected Behavior

When defining the azure_blob_storage_data_source for the transfer_spec in google_storage_transfer_job, we should be able to define the credentials_secret and NOT define the azure_credentials explicitly.

This should be in beta from v5.21.0

Actual Behavior

Trying to deploy the storage transfer job resource with credentials_secret defined and azure_credentials leads to the following errors:

"cdktf-deploy-all": At least 1 "azure_credentials" blocks are required.
"cdktf-deploy-all": No argument or block type is named "credentials_secret"

stack trace:

INFO 2024-07-03T21:40:24.736690389Z Step #4 - "cdktf-deploy-all": redacted Error: Extraneous JSON object property
INFO 2024-07-03T21:40:24.736701498Z Step #4 - "cdktf-deploy-all":
INFO 2024-07-03T21:40:24.736703538Z Step #4 - "cdktf-deploy-all": on cdk.tf.json line 1266, in resource.google_storage_transfer_job.azure-to-gcs-transfer-job-redacted (azure-to-gcs-transfer-job-redacted).transfer_spec.azure_blob_storage_data_source:
INFO 2024-07-03T21:40:24.736704656Z Step #4 - "cdktf-deploy-all": 1266: "credentials_secret": "${google_secret_manager_secret.secret-manager-redacted_azure_sas_token (secret-manager-redacted_azure_sas_token).name}",
INFO 2024-07-03T21:40:24.736705648Z Step #4 - "cdktf-deploy-all":
INFO 2024-07-03T21:40:24.736706710Z Step #4 - "cdktf-deploy-all": No argument or block type is named "credentials_secret".
INFO 2024-07-03T21:40:24.786494193Z Step #4 - "cdktf-deploy-all": redacted Error: Insufficient azure_credentials blocks
INFO 2024-07-03T21:40:24.786505550Z Step #4 - "cdktf-deploy-all":
INFO 2024-07-03T21:40:24.786507566Z Step #4 - "cdktf-deploy-all": on cdk.tf.json line 1269, in resource.google_storage_transfer_job.azure-to-gcs-transfer-job-redacted (azure-to-gcs-transfer-job-redacted).transfer_spec.azure_blob_storage_data_source:
INFO 2024-07-03T21:40:24.786508525Z Step #4 - "cdktf-deploy-all": 1269: },
INFO 2024-07-03T21:40:24.786508875Z Step #4 - "cdktf-deploy-all":
INFO 2024-07-03T21:40:24.786509546Z Step #4 - "cdktf-deploy-all": At least 1 "azure_credentials" blocks are required.

Steps to reproduce

Create a google_storage_transfer_job.GoogleStorageTransferJob to transfer data from Azure to GCS:
1. For the azure_blob_storage_data_source, define the container, storage_account, path and credentials_secret but not the azure_credentials
Try to deploy the resource

Important Factoids

No response

References

No response

b/352823434

kautikdk commented 1 month ago

Hi @fathom-farhan, credentials_secret field is for google-beta provider only. Reference:https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_transfer_job#credentials_secret If you want to use credentials_secret field, You can follow this documentation to use beta provider. For this case, It should be done by adding this line provider = google-beta in the resource definition.

fathom-parth commented 1 month ago

Thanks @kautikdk, we did have the google-beta provider installed but realized we weren't passing it directly to the sts job resource.

We'll try your suggestion and update here when we get to it (may not be for a bit since our need for this job has passed)

hashicorp / terraform-provider-google