Can't use a global location with an eventarc trigger and a workflow

Airmanbzh commented 1 year ago

I want to trigger a workflow with Eventarc when a secret is created. I can do it using the console and having a global location set on the Eventarc trigger (as secrets are global) but I can't do it using Terraform as it checks if the workflow is on the same location as the trigger

A workaround could be to not check the workflow location when we set a global location on the trigger or allow to have different locations

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.5.3
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.73.0

Affected Resource(s)

google_eventarc_trigger
google_workflows_workflow

Terraform Configuration Files

resource "google_eventarc_trigger" "eventarc_trigger" {
    name = "trigger-on-secret-creation"
    location = "global"

    matching_criteria {
        attribute = "type"
        value     = "google.cloud.audit.log.v1.written"
    }
    matching_criteria {
        attribute = "serviceName"
        value     = "secretmanager.googleapis.com"
    }
    matching_criteria {
        attribute = "methodName"
        value     = "google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion"
    }

    destination {
        workflow = google_workflows_workflow.eventarc_trigger_workflow.name
    }

    service_account = google_service_account.eventarc_sa.email
}

resource "google_workflows_workflow" "eventarc_trigger_workflow" {
  name            = "eventarc-trigger-workflow"
  project         = var.project_id
  region          = var.region # set to europe-west1
  description     = "Workflow to compute eventarc events on secret creation"
  service_account = google_service_account.eventarc_trigger_workflow_sa.email
  source_contents = join("", [
    file("${path.module}/src/eventarc-workflows.yaml"),
  ])
}

Debug Output

Gist: https://gist.github.com/Airmanbzh/a5dc4c7d33c7438ed3ae1b464d1a5e74#file-gistfile1-txt

Error given:

╷
│ Error: Error creating Trigger: googleapi: Error 400: unsupported cloud workflows region "global"
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.RequestInfo",
│     "requestId": "a20b872c7114bfe7"
│   }
│ ]
│ 
│   with module.eventarc-trigger.google_eventarc_trigger.eventarc_trigger,
│   on modules/eventarc-trigger/eventarc.tf line 1, in resource "google_eventarc_trigger" "eventarc_trigger":
│    1: resource "google_eventarc_trigger" "eventarc_trigger" {
│ 
╵

Panic Output

Expected Behavior

We should be able to have a global Eventarc trigger using a regional workflow

Actual Behavior

Terraform fails if the trigger and workflow don't have the same location

Steps to Reproduce

terraform apply

Important Factoids

gcloud has all the necessary options to do so. E.g. from GCP documentation https://cloud.google.com/eventarc/docs/workflows/route-trigger-cloud-audit-logs#gcloud_2

gcloud eventarc triggers create helloworld-trigger \
    --location=us-central1 \
    --destination-workflow=my-workflow \
    --destination-workflow-location=europe-west4 \
    --event-filters="type=google.cloud.audit.log.v1.written" \
    --event-filters="serviceName=bigquery.googleapis.com" \
    --event-filters="methodName=jobservice.jobcompleted" \
    --service-account="${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

References

0000

edwardmedia commented 1 year ago

@Airmanbzh I see you provided global for google_eventarc_trigger. Do you want it to be different region (location) from the one on google_workflows_workflow? Can you try by making them the same?

Based on the error from the api, it is unlikely you can choose global for the trigger.

Error 400: unsupported cloud workflows region "global"

Airmanbzh commented 1 year ago

@Airmanbzh I see you provided global for google_eventarc_trigger. Do you want it to be different region (location) from the one on google_workflows_workflow? Can you try by making them the same?

Based on the error from the api, it is unlikely you can choose global for the trigger.
Error 400: unsupported cloud workflows region "global"

I can set the same location as workflow (e.g.: 'europe-west1`) but then, eventarc doesn't detect secret creations as they are global and not related to a region.

And, when I'm using gcp UI, I can set a global trigger and a specific location for the workflow

Airmanbzh commented 1 year ago

@Airmanbzh I see you provided global for google_eventarc_trigger. Do you want it to be different region (location) from the one on google_workflows_workflow? Can you try by making them the same?

Based on the error from the api, it is unlikely you can choose global for the trigger.
Error 400: unsupported cloud workflows region "global"

And to go further, this error seems to come from a test done before the trigger creation that checks if the destination workflow exists in the same location.

To do so, I think the provider tries to recreate the ID of the workflow using the trigger location. So, if I have a trigger in a global location and a workflow in europe-west1, the provider checks if the workflow exists in the global location: {"destination":{"workflow":"projects//locations/global/workflows/eventarc-trigger-workflow"}

2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: -----------[REQUEST]----------
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: POST /v1/projects/<PROJECT_ID>/locations/global/triggers?alt=json&triggerId=trigger-on-secret-creation HTTP/1.1
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: Host: eventarc.googleapis.com
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: User-Agent: Terraform/1.5.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.73.0 DeclarativeClientLib/0.0.1
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: Content-Length: 571
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: Content-Type: application/json
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: Accept-Encoding: gzip
2023-07-18T13:51:27.304+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: 
2023-07-18T13:51:27.305+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: {"destination":{"workflow":"projects/<PROJECT_ID>/locations/global/workflows/eventarc-trigger-workflow"},"eventFilters":[{"attribute":"serviceName","value":"secretmanager.googleapis.com"},{"attribute":"type","value":"google.cloud.audit.log.v1.written"},{"attribute":"methodName","value":"google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion"}],"name":"projects/<PROJECT_ID>/locations/global/triggers/trigger-on-secret-creation","serviceAccount":"eventarc-listener-sa@<PROJECT_ID>.iam.gserviceaccount.com"}
2023-07-18T13:51:27.305+0200 [DEBUG] provider.terraform-provider-google_v4.73.0_x5: -------[END REQUEST]--------

edwardmedia commented 1 year ago

@Airmanbzh I do see the same error when I provide global for the trigger. But after changing the location to the same on workflow, it works fine.

The error comes from the api. I wonder how the UI allows users to create a global trigger. Can you import your global trigger and share its state? In the meantime, can you detail the steps to create the global trigger on UI?

And, when I'm using gcp UI, I can set a global trigger and a specific location for the workflow

Airmanbzh commented 1 year ago

@Airmanbzh I do see the same error when I provide global for the trigger. But after changing the location to the same on workflow, it works fine.

The error comes from the api. I wonder how the UI allows users to create a global trigger. Can you import your global trigger and share its state? In the meantime, can you detail the steps to create the global trigger on UI?
And, when I'm using gcp UI, I can set a global trigger and a specific location for the workflow

The terraform state given after importing the global trigger:

resource "google_eventarc_trigger" "eventarc_trigger" {
    conditions      = {}
    create_time     = "2023-07-17T15:05:37.839302936Z"
    id              = "projects/<PROJECT_ID>/locations/global/triggers/test-manual-trigger2"
    labels          = {}
    location        = "global"
    name            = "test-manual-trigger2"
    project         = "<PROJECT_ID>"
    service_account = "eventarc-listener-sa@<PROJECT_ID>.iam.gserviceaccount.com"
    uid             = "56a14a40-c39f-41f7-aa9b-13dd655077f0"
    update_time     = "2023-07-17T15:05:48.642988340Z"

    destination {
        workflow = "projects/<PROJECT_ID>/locations/europe-west1/workflows/eventarc-trigger-workflow"
    }

    matching_criteria {
        attribute = "methodName"
        value     = "google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion"
    }
    matching_criteria {
        attribute = "serviceName"
        value     = "secretmanager.googleapis.com"
    }
    matching_criteria {
        attribute = "type"
        value     = "google.cloud.audit.log.v1.written"
    }

    timeouts {}

    transport {
        pubsub {
            subscription = "projects/<PROJECT_ID>/subscriptions/eventarc-global-test-manual-trigger2-sub-097"
            topic        = "projects/<PROJECT_ID>/topics/eventarc-global-test-manual-trigger2-233"
        }
    }
}

Airmanbzh commented 1 year ago

Screenshot of the trigger creation using the UI where we can see the global location:

And the end result: Screenshot from 2023-07-19 10-09-23

edwardmedia commented 1 year ago

b/291906863

Milan-ByCape commented 4 months ago

I am running into the same issue when trying to make a trigger for a high availability storage bucket. The location is set to EU for the bucket, therefor the trigger has to be in EU too. But I can not set the location for the workflow to EU.

marknelissen commented 2 months ago

I am running into the same issue when trying to make a trigger for a high availability storage bucket. The location is set to EU for the bucket, therefor the trigger has to be in EU too. But I can not set the location for the workflow to EU.

I had the same issue, but it seems it was caused by taking the location property of bucket resource. That output is in uppercase, but the location of the eventarc trigger needs to be lower case. So by litterally writing location = "eu" it worked. What didn't work was location = google_storage_bucket.default.location

hashicorp / terraform-provider-google