search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.32k stars 1.73k forks source link

Discovery Engine Target Site: Placeholder Bug #19621

Open aytang-telus opened 1 week ago

aytang-telus commented 1 week ago

Community Note

This was originally an expedited feature request coming in from my team, to better support Vertex AI Datastore site configurations. Our use case involves Agent Builder from the Agent Assist product line.

Terraform Version & Provider Version(s)

Terraform v0.13.3 on Cloud Build (Image: /builders/terraform:0.13.3)

git tag used to trigger Cloud Build (which runs terraform apply) on GCP. 

provider "google-beta" {
  region  = var.region
  version = ">3.1.0"
}
provider "google" {
  region  = var.region
  version = ">3.1.0"
}

Affected Resource(s)

google_discovery_engine_target_site

Terraform Configuration

resource "google_discovery_engine_target_site" "tcom_site" {
  count                = contains(["lab", "np"], var.env) ? 1 : 0
  project              = var.project_id
  location             = "global"
  data_store_id        = google_discovery_engine_data_store.gen_dfcx_ent_tcom[0].data_store_id
  provided_uri_pattern = "www.telus.com/en*"
  type                 = "INCLUDE"
  exact_match          = true
}

resource "google_discovery_engine_target_site" "mob_site" {
  count                = contains(["lab", "np"], var.env) ? 1 : 0
  project              = var.project_id
  location             = "global"
  data_store_id        = google_discovery_engine_data_store.gen_ds_web_en_mob[0].data_store_id
  provided_uri_pattern = "www.telus.com/en/mobility*"
  type                 = "INCLUDE"
  exact_match          = true
}

Debug Output

Link to Gist: https://gist.github.com/aytang-telus/8b492953b5dc9107c2b08cb2cc5e38c2

Scroll down for specific error:

Step #2 - "terraform-apply": Error: Error waiting to create TargetSite: Error waiting for Creating TargetSite: error while retrieving operation: parse "https://{{location}}-discoveryengine.googleapis.com/v1/projects/863414079250/locations/global/collections/default_collection/dataStores/gen_dfcx_ent_tcom/operations/create-target-site-8446460370749946762": invalid character "{" in host name
Step #2 - "terraform-apply": 
Step #2 - "terraform-apply":   on datastore-sites.tf line 1, in resource "google_discovery_engine_target_site" "tcom_site":
Step #2 - "terraform-apply":    1: resource "google_discovery_engine_target_site" "tcom_site" {

Expected Behavior

Target site/resource should have been created successfully?

Actual Behavior

It looks like the {{location}} in https://{{location}}-discoveryengine.googleapis.com/v1/projects/863414079250/locations/global/collections/default_collection/dataStores/gen_dfcx_ent_tcom/operations/create-target-site-8446460370749946762 didn't get replaced with global, or the value of google_discovery_engine_target_site.tcom_site[0].location - hence the parse error.

Please let me know if this is the case, or if its an error on my side. Thanks.

Steps to reproduce

  1. terraform apply

Important Factoids

Using a service account to run terraform apply through Cloud Build. Will be looping in our TAM at Google as well - TIA!

References

No response

b/370089093

jialei-chen commented 1 week ago

Hi @aytang-telus, thanks for sharing the context. I am trying to reproduce the same error and figuring out the root cause. Will add comments here once I have any update.

aytang-telus commented 1 week ago

hi @jialei-chen - thanks for replying. I do want to mention that terraform plan works as expected - 

locals {
  equip_return_sites = [
    "www.telus.com/en/support/article/cancel-telus-home-services",
    "www.telus.com/en/support/article/equipment-warranty-upgrades-returns*"
  ]
}

resource "google_discovery_engine_target_site" "equip_return_site" {
  for_each      = contains(["lab", "np"], var.env) ? { for idx, site in local.equip_return_sites : idx => site } : {}
  project       = var.project_id
  location      = "global"
  data_store_id = google_discovery_engine_data_store.ds_ffh_equipment_return[0].data_store_id
  type          = "INCLUDE"
  exact_match   = true

  provided_uri_pattern = each.value
}

Output:

Step #2 - "terraform-plan": Terraform will perform the following actions:
Step #2 - "terraform-plan": 
Step #2 - "terraform-plan":   # google_discovery_engine_target_site.equip_return_site["0"] will be created
Step #2 - "terraform-plan":   + resource "google_discovery_engine_target_site" "equip_return_site" {
Step #2 - "terraform-plan":       + data_store_id          = "ds_ffh_equipment_return"
Step #2 - "terraform-plan":       + exact_match            = true
Step #2 - "terraform-plan":       + failure_reason         = (known after apply)
Step #2 - "terraform-plan":       + generated_uri_pattern  = (known after apply)
Step #2 - "terraform-plan":       + id                     = (known after apply)
Step #2 - "terraform-plan":       + indexing_status        = (known after apply)
Step #2 - "terraform-plan":       + location               = "global"
Step #2 - "terraform-plan":       + name                   = (known after apply)
Step #2 - "terraform-plan":       + project                = "cdo-ccai-lab-f7edc3"
Step #2 - "terraform-plan":       + provided_uri_pattern   = "www.telus.com/en/support/article/cancel-telus-home-services"
Step #2 - "terraform-plan":       + root_domain_uri        = (known after apply)
Step #2 - "terraform-plan":       + site_verification_info = (known after apply)
Step #2 - "terraform-plan":       + target_site_id         = (known after apply)
Step #2 - "terraform-plan":       + type                   = "INCLUDE"
Step #2 - "terraform-plan":       + update_time            = (known after apply)
Step #2 - "terraform-plan":     }
Step #2 - "terraform-plan": 
Step #2 - "terraform-plan":   # google_discovery_engine_target_site.equip_return_site["1"] will be created
Step #2 - "terraform-plan":   + resource "google_discovery_engine_target_site" "equip_return_site" {
Step #2 - "terraform-plan":       + data_store_id          = "ds_ffh_equipment_return"
Step #2 - "terraform-plan":       + exact_match            = true
Step #2 - "terraform-plan":       + failure_reason         = (known after apply)
Step #2 - "terraform-plan":       + generated_uri_pattern  = (known after apply)
Step #2 - "terraform-plan":       + id                     = (known after apply)
Step #2 - "terraform-plan":       + indexing_status        = (known after apply)
Step #2 - "terraform-plan":       + location               = "global"
Step #2 - "terraform-plan":       + name                   = (known after apply)
Step #2 - "terraform-plan":       + project                = "cdo-ccai-lab-f7edc3"
Step #2 - "terraform-plan":       + provided_uri_pattern   = "www.telus.com/en/support/article/equipment-warranty-upgrades-returns*"
Step #2 - "terraform-plan":       + root_domain_uri        = (known after apply)
Step #2 - "terraform-plan":       + site_verification_info = (known after apply)
Step #2 - "terraform-plan":       + target_site_id         = (known after apply)
Step #2 - "terraform-plan":       + type                   = "INCLUDE"
Step #2 - "terraform-plan":       + update_time            = (known after apply)
Step #2 - "terraform-plan":     }
Step #2 - "terraform-plan": 
Step #2 - "terraform-plan": Plan: 2 to add, 0 to change, 0 to destroy.
aytang-telus commented 1 week ago

I think the resource gets successfully created and attached to the datastore in the UI but it doesn't get reflected in terraform state file.

image

ggtisc commented 1 week ago

Hi @aytang-telus!

I tried to replicate this issue with the next code but everything was successful without errors and all resources were saved correctly in the tfstatefile:

provider "google-beta" {
  region  = "us-central1"
  version = ">3.1.0"
}
provider "google" {
  region  = "us-central1"
  version = ">3.1.0"
}

resource "google_discovery_engine_data_store" "de_ds_19621_a" {
  location                     = "global"
  data_store_id                = "de-ds-19621-a"
  display_name                 = "DE DS 19621 A"
  industry_vertical            = "GENERIC"
  content_config               = "PUBLIC_WEBSITE"
  solution_types               = ["SOLUTION_TYPE_SEARCH"]
  create_advanced_site_search  = false
  skip_default_schema_creation = false
}

resource "google_discovery_engine_target_site" "de_ts_19621_a" {
  project              = "my-project"
  location             = "global"
  data_store_id        = google_discovery_engine_data_store.de_ds_19621_a.data_store_id
  provided_uri_pattern = "www.telus.com/en*"
  type                 = "INCLUDE"
  exact_match          = true
}

resource "google_discovery_engine_data_store" "de_ds_19621_b" {
  location                     = "global"
  data_store_id                = "de-ds-19621-b"
  display_name                 = "DE DS 19621 B"
  industry_vertical            = "GENERIC"
  content_config               = "PUBLIC_WEBSITE"
  solution_types               = ["SOLUTION_TYPE_SEARCH"]
  create_advanced_site_search  = false
  skip_default_schema_creation = false
}

resource "google_discovery_engine_target_site" "de_ts_19621_b" {
  project              = "my-project"
  location             = "global"
  data_store_id        = google_discovery_engine_data_store.de_ds_19621_b.data_store_id
  provided_uri_pattern = "www.telus.com/en/mobility*"
  type                 = "INCLUDE"
  exact_match          = true
}

I just changed names to not have duplicates in my environment, I suggest you to check your for each loop and your environment variables values and if you still continue with issues please share the full code of the lacking variables and google_discovery_engine_data_store resources

aytang-telus commented 1 week ago

Hi @ggtisc - thanks for replying. I'm confident that the for each loops are not the issue here since the terraform plan renders correctly for each site that I'm trying to create.

Datastore Resources: https://gist.github.com/aytang-telus/d4bdf94517bb5cdc961a887fafd4c19f Target Site Resources: https://gist.github.com/aytang-telus/50842b47ec9b60d758cf5a8db47bbcd4

The only env var var.project_id, is pretty self-explanatory so I'll leave that out.

Let me know if I can provide anymore info!

ggtisc commented 1 week ago

@aytang-telus could you share the value of the env.var and data_store_id = google_discovery_engine_data_store.ds_ffh_tv_troubleshoot[0].data_store_id?

aytang-telus commented 1 week ago

@ggtisc

In this particular run, env.var = np. google_discovery_engine_data_store.ds_ffh_tv_troubleshoot[0].data_store_id should just be the name of the datastore itself, or whats defined in google_discovery_engine_data_store.<datastore instance>.data_store_id.

In this case it would be ds_ffh_tv_troubleshoot.

Hope this helps!

ggtisc commented 1 week ago

Confirmed issue!

This is probably due to the amount of resources, but in addition to the long time to create them(up to 59 minutes) it sometimes results in a 500 error

aytang-telus commented 1 week ago

Noted. When you say amount of resources - is it the number of target sites or data stores being created? Is there a current workaround or what did you do to make it work the first time around?

Thanks and have a good weekend :)