search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.33k stars 1.73k forks source link

google_container_registry `apply` error "an unexpected new value: Root object was present, but now absent." #17605

Closed Jerome-Paddick closed 6 months ago

Jerome-Paddick commented 7 months ago

Community Note

Terraform Version

Terraform v1.7.4 on windows_amd64

Affected Resource(s)

google_container_registry

Terraform Configuration


variable "project_id" {
  description = "The GCP project ID"
}

variable "location" {
  description = "GCP location"
}

variable "zone" {
  description = "GCP Zone"
}

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.20.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.20.0"
    }
  }
}

provider "google" {
  credentials = file("${path.module}/../shared/creds.json")
  project     = var.project_id
  region      = var.location
}

resource "google_container_registry" "jjmc_container_registry" {
  location = "EU"
}

output "registry_url" {
  value = google_container_registry.jjmc_container_registry.bucket_self_link
}

Debug Output

> make apply
terraform apply -var-file="..\shared\shared_variabled.tfvars" -auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_container_registry.jjmc_container_registry will be created
  + resource "google_container_registry" "jjmc_container_registry" {
      + bucket_self_link = (known after apply)
      + id               = (known after apply)
      + location         = "EU"
      + project          = "mc-server-416313"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + registry_url = (known after apply)
google_container_registry.jjmc_container_registry: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to google_container_registry.jjmc_container_registry, provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced 
│ an unexpected new value: Root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
make: *** [makefile:10: apply] Error 1

Expected Behavior

terraform provisions the google_container_registry

Actual Behavior

terraform failed to provision the google_container_registry

Steps to reproduce

  1. `terraform apply

Important Factoids

Windows 11

References

No response

EDIT ---

GCP is depreciating google_container_registry in favour of google_artifact_registry_repository in may 2024

So if anyone else has this issue then use that instead

eg.

resource "google_artifact_registry_repository" "jjmc_repo" {
  location         = "europe-west2"
  repository_id = "repo_name"
  description    = "example docker repository"
  format           = "DOCKER"
}
ggtisc commented 7 months ago

Hi @Jerome-Paddick the issue was replicated without errors and the resource was created successfully. It looks like you are having environment variables conflict. As I see you are creating your variables in the code while at the same time you are loading the variables from a file -var-file="..\shared\shared_variabled.tfvars" and finally in your resource google_container_registry you are assigning other value to the location that may not match with the value of that variable.

Due the data you are protecting may be sensitive, I am sharing a link where you can find help on how to manage your environment variables correctly and prevent conflicts from being generated: https://developer.hashicorp.com/terraform/enterprise/workspaces/variables/managing-variables#priority-variable-sets

Jerome-Paddick commented 7 months ago

HI @ggtisc

I tried hardcoding the values with the same result

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.20.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.20.0"
    }
  }
}

provider "google" {
  credentials = file("${path.module}/../shared/creds.json")
  project     = "mc-server-416313"
  region      = "europe-west2"
}

resource "google_container_registry" "jjmc_container_registry" {
  location = "EU"
}
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to google_container_registry.jjmc_container_registry, provider
│ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.  
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

My config also works fine with other resources 

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.20.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.20.0"
    }
  }
}

provider "google" {
  credentials = file("${path.module}/../shared/creds.json")
  project     = "mc-server-416313"
  region      = "europe-west2"
}

resource "google_storage_bucket" "server_backups" {
  name          = "jjmc_server_backups"
  location      = "europe-west2"
  force_destroy = true # Allows Terraform to delete the bucket even if it contains objects
}

I have also tried

Explicity adding the project to the google_container_registry resource -> same error changing the location to "US" -> same error changing location to "ASIA" -> same error changing location to "europe-west2" -> Invalid host name europe-west2.gcr.io.

I can now see from the Google Cloud GUI that terraform is provisioning the registries but still throwing the error and not adding them to the .tfstate file image

ggtisc commented 7 months ago

@Jerome-Paddick can you check your permissions and share debug logs to see what is happening with your configuration?

Jerome-Paddick commented 7 months ago

@ggtisc

I checked my permissions, my service user has owner permissions so should be able to do anything

I checked my apply logs, looks like the terraform sends the request to make the registry (and gets 200) and then attempts to find the registry at

https://storage.googleapis.com/storage/v1/b/eu.artifacts.mc-server-416313.appspot.com

and gets a 404 - the specified bucket does not exist

Jerome-Paddick commented 7 months ago 
2024-03-20T00:26:37.850Z [INFO]  Terraform version: 1.7.4
2024-03-20T00:26:37.851Z [DEBUG] using github.com/hashicorp/go-tfe v1.41.0
2024-03-20T00:26:37.851Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2024-03-20T00:26:37.851Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2024-03-20T00:26:37.851Z [DEBUG] using github.com/zclconf/go-cty v1.14.1
2024-03-20T00:26:37.851Z [INFO]  Go runtime version: go1.21.5
2024-03-20T00:26:37.851Z [INFO]  CLI args: []string{"C:\\ProgramData\\chocolatey\\lib\\terraform\\tools\\terraform.exe", "apply", "-var-file=..\\shared\\shared_variabled.tfvars", "-auto-approve"}
2024-03-20T00:26:37.853Z [DEBUG] Attempting to open CLI config file: REDACTED
2024-03-20T00:26:37.853Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-03-20T00:26:37.854Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-03-20T00:26:37.854Z [DEBUG] ignoring non-existing provider search directory REDACTED
2024-03-20T00:26:37.854Z [DEBUG] ignoring non-existing provider search directory REDACTED
2024-03-20T00:26:37.854Z [INFO]  CLI command args: []string{"apply", "-var-file=..\\shared\\shared_variabled.tfvars", "-auto-approve"}
2024-03-20T00:26:37.862Z [DEBUG] New state was assigned lineage REDACTED
2024-03-20T00:26:37.987Z [DEBUG] checking for provisioner in "."
2024-03-20T00:26:37.987Z [DEBUG] checking for provisioner in "C:\\ProgramData\\chocolatey\\lib\\terraform\\tools"
2024-03-20T00:26:37.989Z [INFO]  backend/local: starting Apply operation
2024-03-20T00:26:37.994Z [DEBUG] created provider logger: level=debug
2024-03-20T00:26:37.994Z [INFO]  provider: configuring client automatic mTLS
2024-03-20T00:26:38.003Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe"]
2024-03-20T00:26:38.006Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=265320
2024-03-20T00:26:38.006Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:38.035Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: configuring server automatic mTLS: timestamp=2024-03-20T00:26:38.034Z
2024-03-20T00:26:38.057Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2024-03-20T00:26:38.057Z
2024-03-20T00:26:38.057Z [DEBUG] provider: using plugin: version=5
2024-03-20T00:26:38.140Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-20T00:26:38.162Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=265320
2024-03-20T00:26:38.162Z [DEBUG] provider: plugin exited
2024-03-20T00:26:38.162Z [DEBUG] created provider logger: level=debug
2024-03-20T00:26:38.162Z [INFO]  provider: configuring client automatic mTLS
2024-03-20T00:26:38.166Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/5.20.0/windows_amd64/terraform-provider-google-beta_v5.20.0_x5.exe args=[".terraform/providers/registry.terraform.io/hashicorp/google-beta/5.20.0/windows_amd64/terraform-provider-google-beta_v5.20.0_x5.exe"]
2024-03-20T00:26:38.169Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/5.20.0/windows_amd64/terraform-provider-google-beta_v5.20.0_x5.exe pid=276084
2024-03-20T00:26:38.169Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/5.20.0/windows_amd64/terraform-provider-google-beta_v5.20.0_x5.exe
2024-03-20T00:26:38.199Z [INFO]  provider.terraform-provider-google-beta_v5.20.0_x5.exe: configuring server automatic mTLS: timestamp=2024-03-20T00:26:38.199Z
2024-03-20T00:26:38.222Z [DEBUG] provider.terraform-provider-google-beta_v5.20.0_x5.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2024-03-20T00:26:38.222Z
2024-03-20T00:26:38.222Z [DEBUG] provider: using plugin: version=5
2024-03-20T00:26:38.303Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-20T00:26:38.315Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/5.20.0/windows_amd64/terraform-provider-google-beta_v5.20.0_x5.exe pid=276084
2024-03-20T00:26:38.315Z [DEBUG] provider: plugin exited
2024-03-20T00:26:38.315Z [DEBUG] Building and walking validate graph
2024-03-20T00:26:38.315Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.315Z [DEBUG] pruning unused provider["registry.terraform.io/hashicorp/google-beta"]
2024-03-20T00:26:38.315Z [DEBUG] ReferenceTransformer: "output.registry_url (expand)" references: [google_container_registry.jjmc_container_registry]
2024-03-20T00:26:38.315Z [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: []
2024-03-20T00:26:38.315Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry" references: []
2024-03-20T00:26:38.316Z [DEBUG] Starting graph walk: walkValidate
2024-03-20T00:26:38.316Z [DEBUG] created provider logger: level=debug
2024-03-20T00:26:38.316Z [INFO]  provider: configuring client automatic mTLS
2024-03-20T00:26:38.319Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe"]
2024-03-20T00:26:38.321Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=271452
2024-03-20T00:26:38.321Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:38.350Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: configuring server automatic mTLS: timestamp=2024-03-20T00:26:38.349Z
2024-03-20T00:26:38.372Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2024-03-20T00:26:38.372Z
2024-03-20T00:26:38.372Z [DEBUG] provider: using plugin: version=5
2024-03-20T00:26:38.399Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-20T00:26:38.408Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=271452
2024-03-20T00:26:38.408Z [DEBUG] provider: plugin exited
2024-03-20T00:26:38.408Z [INFO]  backend/local: apply calling Plan
2024-03-20T00:26:38.408Z [DEBUG] Building and walking plan graph for NormalMode
2024-03-20T00:26:38.408Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.408Z [DEBUG] pruning unused provider["registry.terraform.io/hashicorp/google-beta"]
2024-03-20T00:26:38.408Z [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: []
2024-03-20T00:26:38.408Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry (expand)" references: []
2024-03-20T00:26:38.408Z [DEBUG] ReferenceTransformer: "output.registry_url (expand)" references: [google_container_registry.jjmc_container_registry (expand)]
2024-03-20T00:26:38.408Z [DEBUG] Starting graph walk: walkPlan
2024-03-20T00:26:38.409Z [DEBUG] created provider logger: level=debug
2024-03-20T00:26:38.409Z [INFO]  provider: configuring client automatic mTLS
2024-03-20T00:26:38.412Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe"]
2024-03-20T00:26:38.414Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=275768
2024-03-20T00:26:38.414Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:38.441Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: configuring server automatic mTLS: timestamp=2024-03-20T00:26:38.441Z
2024-03-20T00:26:38.463Z [DEBUG] provider: using plugin: version=5
2024-03-20T00:26:38.464Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2024-03-20T00:26:38.463Z
2024-03-20T00:26:38.492Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Authenticating using configured Google JSON 'credentials'...: @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1731 @module=google tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=REDACTED tf_mux_provider="*proto5server.Server" tf_rpc=ConfigureProvider timestamp=2024-03-20T00:26:38.492Z
2024-03-20T00:26:38.492Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe:   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: tf_req_id=REDACTED @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1732 @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_rpc=ConfigureProvider timestamp=2024-03-20T00:26:38.492Z
2024-03-20T00:26:38.492Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Authenticating using configured Google JSON 'credentials'...: tf_req_id=REDACTED tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1731 @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google timestamp=2024-03-20T00:26:38.492Z
2024-03-20T00:26:38.492Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe:   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1732 tf_rpc=ConfigureProvider @module=google tf_req_id=REDACTED timestamp=2024-03-20T00:26:38.492Z
2024-03-20T00:26:38.492Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] Waiting for state to become: [success]
2024-03-20T00:26:38.747Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Terraform is using this identity: REDACTED@mc-server-416313.iam.gserviceaccount.com: @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1621 @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_rpc=ConfigureProvider tf_req_id=REDACTED timestamp=2024-03-20T00:26:38.747Z
2024-03-20T00:26:38.749Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [INFO] Authenticating using configured Google JSON 'credentials'...
2024-03-20T00:26:38.749Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-03-20T00:26:38.749Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [INFO] Authenticating using configured Google JSON 'credentials'...
2024-03-20T00:26:38.749Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-03-20T00:26:38.749Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] Waiting for state to become: [success]
2024-03-20T00:26:38.834Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [INFO] Terraform is using this identity: REDACTED@mc-server-416313.iam.gserviceaccount.com
2024-03-20T00:26:38.834Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] parent context canceled, cleaning up batcher batches
2024-03-20T00:26:38.834Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] Stopping batcher "Service Usage"
2024-03-20T00:26:38.834Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] parent context canceled, cleaning up batcher batches
2024-03-20T00:26:38.834Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] Stopping batcher "IAM"
2024-03-20T00:26:38.835Z [DEBUG] Resource instance state not found for node "google_container_registry.jjmc_container_registry", instance google_container_registry.jjmc_container_registry
2024-03-20T00:26:38.835Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry" references: []
2024-03-20T00:26:38.835Z [DEBUG] refresh: google_container_registry.jjmc_container_registry: no state, so not refreshing
2024-03-20T00:26:38.836Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] [transport] [server-transport 0xc0006fe000] Closing: Server.Stop called 
2024-03-20T00:26:38.836Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] [transport] [server-transport 0xc0006fe000] loopyWriter exiting with error: transport closed by client 
2024-03-20T00:26:38.836Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-20T00:26:38.852Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=275768
2024-03-20T00:26:38.852Z [DEBUG] provider: plugin exited
2024-03-20T00:26:38.852Z [DEBUG] building apply graph to check for errors
2024-03-20T00:26:38.852Z [DEBUG] Resource state not found for node "google_container_registry.jjmc_container_registry", instance google_container_registry.jjmc_container_registry
2024-03-20T00:26:38.852Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.852Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.852Z [DEBUG] pruning unused provider["registry.terraform.io/hashicorp/google-beta"]
2024-03-20T00:26:38.852Z [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: []
2024-03-20T00:26:38.852Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry (expand)" references: []
2024-03-20T00:26:38.852Z [DEBUG] ReferenceTransformer: "output.registry_url (expand)" references: [google_container_registry.jjmc_container_registry (expand) google_container_registry.jjmc_container_registry google_container_registry.jjmc_container_registry]
2024-03-20T00:26:38.852Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry" references: []
2024-03-20T00:26:38.864Z [INFO]  backend/local: apply calling Apply
2024-03-20T00:26:38.864Z [DEBUG] Building and walking apply graph for NormalMode plan
2024-03-20T00:26:38.864Z [DEBUG] Resource state not found for node "google_container_registry.jjmc_container_registry", instance google_container_registry.jjmc_container_registry
2024-03-20T00:26:38.864Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.864Z [DEBUG] ProviderTransformer: "google_container_registry.jjmc_container_registry" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"]
2024-03-20T00:26:38.864Z [DEBUG] pruning unused provider["registry.terraform.io/hashicorp/google-beta"]
2024-03-20T00:26:38.864Z [DEBUG] ReferenceTransformer: "output.registry_url (expand)" references: [google_container_registry.jjmc_container_registry google_container_registry.jjmc_container_registry google_container_registry.jjmc_container_registry (expand)]
2024-03-20T00:26:38.864Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry" references: []
2024-03-20T00:26:38.864Z [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: []
2024-03-20T00:26:38.864Z [DEBUG] ReferenceTransformer: "google_container_registry.jjmc_container_registry (expand)" references: []
2024-03-20T00:26:38.864Z [DEBUG] Starting graph walk: walkApply
2024-03-20T00:26:38.864Z [DEBUG] created provider logger: level=debug
2024-03-20T00:26:38.864Z [INFO]  provider: configuring client automatic mTLS
2024-03-20T00:26:38.868Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe"]
2024-03-20T00:26:38.877Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=273668
2024-03-20T00:26:38.877Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:38.906Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: configuring server automatic mTLS: timestamp=2024-03-20T00:26:38.905Z
2024-03-20T00:26:38.929Z [DEBUG] provider: using plugin: version=5
2024-03-20T00:26:38.929Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: plugin address: address=127.0.0.1:10000 network=tcp timestamp=2024-03-20T00:26:38.929Z
2024-03-20T00:26:38.959Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Authenticating using configured Google JSON 'credentials'...: @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=REDACTED tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1731 timestamp=2024-03-20T00:26:38.959Z
2024-03-20T00:26:38.959Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe:   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: @module=google tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1732 tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=REDACTED timestamp=2024-03-20T00:26:38.959Z
2024-03-20T00:26:38.959Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Authenticating using configured Google JSON 'credentials'...: tf_req_id=REDACTED @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1731 @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_rpc=ConfigureProvider timestamp=2024-03-20T00:26:38.959Z
2024-03-20T00:26:38.959Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe:   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: tf_req_id=REDACTED tf_rpc=ConfigureProvider tf_provider_addr=registry.terraform.io/hashicorp/google tf_mux_provider="*proto5server.Server" @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1732 @module=google timestamp=2024-03-20T00:26:38.959Z
2024-03-20T00:26:38.959Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:38 [DEBUG] Waiting for state to become: [success]
2024-03-20T00:26:39.161Z [INFO]  provider.terraform-provider-google_v5.20.0_x5.exe: Terraform is using this identity: REDACTED@mc-server-416313.iam.gserviceaccount.com: @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1621 @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=REDACTED tf_rpc=ConfigureProvider timestamp=2024-03-20T00:26:39.161Z
2024-03-20T00:26:39.162Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [INFO] Authenticating using configured Google JSON 'credentials'...
2024-03-20T00:26:39.162Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-03-20T00:26:39.162Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [INFO] Authenticating using configured Google JSON 'credentials'...
2024-03-20T00:26:39.162Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-03-20T00:26:39.162Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Waiting for state to become: [success]
2024-03-20T00:26:39.239Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [INFO] Terraform is using this identity: REDACTED@mc-server-416313.iam.gserviceaccount.com
2024-03-20T00:26:39.239Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] parent context canceled, cleaning up batcher batches
2024-03-20T00:26:39.239Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Stopping batcher "Service Usage"
2024-03-20T00:26:39.239Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] parent context canceled, cleaning up batcher batches
2024-03-20T00:26:39.239Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Stopping batcher "IAM"
2024-03-20T00:26:39.241Z [INFO]  Starting apply for google_container_registry.jjmc_container_registry
2024-03-20T00:26:39.241Z [DEBUG] google_container_registry.jjmc_container_registry: applying the planned Create change
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Project: mc-server-416313
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] location: EU
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Waiting for state to become: [success]
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Retry Transport: request attempt 0
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:39 [DEBUG] Google API Request Details:
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: ---[ REQUEST ]---------------------------------------
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: GET /v2/token?alt=json&scope=repository%3Amc-server-416313%2Fmy-repo%3Apush%2Cpull&service=gcr.io HTTP/1.1
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Host: eu.gcr.io
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: User-Agent: Terraform/1.7.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.31.0 terraform-provider-google/5.20.0
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Type: application/json
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Accept-Encoding: gzip
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:39.241Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: -----------------------------------------------------
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Google API Response Details:
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: ---[ RESPONSE ]--------------------------------------
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: HTTP/2.0 200 OK
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Cache-Control: private
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Type: application/json
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Date: Wed, 20 Mar 2024 00:26:40 GMT
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Server: Docker Registry
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Content-Type-Options: nosniff
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Frame-Options: SAMEORIGIN
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Gcr-Using-Artifact-Registry: true
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Xss-Protection: 0
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: {
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:  "token": REDACTED,
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:  "expires_in": 43200
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: }
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: -----------------------------------------------------
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Retry Transport: Stopping retries, last request was successful
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Retry Transport: Returning after 1 attempts
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [INFO] Instantiating Google Storage client for path https://storage.googleapis.com/storage/v1/
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Retry Transport: request attempt 0
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:41 [DEBUG] Google API Request Details:
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: ---[ REQUEST ]---------------------------------------
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: GET /storage/v1/b/eu.artifacts.mc-server-416313.appspot.com?alt=json&prettyPrint=false HTTP/1.1
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Host: storage.googleapis.com
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: User-Agent: google-api-go-client/0.5 Terraform/1.7.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.31.0 terraform-provider-google/5.20.0
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Goog-Api-Client: gl-go/1.20.14 gdcl/0.167.0
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Accept-Encoding: gzip
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:41.694Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: -----------------------------------------------------
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [DEBUG] Google API Response Details:
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: ---[ RESPONSE ]--------------------------------------
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: HTTP/2.0 404 Not Found
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Length: 171
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Type: application/json; charset=UTF-8
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Date: Wed, 20 Mar 2024 00:26:40 GMT
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Pragma: no-cache
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Server: UploadServer
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Vary: Origin
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Vary: X-Origin
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Guploader-Uploadid: REDACTED
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: {
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:  "error": {
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:   "code": 404,
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:   "message": "The specified bucket does not exist.",
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:   "errors": [
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:    {
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:     "message": "The specified bucket does not exist.",
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:     "domain": "global",
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:     "reason": "notFound"
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:    }
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:   ]
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe:  }
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: }
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: -----------------------------------------------------
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Length: 171
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Content-Type: application/json; charset=UTF-8
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Date: Wed, 20 Mar 2024 00:26:40 GMT
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Pragma: no-cache
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Server: UploadServer
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Vary: Origin
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: Vary: X-Origin
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: X-Guploader-Uploadid: REDACTED
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: {"error":{"code":404,"message":"The specified bucket does not exist.","errors":[{"message":"The specified bucket does not exist.","domain":"global","reason":"notFound"}]}}
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [DEBUG] Retry Transport: Returning after 1 attempts
2024-03-20T00:26:42.066Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [WARN] Removing Container Registry Storage Bucket "eu.artifacts.mc-server-416313.appspot.com" because it's gone
2024-03-20T00:26:42.069Z [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-03-20T00:26:42.069Z [ERROR] vertex "google_container_registry.jjmc_container_registry" error: Provider produced inconsistent result after apply
2024-03-20T00:26:42.074Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [DEBUG] [transport] [server-transport 0xc000bca1a0] Closing: Server.Stop called 
2024-03-20T00:26:42.074Z [DEBUG] provider.terraform-provider-google_v5.20.0_x5.exe: 2024/03/20 00:26:42 [DEBUG] [transport] [server-transport 0xc000bca1a0] loopyWriter exiting with error: transport closed by client 
2024-03-20T00:26:42.074Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-20T00:26:42.098Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.20.0/windows_amd64/terraform-provider-google_v5.20.0_x5.exe pid=273668
2024-03-20T00:26:42.098Z [DEBUG] provider: plugin exited
ggtisc commented 7 months ago

It had originally been reported as an error with google_container_registry but then you reported the google_storage_bucket service, which is the one that is having the error according to your logs. Could you please confirm if you are creating the next resources as shown?

resource "google_container_registry" "jjmc_container_registry" {
  location = "EU"
}

resource "google_storage_bucket" "server_backups" {
  name          = "jjmc_server_backups"
  location      = "europe-west2"
  force_destroy = true
}

The 1st error said the next:

│ When applying changes to google_container_registry.jjmc_container_registry, provider
│ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.

But now you have this one:

"error": {
   "code": 404,
   "message": "The specified bucket does not exist.",
   "errors": [
    {
     "message": "The specified bucket does not exist.",
     "domain": "global",
     "reason": "notFound"
    }
   ]
}

Could you please confirm that your terraform state is clear with terraform state list?

Jerome-Paddick commented 7 months ago

My issue is with the google_container_registry.

I used the storage bucket as an example of a resource that I could create with my configuration, but I am not provisioning that currently

Here is my entire terraform 

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.20.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.20.0"
    }
  }
}

provider "google" {
  credentials = file("${path.module}/../shared/creds.json")
  project     = "mc-server-416313"
  region      = "europe-west2"
}

resource "google_container_registry" "jjmc_container_registry" {
  location = "EU"
}

output "registry_url" {
  value = google_container_registry.jjmc_container_registry.bucket_self_link
}

and the error

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to google_container_registry.jjmc_container_registry, provider
│ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.  
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

My state file terraform.tfstate

{
  "version": 4,
  "terraform_version": "1.7.4",
  "serial": 1,
  "lineage": "031fe179-365e-0f8d-8072-629ed8ea78bf",
  "outputs": {},
  "resources": [],
  "check_results": null
}

terraform state list returns nothing

Jerome-Paddick commented 7 months ago

Im trying to figure out why terraform was trying to access that bucket in the logs

Its attempting to hit this endpoint when it got the error GET /storage/v1/b/eu.artifacts.mc-server-416313.appspot.com?alt=json&prettyPrint=false HTTP/1.1

From looking at the google api docs It would appear that its trying to access this bucket eu.artifacts.mc-server-416313.appspot.com which currently doesnt exist

According to this forum post https://issuetracker.google.com/issues/254292379?pli=1

The [region].artifacts.[account_id].appspot.com buckets are created by google to store container artifacts.

Could it be that google no longer automatically creates the container artifacts buckets but terraform is still looking for it?

roaks3 commented 7 months ago

Hi @Jerome-Paddick , this resource should "create the backing bucket if it does not exist, or do nothing if the bucket already exists", and it seems that isn't working in your case. I suspect this could be due to eventual consistency, which is a common issue with buckets. The resource creation could be creating the bucket as a by-product of the /token call, but then when we try to read back the bucket with /storage/v1/b/..., it could return a 404 for a short time.

Unfortunately, we haven't been able to reproduce thus far, so I will give it a few more tries tomorrow morning. If it happens inconsistently, a timing problem would be a likely explanation. The solution would most likely involve a code change to have the Read function poll for the bucket to become available.

roaks3 commented 6 months ago

I was still unsuccessful in reproducing this, so even though we have a hypothesis around what is happening, I'm hesitant to make any code changes until we know there is an observable issue that it will resolve.

I was able to confirm that applying the config the first time will create the bucket correctly for me. Then, on subsequent apply/destroy operations, the bucket will be reused. It also applies so quickly that I'm skeptical of a timing issue.

@Jerome-Paddick to move forward on this could you confirm the following:

  1. Have you been able to reproduce this issue, or are your messages referencing a single time where this occurred?
  2. After you apply and see the error, can you confirm through the UI that the bucket was created? Even after waiting ~5 mins?

My guess at this moment would be that there is a permission issue causing your bucket to not be created. Specifically, I would look at adding the roles/storage.admin as mentioned in the docs, which I'm fairly certain includes some permissions that are not present in roles/owner.

Jerome-Paddick commented 6 months ago

HI @roaks3

  1. I get the error every time I run the apply command, which has been multiple times over multiple days

  2. When running the apply command, the eu.gcr.io container registry is created instantly. The eu.artifacts.mc-server-416313.appspot.com bucket is never created. I have given my service account both Owner and Storage Admin permissions

FWIW- Trying to manually create the eu.artifacts.mc-server-416313.appspot.com bucket results in the following error message To create a bucket named eu.artifacts.mc-server-416313.appspot.com, you must verify that you're authorized to use that domain name. Follow the link in the 'Name your bucket' step for verification instructions.

roaks3 commented 6 months ago

Yea, this is strange.

We don't have a service team to forward this to, so unfortunately it is much harder for us to prioritize further investigation and a fix (which is looking like it will be time consuming). If there is any other information you can provide that would certainly help, otherwise I think this will be effectively blocked until we can determine a minimal reproduction.

FWIW, I was not able to find any indication of a change that was made to GCR to cause this, but since the service is being deprecated, it is possible that something was changed in a way that broke certain workflows. There is also this internal ticket b/295914313 mentioning a high volume of bucket 404s from Sept 2023, which is still unexplained, and could be related to this use case.

Jerome-Paddick commented 6 months ago

@roaks3 Thanks for looking into it

yeah, looks like gcp is depreciating google_container_registry in favour of google_artifact_registry_repository in the next month so probably not worth chasing a complicated fix

Maybe that info could be added to the google_container_registry docs in the mean time

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.