Closed nbugden closed 2 months ago
The same fix will be needed in the locals default for 3-networks-hub-and-spoke
Good catch @fmichaelobrien ! Fixed in this commit: https://github.com/terraform-google-modules/terraform-example-foundation/pull/1181/commits/7c4409fe3f036eece967fce9f688794b0208578f
@rjerrems @eeaton @gtsorbo can one of you folks take a look. PR needs /gcbrun
and an approving review.
@rjerrems @eeaton @gtsorbo can one of you folks take a look. PR needs
/gcbrun
and an approving review.
Nudge, can I get a review. If there is something else required before review, please let me know.
@gtsorbo @daniel-cit I believe I've addressed everything in your change requests. Can you review.
@nbugden the PR should also be renamed to fix!: use bootstrap.outputs.common_config as default region
since this is a braking change (subnetworks need to be recreated).
@nbugden is the documentation still reflected in the code based on the changes in the subnetwork ? https://cloud.google.com/architecture/security-foundations/networking#ip-address-allocation
@nbugden the PR should also be renamed to
fix!: use bootstrap.outputs.common_config as default region
since this is a braking change (subnetworks need to be recreated).
Done
@nbugden is the documentation still reflected in the code based on the changes in the subnetwork ? https://cloud.google.com/architecture/security-foundations/networking#ip-address-allocation
@daniel-cit Yes, the changes to the subnets are still inline with the documentation. The doc you shared states region1
and region2
. It does not make reference to specific data centers (us-west1
or us-central1
). This PR does not change the CIDRs, just the locations in which they are provisioned.
/gcbrun
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: Error: Duplicate object key
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185:
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: on .terraform/modules/dns_hub_vpc/modules/subnets/main.tf line 20, in locals:
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: 18: subnets = {
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: 19: for x in var.subnets :
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: 20: "${x.subnet_region}/${x.subnet_name}" => x
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: 21: }
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: ├────────────────
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: │ x.subnet_name is "sb-c-dns-hub-us-west1"
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: │ x.subnet_region is "us-west1"
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185:
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: Two different items produced the key "us-west1/sb-c-dns-hub-us-west1" in this
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: 'for' expression. If duplicates are expected, use the ellipsis (...) after
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z command.go:185: the value expression to enable grouping by key.
Step #13 - "converge-shared": TestShared 2024-04-26T20:29:18Z retry.go:99: Returning due to fatal error: FatalError{Underlying: error while running command: exit status 1;
Step #13 - "converge-shared": Error: Duplicate object key
Step #13 - "converge-shared":
Step #13 - "converge-shared": on .terraform/modules/dns_hub_vpc/modules/subnets/main.tf line 20, in locals:
Step #13 - "converge-shared": 18: subnets = {
Step #13 - "converge-shared": 19: for x in var.subnets :
Step #13 - "converge-shared": 20: "${x.subnet_region}/${x.subnet_name}" => x
Step #13 - "converge-shared": 21: }
Step #13 - "converge-shared": ├────────────────
Step #13 - "converge-shared": │ x.subnet_name is "sb-c-dns-hub-us-west1"
Step #13 - "converge-shared": │ x.subnet_region is "us-west1"
Step #13 - "converge-shared":
Step #13 - "converge-shared": Two different items produced the key "us-west1/sb-c-dns-hub-us-west1" in this
Step #13 - "converge-shared": 'for' expression. If duplicates are expected, use the ellipsis (...) after
Step #13 - "converge-shared": the value expression to enable grouping by key.}
Step #13 - "converge-shared": plan.go:34:
Step #13 - "converge-shared": Error Trace: /builder/home/go/pkg/mod/github.com/gruntwork-io/terratest@v0.46.11/modules/terraform/plan.go:34
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/tft/terraform.go:490
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/tft/terraform.go:503
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/tft/terraform.go:527
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/tft/terraform.go:557
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/utils/stages.go:31
Step #13 - "converge-shared": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.12.1/pkg/tft/terraform.go:557
Step #13 - "converge-shared": Error: Received unexpected error:
Step #13 - "converge-shared": FatalError{Underlying: error while running command: exit status 1;
Step #13 - "converge-shared": Error: Duplicate object key
Step #13 - "converge-shared":
Step #13 - "converge-shared": on .terraform/modules/dns_hub_vpc/modules/subnets/main.tf line 20, in locals:
Step #13 - "converge-shared": 18: subnets = {
Step #13 - "converge-shared": 19: for x in var.subnets :
Step #13 - "converge-shared": 20: "${x.subnet_region}/${x.subnet_name}" => x
Step #13 - "converge-shared": 21: }
Step #13 - "converge-shared": ├────────────────
Step #13 - "converge-shared": │ x.subnet_name is "sb-c-dns-hub-us-west1"
Step #13 - "converge-shared": │ x.subnet_region is "us-west1"
Step #13 - "converge-shared":
Step #13 - "converge-shared": Two different items produced the key "us-west1/sb-c-dns-hub-us-west1" in this
Step #13 - "converge-shared": 'for' expression. If duplicates are expected, use the ellipsis (...) after
Step #13 - "converge-shared": the value expression to enable grouping by key.}
Step #13 - "converge-shared": Test: TestShared
@daniel-cit thanks for sending over the error logs. I took a look and I can't seem to find the relevant code that is producing it.
The dns-hub subnets are provisioned as part of this module. It shows 2 subnets, one with local.default_region1
and one with local.default_region2
. When I look at the locals, I see that it is referencing the outputs correctly. Furthermore, when I double check the outputs from the bootstrap, the defaults are correct as well.
I've also used this config in a new org set up recently and wasn't able to see any duplicate names in the dns-hub set up. Any thoughts?
@daniel-cit thanks for sending over the error logs. I took a look and I can't seem to find the relevant code that is producing it.
The dns-hub subnets are provisioned as part of this module. It shows 2 subnets, one with
local.default_region1
and one withlocal.default_region2
. When I look at the locals, I see that it is referencing the outputs correctly. Furthermore, when I double check the outputs from the bootstrap, the defaults are correct as well.I've also used this config in a new org set up recently and wasn't able to see any duplicate names in the dns-hub set up. Any thoughts?
@nbugden maybe here? https://github.com/nbugden/terraform-example-foundation/blob/fix/bootstrap-default-region/build/int.cloudbuild.yaml#L174C1-L177C38
@nbugden maybe here? https://github.com/nbugden/terraform-example-foundation/blob/fix/bootstrap-default-region/build/int.cloudbuild.yaml#L174C1-L177C38
Thanks @daniel-cit! I totally missed the TF_VARS
in the pipeline. Just pushed the correct values. I also found a couple issues in the shared int tests that I fixed in 0aaf492
Can you /gcbrun again please.
/gcbrun
@daniel-cit do you know if there is reason there are TF_VAR
s set for the regions in the integration pipeline? Seems redundant to set the values to what are already the default values. But perhaps there is a reason for it that I'm unaware?
/gcbrun
us-central
=> us-central1
.
Step #14 - "verify-shared": TestShared 2024-05-02T19:11:54Z command.go:100: Running command gcloud with args [compute networks subnets describe sb-c-dns-hub-us-central1 --region us-central --project cdg-c-dns-hub-42ex --format json]
Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185: ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource:
Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185: - Invalid value for field 'region': 'us-central'. Unknown region.
Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185:
Step #14 - "verify-shared": gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource:
Step #14 - "verify-shared": - Invalid value for field 'region': 'us-central'. Unknown region.
us-central
=>us-central1
.Step #14 - "verify-shared": TestShared 2024-05-02T19:11:54Z command.go:100: Running command gcloud with args [compute networks subnets describe sb-c-dns-hub-us-central1 --region us-central --project cdg-c-dns-hub-42ex --format json] Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185: ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource: Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185: - Invalid value for field 'region': 'us-central'. Unknown region. Step #14 - "verify-shared": TestShared 2024-05-02T19:11:55Z command.go:185: Step #14 - "verify-shared": gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource: Step #14 - "verify-shared": - Invalid value for field 'region': 'us-central'. Unknown region.
Thanks @daniel-cit , fixed in b62b3c029245987648ee4d180d0b0b1bd307875b
/gcbrun
/gcbrun
@nbugden
Step #17 - "verify-networks": TestNetworks/nonproduction 2024-05-03T15:19:50Z command.go:185: ERROR: (gcloud.compute.routers.describe) Could not fetch resource:
Step #17 - "verify-networks": TestNetworks/nonproduction 2024-05-03T15:19:50Z command.go:185: - The resource 'projects/r27-n-shared-restricted-kylv/regions/us-central1/routers/cr-n-shared-restricted-us-central-cr5' was not found
Step #17 - "verify-networks": TestNetworks/nonproduction 2024-05-03T15:19:50Z command.go:185:
Step #17 - "verify-networks": === NAME TestNetworks/nonproduction
Step #17 - "verify-networks": gcloud.go:84: error while running command: exit status 1; WARNING: This command is using service account impersonation. All API calls will be executed as [sa-terraform-net@r27-b-seed-703a.iam.gserviceaccount.com].
Step #17 - "verify-networks": WARNING: This command is using service account impersonation. All API calls will be executed as [sa-terraform-net@r27-b-seed-703a.iam.gserviceaccount.com].
Step #17 - "verify-networks": ERROR: (gcloud.compute.routers.describe) Could not fetch resource:
cr-n-shared-restricted-us-central-cr5
=> cr-n-shared-restricted-us-central1-cr5
@daniel-cit do you know if there is reason there are
TF_VAR
s set for the regions in the integration pipeline? Seems redundant to set the values to what are already the default values. But perhaps there is a reason for it that I'm unaware?
Think that now they are redundant and could be removed since they are been read from the remote state.
The only renaming usage is here https://github.com/terraform-google-modules/terraform-example-foundation/blob/6fcd442fef22cf1cafeae23c2a41be2c9e0534ca/test/integration/projects/projects_test.go#L239
this one should be read from the bootstrap output
/gcbrun
Sorry @daniel-cit needs /gcbrun again. I just refactored the test to use the bootstrap outputs
/gcbrun
Sorry @daniel-cit one more time... just fixed cr-n-shared-restricted-us-central-cr5 => cr-n-shared-restricted-us-central1-cr5
/gcbrun
@nbugden
"converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: Error: Unsupported attribute
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185:
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: on ../../modules/base_env/remote.tf line 22, in locals:
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: 22: perimeter_name = data.terraform_remote_state.network_env.outputs.restricted_service_perimeter_name
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: ├────────────────
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: │ data.terraform_remote_state.network_env.outputs is object with no attributes
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185:
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: This object does not have an attribute named
Step #22 - "converge-projects": TestProjects/bu1_nonproduction 2024-05-03T22:39:05Z command.go:185: "restricted_service_perimeter_name".
This one is missing the non-production
=> nonproduction
fix
env = "non-production"
/gcbrun
@daniel-cit can you share the error from the latest run
@nbugden
Step #25 - "converge-appinfra": TestAppInfra/nonproduction 2024-05-06T15:56:14Z retry.go:99: Returning due to fatal error: FatalError{Underlying: error while running command: exit status 1;
Step #25 - "converge-appinfra": Error: Error loading state error
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": with data.terraform_remote_state.bootstrap,
Step #25 - "converge-appinfra": on remote.tf line 22, in data "terraform_remote_state" "bootstrap":
Step #25 - "converge-appinfra": 22: backend = "gcs"
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": error loading the remote state: 2 errors occurred:
Step #25 - "converge-appinfra": * writing
Step #25 - "converge-appinfra": "gs://bkt-kb0-b-seed-398c-gcp-projects-tfstate/terraform/bootstrap/state/default.tflock"
Step #25 - "converge-appinfra": failed: googleapi: Error 403:
Step #25 - "converge-appinfra": sa-tf-cb-bu1-example-app@kb0-c-bu1-infra-pipeline-qewz.iam.gserviceaccount.com
Step #25 - "converge-appinfra": does not have storage.objects.create access to the Google Cloud Storage
Step #25 - "converge-appinfra": object. Permission 'storage.objects.create' denied on resource (or it may not
Step #25 - "converge-appinfra": exist)., forbidden
Step #25 - "converge-appinfra": * storage: object doesn't exist
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": }
Step #25 - "converge-appinfra": === NAME TestAppInfra/nonproduction
Step #25 - "converge-appinfra": plan.go:34:
Step #25 - "converge-appinfra": Error Trace: /builder/home/go/pkg/mod/github.com/gruntwork-io/terratest@v0.46.14/modules/terraform/plan.go:34
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:551
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:501
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:516
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:577
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:616
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/utils/stages.go:31
Step #25 - "converge-appinfra": /builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.14.0/pkg/tft/terraform.go:616
Step #25 - "converge-appinfra": Error: Received unexpected error:
Step #25 - "converge-appinfra": FatalError{Underlying: error while running command: exit status 1;
Step #25 - "converge-appinfra": Error: Error loading state error
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": with data.terraform_remote_state.bootstrap,
Step #25 - "converge-appinfra": on remote.tf line 22, in data "terraform_remote_state" "bootstrap":
Step #25 - "converge-appinfra": 22: backend = "gcs"
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": error loading the remote state: 2 errors occurred:
Step #25 - "converge-appinfra": * writing
Step #25 - "converge-appinfra": "gs://bkt-kb0-b-seed-398c-gcp-projects-tfstate/terraform/bootstrap/state/default.tflock"
Step #25 - "converge-appinfra": failed: googleapi: Error 403:
Step #25 - "converge-appinfra": sa-tf-cb-bu1-example-app@kb0-c-bu1-infra-pipeline-qewz.iam.gserviceaccount.com
Step #25 - "converge-appinfra": does not have storage.objects.create access to the Google Cloud Storage
Step #25 - "converge-appinfra": object. Permission 'storage.objects.create' denied on resource (or it may not
Step #25 - "converge-appinfra": exist)., forbidden
Step #25 - "converge-appinfra": * storage: object doesn't exist
Step #25 - "converge-appinfra":
Step #25 - "converge-appinfra": }
Step #25 - "converge-appinfra": Test: TestAppInfra/nonproduction
App infra service account does not have access to the bootstrap bucket in the seed project, 5-app-infra reads data from the 4-project state bucket
this was done to isolate access to the main bucket from the step 5
data from other steps are shared thru the output os step 4 like
@daniel-cit fixed in 7989afed2ff4c62271fba579da1dc4bf6436e0d7
/gcbrun
/gcbrun
/gcbrun
@daniel-cit one of the pipelines has already failed and I had to resolve some conflicts in test/integration/shared/shared_test.go
.
Can you /gcbrun again?
/gcbrun
/gcbrun
@nbugden
it looks like a conflict with the changes from #1199
Step #14 - "verify-shared": TestShared 2024-05-09T14:44:08Z command.go:100: Running command gcloud with args [compute networks subnets describe sb-net-dns-hub-us-west1 --region us-west1 --project wp8-net-dns-rky3 --format json]
Step #14 - "verify-shared": TestShared 2024-05-09T14:44:09Z command.go:185: ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource:
Step #14 - "verify-shared": TestShared 2024-05-09T14:44:09Z command.go:185: - The resource 'projects/wp8-net-dns-rky3/regions/us-west1/subnetworks/sb-net-dns-hub-us-west1' was not found
Step #14 - "verify-shared": TestShared 2024-05-09T14:44:09Z command.go:185:
Step #14 - "verify-shared": gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.networks.subnets.describe) Could not fetch resource:
Step #14 - "verify-shared": - The resource 'projects/wp8-net-dns-rky3/regions/us-west1/subnetworks/sb-net-dns-hub-us-west1' was not found
Step #14 - "verify-shared":
Step #14 - "verify-shared": 2024/05/09 14:44:09 RUN_STAGE env var set to verify
Step #14 - "verify-shared": 2024/05/09 14:44:09 Skipping stage teardown
Step #14 - "verify-shared": --- FAIL: TestShared (82.33s)
Step #14 - "verify-shared": FAIL
Step #14 - "verify-shared": FAIL github.com/terraform-google-modules/terraform-example-foundation/test/integration/shared 82.396s
Step #14 - "verify-shared": FAIL
Step #14 - "verify-shared": t=2024-05-09T14:44:09+0000 lvl=eror msg="error running command: exit status 1"
Finished Step #14 - "verify-shared"
/gcbrun
/gcbrun
/gcbrun
/gcbrun
/gcbrun
@daniel-cit one of the integration test pipelines failed
/gcbrun
/gcbrun
/gcbrun
/gcbrun
/gcbrun
This PR adds some additional outputs
bootstrap.outputs.common_config
and uses these values by default. This fixes the issue of resources being deployed inus-central1
andus-west1
which was caused by module defaults (set in variables.tf) not being overridden when modules invoked. It also maintains the ability to override the defaults using theauto.tfvars
should that be desired.Issue(s) Resolved: https://github.com/terraform-google-modules/terraform-example-foundation/issues/1172
BEGIN_COMMIT_OVERRIDE fix: use bootstrap.outputs.common_config as default region (#1181) END_COMMIT_OVERRIDE