GoogleCloudPlatform / pbmm-on-gcp-onboarding

GCP Canadian Public Sector Landing Zone overlay on top of the TEF via CFT modules - a secure cloud foundation
https://cloud.google.com/architecture/security-foundations
Apache License 2.0
39 stars 55 forks source link

TEF upstream git resync and terraform provider updates cadence #387

Open fmichaelobrien opened 2 months ago

fmichaelobrien commented 2 months ago

TEF V5 tracking https://github.com/terraform-google-modules/terraform-example-foundation/labels/v5.0

see fork reference https://github.com/CloudLandingZone/terraform-example-foundation

Branch: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/tree/gh387-upstream-sync

PRs merged to the terraform-example-foundation upstream repo

terraform-example-foundation upstream resync procedure - copy not fork

cd _upstream
git clone https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding.git
cd pbmm-on-gcp-onboarding 
git checkout gh387-upstream-sync 

michaelobrien@mbp7 pbmm-on-gcp-onboarding % git status
On branch gh387-upstream-sync
Your branch is up to date with 'origin/gh387-upstream-sync'.

git remote add upstream https://github.com/terraform-google-modules/terraform-example-foundation.git
git fetch upstream
git merge upstream/master gh387-upstream-sync
fatal: refusing to merge unrelated histories

Indirect terraform provider updates

@nbugden thank you for the bucket names fix Merge in 63 character bucket naming standard fix indirectly via the next terraform init - that pull in in the latest generated terraform-google-provider via the magic modules compile

https://github.com/GoogleCloudPlatform/magic-modules/pull/10426 for https://github.com/hashicorp/terraform-provider-google/issues/17831

following previous pr https://github.com/terraform-google-modules/terraform-google-cloud-storage/pull/308

closes https://github.com/terraform-google-modules/terraform-example-foundation/issues/1169 https://github.com/terraform-google-modules/terraform-google-cloud-storage/issues/307

upgrade the provider on the next init (1.3.10) and 1.6 (we are not using the default 1.7.5 yet) https://github.com/hashicorp/terraform-provider-google?tab=readme-ov-file#upgrading-the-provider

terraform init -upgrade

Sync procedure jira needed for https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/376

fmichaelobrien commented 2 months ago

Indirect

Direct

obriensystems commented 2 months ago

Todo: keep the repos relatively in sync nonproduction vs non-production for example

Screenshot 2024-04-24 at 11 57 17
obriensystems commented 2 months ago
  nonproduction_folder_name         = data.terraform_remote_state.env_nonproduction.outputs.env_folder

resource "gitlab_branch_protection" "non_production" {
 tf_apply_branches = ["development", "non\\-production", "production"]

Align with PR from 3 weeks ago upstream https://github.com/terraform-google-modules/terraform-example-foundation/pull/1107 https://github.com/terraform-google-modules/terraform-example-foundation/commit/eb28ecb1ca659b1279db83b0b7f9667d56698c22#diff-446d88326ec6ad0370d30957f6546694390e92cc997a810e60783bfeb016efc1L308

obriensystems commented 2 months ago

prepare for TEF to PBMM sync fix the main branch first - by re-adding cloud build support

421

michaelobrien@mbp7 terraform-example-foundation % git pull
remote: Enumerating objects: 120, done.
remote: Counting objects: 100% (119/119), done.
remote: Compressing objects: 100% (80/80), done.
remote: Total 120 (delta 52), reused 78 (delta 33), pack-reused 1
Receiving objects: 100% (120/120), 103.94 KiB | 1.07 MiB/s, done.
Resolving deltas: 100% (52/52), completed with 7 local objects.
From https://github.com/terraform-google-modules/terraform-example-foundation
   a63dc88..034b995  master                                                         -> origin/master
 * [new branch]      dependabot/go_modules/helpers/foundation-deployer/github.com/hashicorp/go-getter-1.7.4 -> origin/dependabot/go_modules/helpers/foundation-deployer/github.com/hashicorp/go-getter-1.7.4
 + 5653048...218bb21 dependabot/go_modules/helpers/foundation-deployer/golang.org/x/net-0.23.0 -> origin/dependabot/go_modules/helpers/foundation-deployer/golang.org/x/net-0.23.0  (forced update)
 * [new branch]      dependabot/go_modules/test/integration/github.com/hashicorp/go-getter-1.7.4 -> origin/dependabot/go_modules/test/integration/github.com/hashicorp/go-getter-1.7.4
 + 4d6d15d...79f5c67 dependabot/go_modules/test/integration/golang.org/x/net-0.23.0 -> origin/dependabot/go_modules/test/integration/golang.org/x/net-0.23.0  (forced update)
 + d4c27d3...4318c63 release-please--branches--master                               -> origin/release-please--branches--master  (forced update)
   ddeae49..1c2e4a8  renovate/go-modules-andor-dev-tools                            -> origin/renovate/go-modules-andor-dev-tools
 + 38e9441...7a1e347 renovate/google-cloud-security-center-8.x                      -> origin/renovate/google-cloud-security-center-8.x  (forced update)
Updating a63dc88..034b995
Fast-forward
 .github/workflows/go-lint.yaml                                     |  6 +++---
 0-bootstrap/README.md                                              | 14 ++++++++++++++
 1-org/envs/shared/terraform.example.tfvars                         |  4 ----
 2-environments/envs/nonproduction/terraform.tfvars                 |  2 +-
 4-projects/business_unit_1/nonproduction/nonproduction.auto.tfvars |  2 +-
 4-projects/business_unit_2/nonproduction/nonproduction.auto.tfvars |  2 +-
 README.md                                                          |  2 +-
 go.work                                                            |  2 +-
 helpers/foundation-deployer/README.md                              |  2 +-
 helpers/foundation-deployer/go.mod                                 |  2 +-
 test/integration/go.mod                                            |  2 +-
 test/integration/networks/networks_test.go                         |  2 +-
 test/integration/testutils/retry.go                                |  3 +++
 13 files changed, 29 insertions(+), 16 deletions(-)

keep

Sync

michaelobrien@mbp7 _upstream % cd pbmm-on-gcp-onboarding 
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git checkout main
Already on 'main'
Your branch is up to date with 'origin/main'.
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git checkout gh387-upstream-sync
Switched to branch 'gh387-upstream-sync'
Your branch is up to date with 'origin/gh387-upstream-sync'.
michaelobrien@mbp7 pbmm-on-gcp-onboarding % git merge main
Updating b9fe8d39..d7344f5f
Fast-forward
 README.md | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
michaelobrien@mbp7 pbmm-on-gcp-onboarding % 
ichaelobrien@mbp7 pbmm-on-gcp-onboarding % git push origin gh387-upstream-sync
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: 
remote: GitHub found 5 vulnerabilities on GoogleCloudPlatform/pbmm-on-gcp-onboarding's default branch (2 critical, 3 moderate). To find out more, visit:
remote:      https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/security/dependabot
remote: 
To https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding.git
   b9fe8d39..d7344f5f  gh387-upstream-sync -> gh387-upstream-sync
fmichaelobrien commented 2 months ago

20240504: Repo state: CB/CSR are the default for the main branch

The main branch is ready for Cloud Build / Cloud Source Repositories out of the box. For ADO support this is in queue via https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/399 after a TEF upstream merge via https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/387

obriensystems commented 1 month ago

Sync 20240511 version of the following master branch https://github.com/terraform-google-modules/terraform-example-foundation/ via branch https://github.com/terraform-google-modules/terraform-example-foundation/pull/1199 with last PR https://github.com/terraform-google-modules/terraform-example-foundation/pull/1199

into https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/tree/gh387-upstream-sync

0-bootstrap, 1-org done

tef minor bug for label changes in https://github.com/terraform-google-modules/terraform-example-foundation/pull/1199/files#diff-d6697e7c916ba73d6ae87ff4b1ce67cabc9b9738ab31c9ba582e2a3218982838L279

https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/envs/shared/projects.tf#L253 match https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/envs/shared/projects.tf#L237

and -net-interconnect

just like in https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/envs/shared/projects.tf#L295

3-networks-hub-and-spoke

https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars

-../../access_context.auto.tfvars
+../shared/access_context.auto.tfvars

there is a 2nd symlink regression in the same directory in https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars

-../../common.auto.tfvars
+../shared/common.auto.tfvars

I'll raise an issue shortly with the fix after I test a full upstream sync of the TEF.

Screenshot 2024-05-11 at 19 09 03
obriensystems commented 1 month ago

130 files - ignore the 2nd email CLA, now testing the branch before prepping for main to master - after a couple PRs in queue https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/pull/435

fmichaelobrien commented 1 month ago

I completed the TEF upstream sync - 130 changes to get up to 20240511 main branch status. Details in the PR below https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/pull/435 I will hold off on merging the branch below until I test a clean deploy - and also likely wait for your PR - so I will take the merge of your changes - instead of the other way around. https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/tree/gh387-upstream-sync for https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/387

fmichaelobrien commented 1 month ago

as part of https://github.com/terraform-google-modules/terraform-example-foundation/issues/1229 fix symlinks

fmichaelobrien-macbookpro:terraform-example-foundation fmichaelobrien$ git status
On branch gh1229-fix-symlinks
Your branch is up to date with 'origin/gh1229-fix-symlinks'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars
    modified:   3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars

no changes added to commit (use "git add" and/or "git commit -a")
fmichaelobrien-macbookpro:terraform-example-foundation fmichaelobrien$ git diff
diff --git a/3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars b/3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars
index 59c7aa1..b0cccce 120000
--- a/3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars
+++ b/3-networks-hub-and-spoke/envs/nonproduction/access_context.auto.tfvars
@@ -1 +1 @@
-../shared/access_context.auto.tfvars
\ No newline at end of file
+../../access_context.auto.tfvars
\ No newline at end of file
diff --git a/3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars b/3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars
index 3316647..39aaa46 120000
--- a/3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars
+++ b/3-networks-hub-and-spoke/envs/nonproduction/common.auto.tfvars
@@ -1 +1 @@
-../shared/common.auto.tfvars
\ No newline at end of file
+../../common.auto.tfvars
\ No newline at end of file
fmichaelobrien commented 1 month ago

pending June upstream sync

Direct

fmichaelobrien commented 1 month ago

missing https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/test/restore_tf_files.sh architecture.md

fmichaelobrien commented 1 month ago

Merge duplicate environment removal in a patch just after sunday's pull https://github.com/terraform-google-modules/terraform-example-foundation/pull/1199/commits/061e7693d9445923952f3a486fbaf37750eb8678

checked - already removed https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/gh387-upstream-sync/1-org/envs/shared/projects.tf#L294

obriensystems commented 1 month ago

pull 20240517:1500

chaelobrien@mbp7 terraform-example-foundation % git pull
remote: Enumerating objects: 180, done.
remote: Counting objects: 100% (180/180), done.
remote: Compressing objects: 100% (93/93), done.
remote: Total 180 (delta 96), reused 148 (delta 78), pack-reused 0
Receiving objects: 100% (180/180), 116.39 KiB | 3.75 MiB/s, done.
Resolving deltas: 100% (96/96), completed with 15 local objects.
From https://github.com/terraform-google-modules/terraform-example-foundation
   034b995..0a69eac  master                                                        -> origin/master
 + 4318c63...e686f63 release-please--branches--master                              -> origin/release-please--branches--master  (forced update)
 + 1c2e4a8...64f8d52 renovate/go-modules-andor-dev-tools                           -> origin/renovate/go-modules-andor-dev-tools  (forced update)
 * [new branch]      renovate/googlecloudplatform-lb-internal-google-6.x           -> origin/renovate/googlecloudplatform-lb-internal-google-6.x
 * [new branch]      renovate/terraform-google-modules-cloud-storage-google-6.x    -> origin/renovate/terraform-google-modules-cloud-storage-google-6.x
 * [new branch]      renovate/terraform-google-modules-project-factory-google-15.x -> origin/renovate/terraform-google-modules-project-factory-google-15.x
Updating 034b995..0a69eac
Fast-forward
 .github/workflows/go-lint.yaml                                       |  8 ++++----
 .github/workflows/go-test.yaml                                       | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1-org/envs/shared/projects.tf                                        | 13 ++++++-------
 3-networks-dual-svpc/envs/shared/dns-hub.tf                          | 16 ++++++++--------
 3-networks-dual-svpc/envs/shared/interconnect.tf.example             |  2 +-
 3-networks-dual-svpc/envs/shared/partner_interconnect.tf.example     |  2 +-
 3-networks-dual-svpc/modules/base_shared_vpc/dns.tf                  |  2 +-
 3-networks-dual-svpc/modules/partner_interconnect/README.md          |  2 +-
 3-networks-dual-svpc/modules/restricted_shared_vpc/dns.tf            |  2 +-
 3-networks-hub-and-spoke/envs/shared/dns-hub.tf                      | 16 ++++++++--------
 3-networks-hub-and-spoke/envs/shared/interconnect.tf.example         |  2 +-
 3-networks-hub-and-spoke/envs/shared/partner_interconnect.tf.example |  2 +-
 3-networks-hub-and-spoke/modules/base_shared_vpc/dns.tf              |  2 +-
 3-networks-hub-and-spoke/modules/partner_interconnect/README.md      |  2 +-
 3-networks-hub-and-spoke/modules/restricted_shared_vpc/dns.tf        |  2 +-
 Makefile                                                             |  2 +-
 README.md                                                            | 16 ++++++++--------
 build/int.cloudbuild.yaml                                            |  2 +-
 build/lint.cloudbuild.yaml                                           |  2 +-
 docs/upgrading_to_v4.0.md                                            |  8 ++++----
 helpers/foundation-deployer/README.md                                |  2 +-
 helpers/foundation-deployer/go.mod                                   | 27 ++++++++++++++-------------
 helpers/foundation-deployer/go.sum                                   | 40 ++++++++++++++++++++++++++++++----------
 test/integration/go.mod                                              | 34 ++++++++++++++++------------------
 test/integration/go.sum                                              | 70 ++++++++++++++++++++++++++++++++++++----------------------------------
 test/integration/shared/shared_test.go                               | 18 +++++++++---------
 26 files changed, 208 insertions(+), 137 deletions(-)
 create mode 100644 .github/workflows/go-test.yaml
fmichaelobrien commented 1 month ago

Merge Marian's single yaml config with the removal of BU2 - (reduces project count from 51) https://github.com/terraform-google-modules/terraform-example-foundation/pull/1241 as part of https://github.com/terraform-google-modules/terraform-example-foundation/issues/1238