Open fmichaelobrien opened 2 years ago
On Lenovo x1 carbon g9 - to shadow non-OSX clients
Guardrails Install procedure for local SDK client (no gcloud init authentication possible in the browser)
gcloud init
check terraform
C:\opt>terraform --version
Terraform v1.1.0
on windows_amd64
Your version of Terraform is out of date! The latest version
is 1.2.1. You can update by downloading from https://www.terraform.io/downloads.html
update terraform
C:\wse_github\canada-ca>terraform --version
Terraform v1.2.1
on windows_amd64
switch to the current project
C:\wse_github\canada-ca>gcloud projects list
PROJECT_ID: accelerator-dev-cd
NAME: accelerator-dev-cd
PROJECT_NUMBER: 1044971174628
C:\wse_github\canada-ca>gcloud config set project accelerator-dev-cd
Updated property [core/project].
clone the repo
C:\wse_github\canada-ca>git clone https://github.com/canada-ca/accelerators_accelerateurs-gcp.git
update gcloud
gcloud components update
RROR: gcloud crashed (Error): [(u'C:\\opt\\gcloud\\google-cloud-sdk\\platform\\gsutil\\third_party\\funcsigs\\docs\\index.rst', u'C:\\opt\\gcloud\\google-cloud-sdk.staging\\platform\\gsutil\\third_party\\funcsigs\\docs\\index.rst', "[Errno 22] invalid mode ('rb') or filename: u'C:\\\\opt\\\\gcloud\\\\google-cloud-sdk\\\\platform\\\\gsutil\\\\third_party\\\\funcsigs\\\\docs\\\\index.rst'"), (u'C:\\opt\\gcloud\\google-cloud-sdk\\platform\\gsutil\\third_party\\mock\\docs\\changelog.txt', u'C:\\opt\\gcloud\\google-cloud-sdk.staging\\platform\\gsutil\\third_party\\mock\\docs\\changelog.txt', "[Errno 22] invalid mode ('rb') or filename: u'C:\\\\opt\\\\gcloud\\\\google-cloud-sdk\\\\platform\\\\gsutil\\\\third_party\\\\mock\\\\docs\\\\changelog.txt'"), (u'C:\\opt\\gcloud\\google-cloud-sdk\\platform\\gsutil_py2\\third_party\\funcsigs\\docs\\index.rst', u'C:\\opt\\gcloud\\google-cloud-sdk.staging\\platform\\gsutil_py2\\third_party\\funcsigs\\docs\\index.rst', "[Errno 22] invalid mode ('rb') or filename: u'C:\\\\opt\\\\gcloud\\\\google-cloud-sdk\\\\platform\\\\gsutil_py2\\\\third_party\\\\funcsigs\\\\docs\\\\index.rst'"), (u'C:\\opt\\gcloud\\google-cloud-sdk\\platform\\gsutil_py2\\third_party\\mock\\docs\\changelog.txt', u'C:\\opt\\gcloud\\google-cloud-sdk.staging\\platform\\gsutil_py2\\third_party\\mock\\docs\\changelog.txt', "[Errno 22] invalid mode ('rb') or filename: u'C:\\\\opt\\\\gcloud\\\\google-cloud-sdk\\\\platform\\\\gsutil_py2\\\\third_party\\\\mock\\\\docs\\\\changelog.txt'")]
update in the gcloud console first
ERROR: gcloud crashed (LookupError): unknown encoding: cp65001
stick with instead
C:\wse_github\canada-ca\accelerators_accelerateurs-gcp\deployment-templates\Terraform\guardrails\0-bootstrap>gcloud --version
Google Cloud SDK 365.0.0
bq 2.0.71
core 2021.11.12
gsutil 5.5
Updates are available for some Cloud SDK components. To install them,
please run:
$ gcloud components update
hangs
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ gcloud init
Welcome! This command will take you through the configuration of gcloud.
Settings from your current configuration [default] are:
accessibility:
screen_reader: 'True'
core:
account: michael@clouddevops.dev
disable_usage_reporting: 'False'
project: accelerator-dev-cd
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ gcloud projects list
PROJECT_ID: accelerator-dev-cd
NAME: accelerator-dev-cd
PROJECT_NUMBER: 1044971174628
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ gcloud config set project accelerator-dev-cd
Updated property [core/project].
reinstall gcloud
C:\opt\CloudSDK>gcloud version
Google Cloud SDK 387.0.0
bq 2.0.74
core 2022.05.20
gsutil 5.10
in mingw64 (alternate to cloning https
$ eval $(ssh-agent -s)
Agent pid 560
$ ssh-add ~/.ssh/obrienlabs_org_github
micha@carbon MINGW64 /c/wse_github/canada-ca
$ cd accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap/
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ terraform --version
Terraform v1.2.1
on windows_amd64
micha@carbon MINGW64 ~
$ gcloud version
Google Cloud SDK 365.0.0
bq 2.0.71
core 2021.11.12
gsutil 5.5
after reinstalling
micha@carbon MINGW64 ~
$ gcloud version
Google Cloud SDK 387.0.0
bq 2.0.74
core 2022.05.20
gsutil 5.10
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ terraform --version
Terraform v1.2.1
on windows_amd64
micha@carbon MINGW64 /c/wse_github/canada-ca/accelerators_accelerateurs-gcp/deployment-templates/Terraform/guardrails/0-bootstrap (main)
$ ./bootstrap.sh -d dept -p accelerator-dev-cd
You do not currently have this command group installed. Using it
requires the installation of components: [alpha]
Your current Google Cloud CLI version is: 387.0.0
Installing components from version: 387.0.0
+----------------------------------------------+
| These components will be installed. |
+-----------------------+------------+---------+
| Name | Version | Size |
+-----------------------+------------+---------+
| gcloud Alpha Commands | 2022.05.20 | < 1 MiB |
+-----------------------+------------+---------+
For the latest full release notes, please visit:
https://cloud.google.com/sdk/release_notes
run in gcloud shell to upgrade
C:\wse_github>gcloud beta billing projects
You do not currently have this command group installed. Using it
requires the installation of components: [beta]
Your current Google Cloud CLI version is: 387.0.0
Installing components from version: 387.0.0
┌─────────────────────────────────────────────┐
│ These components will be installed. │
├──────────────────────┬────────────┬─────────┤
│ Name │ Version │ Size │
├──────────────────────┼────────────┼─────────┤
│ gcloud Beta Commands │ 2022.05.20 │ < 1 MiB │
└──────────────────────┴────────────┴─────────┘
For the latest full release notes, please visit:
https://cloud.google.com/sdk/release_notes
Do you want to continue (Y/n)? y
╔════════════════════════════════════════════════════════════╗
╠═ Creating update staging area ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Installing: gcloud Beta Commands ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Creating backup and activating new installation ═╣
╚════════════════════════════════════════════════════════════╝
ERROR: (gcloud) Access is denied: [C:\opt\CloudSDK\google-cloud-sdk\platform\PowerShell\GoogleCloud\1.0.1.10\fullclr\Google.Apis.Auth.dll]
Ensure you have the permissions to access the file and that the file is not in use.
The system cannot find the path specified.
I will rerun "as administrator" - same
gcloud powershell
PS C:\opt\CloudSDK> gcloud config set project accelerator-dev-cd
Updated property [core/project].
need wsl because of our sh script
PS C:\wse_github\canada-ca\accelerators_accelerateurs-gcp\deployment-templates\Terraform\guardrails\0-bootstrap> ./bootstrap.sh -d dept -p accelerator-dev-cp
OSX
michaelobrien@mbp7 accelerators_accelerateurs-gcp % gcloud init
Welcome! This command will take you through the configuration of gcloud.
You are logged in as: [mich
Pick cloud project to use:
[1] accelerator-dev-cd
[2] tactile-talon-347416
[3] Enter a project ID
[4] Create a new project
Please enter numeric choice or text value (must exactly match list item): 1
Your current project has been set to: [accelerator-dev-cd].
^C
michaelobrien@mbp7 accelerators_accelerateurs-gcp % gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
accelerator-dev-cd accelerator-dev-cd 1044971174628
tactile-talon-347416 My First Project 241289855975
check beta/alpha settings
michaelobrien@mbp7 accelerators_accelerateurs-gcp % gcloud beta billing projects
You do not currently have this command group installed. Using it
requires the installation of components: [beta]
Your current Google Cloud CLI version is: 387.0.0
Installing components from version: 387.0.0
┌─────────────────────────────────────────────┐
│ These components will be installed. │
├──────────────────────┬────────────┬─────────┤
│ Name │ Version │ Size │
├──────────────────────┼────────────┼─────────┤
│ gcloud Beta Commands │ 2022.05.20 │ < 1 MiB │
└──────────────────────┴────────────┴─────────┘
For the latest full release notes, please visit:
https://cloud.google.com/sdk/release_notes
Do you want to continue (Y/n)? y
╔════════════════════════════════════════════════════════════╗
╠═ Creating update staging area ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Installing: gcloud Beta Commands ═╣
╠════════════════════════════════════════════════════════════╣
╠═ Creating backup and activating new installation ═╣
╚════════════════════════════════════════════════════════════╝
Performing post processing steps...done.
Update done!
Restarting command:
$ gcloud beta billing projects
michaelobrien@mbp7 accelerators_accelerateurs-gcp % gcloud alpha billing projects describe accelerator-dev-cd
billingAccountName: billingAc
michaelobrien@mbp7 0-bootstrap % ./bootstrap.sh -d dept -p accelerator-dev-cd
zsh: permission denied: ./bootstrap.sh
michaelobrien@mbp7 0-bootstrap % chmod 777 bootstrap.sh
michaelobrien@mbp7 0-bootstrap % ./bootstrap.sh -d dept -p accelerator-dev-cd
seed project id: dept-seed-project
boostrap project id: accelerator-dev-cd
org id: 796553858496
billing id: 0128F5-0C5308-B9D171
Updated property [core/project].
checking roles of current account: michael@clouddevops.dev
iam.serviceAccountTokenCreator
roles/iam.serviceAccountTokenCreator role set OK on super admin account
roles/resourcemanager.folderAdmin
roles/resourcemanager.folderAdmin role set OK on super admin account
roles/resourcemanager.organizationAdmin
roles/resourcemanager.organizationAdmin role set OK on super admin account
orgpolicy.policyAdmin
roles/orgpolicy.policyAdmin role missing
michaelobrien@mbp7 0-bootstrap %
set "Organization Policy Administrator" on the SA user
michaelobrien@mbp7 0-bootstrap % ./bootstrap.sh -d dept -p accelerator-dev-cd
seed project id: dept-seed-project
boostrap project id: accelerator-dev-cd
org id: 796553858496
billing id: 0128F5-0C5308-B9D171
Updated property [core/project].
checking roles of current account: michael@clouddevops.dev
iam.serviceAccountTokenCreator
roles/iam.serviceAccountTokenCreator role set OK on super admin account
roles/resourcemanager.folderAdmin
roles/resourcemanager.folderAdmin role set OK on super admin account
roles/resourcemanager.organizationAdmin
roles/resourcemanager.organizationAdmin role set OK on super admin account
orgpolicy.policyAdmin
roles/orgpolicy.policyAdmin role missing
michaelobrien@mbp7 0-bootstrap % ./bootstrap.sh -d dept -p accelerator-dev-cd
seed project id: dept-seed-project
boostrap project id: accelerator-dev-cd
org id: 796553858496
billing id: 0128F5-0C5308-B9D171
Updated property [core/project].
checking roles of current account: michael@clouddevops.dev
iam.serviceAccountTokenCreator
roles/iam.serviceAccountTokenCreator role set OK on super admin account
roles/resourcemanager.folderAdmin
roles/resourcemanager.folderAdmin role set OK on super admin account
roles/resourcemanager.organizationAdmin
roles/resourcemanager.organizationAdmin role set OK on super admin account
orgpolicy.policyAdmin
roles/orgpolicy.policyAdmin role set OK on super admin account
resourcemanager.projectCreator
roles/resourcemanager.projectCreator role set OK on super admin account
billing.projectManager
roles/billing.projectManager role set OK on super admin account
all roles set OK on super admin account: michael@clouddevops.dev - proceeding
enabling pubsub.googleapis.com identitytoolkit cloudresourcemanager iam cloudbilling on accelerator-dev-cd project
Operation "operations/acat.p2-1044971174628-f6cf9a2f-0a78-4d52-8fe2-052fa3cdcfcc" finished successfully.
Listed 0 items.
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/dept-seed-project].
Waiting for [operations/cp.7938501216207354445] to finish...done.
Enabling service [cloudapis.googleapis.com] on project [dept-seed-project]...
Operation "operations/acat.p2-6631106591-0fc65011-5752-4d8a-80b4-631aee5b2a50" finished successfully.
billingAccountName: billingAccounts/0128F5-0C5308-B9D171
billingEnabled: true
name: projects/dept-seed-project/billingInfo
projectId: dept-seed-project
Listed 0 items.
Created service account [tfadmin-dept].
sed: 1: "../1-guardrails/provide ...": invalid command code .
tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
.....
Updated IAM policy for organization [796553858496].
bindings:
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/accesscontextmanager.policyAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/bigquery.dataEditor
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
- user:michael@clouddevops.dev
role: roles/billing.admin
- members:
- domain:clouddevops.dev
role: roles/billing.creator
- members:
- user:michael@clouddevops.dev
role: roles/billing.projectManager
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/billing.user
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/compute.networkAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/compute.xpnAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/iam.organizationRoleAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/iam.serviceAccountAdmin
- members:
- user:michael@clouddevops.dev
role: roles/iam.serviceAccountTokenCreator
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/logging.configWriter
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
- user:michael@clouddevops.dev
role: roles/orgpolicy.policyAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/pubsub.admin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
- user:michael@clouddevops.dev
role: roles/resourcemanager.folderAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
- user:michael@clouddevops.dev
role: roles/resourcemanager.organizationAdmin
- members:
- domain:clouddevops.dev
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
- user:michael@clouddevops.dev
role: roles/resourcemanager.projectCreator
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/resourcemanager.projectDeleter
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/resourcemanager.projectIamAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/resourcemanager.projectMover
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/serviceusage.serviceUsageAdmin
- members:
- serviceAccount:tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
role: roles/storage.admin
etag: BwXgZU4I_sU=
version: 1
gs://dept-seed-project-guardrails
Creating gs://dept-seed-project-guardrails/...
Replace backend.tf bucketname
sed: 1: "../1-guardrails/backend.tf": invalid command code .
Your active configuration is: [michael-clouddevops-dev]
Updated property [core/project].
sed: 1: "../1-guardrails/variabl ...": invalid command code .
sed: 1: "../1-guardrails/variabl ...": invalid command code .
sed: 1: "../1-guardrails/variabl ...": invalid command code .
sed: 1: "../1-guardrails/variabl ...": invalid command code .
sed: 1: "../1-guardrails/provide ...": invalid command code .
wrote TF SA to provider.tf and variables.tfvar along with the bucket, billing account and org id - verify them
enabling pubsub identitytoolkit cloudresourcemanager iam cloudbilling on dept-seed-project project
Operation "operations/acf.p2-6631106591-e509d9fd-481e-49e4-afe2-45ef787b84ab" finished successfully.
cloudresourcemanager.googleapis.com Cloud Resource Manager API
identitytoolkit.googleapis.com Identity Toolkit API
pubsub.googleapis.com Cloud Pub/Sub API
cloudbilling.googleapis.com Cloud Billing API
iam.googleapis.com Identity and Access Management (IAM) API
if you get an iam permission on the guardrails-aaaa project - run gcloud services enable iam.googleapis.com --project guardrails-nnnn
Status: 0
GCP seed project created project id: dept-seed-project
Terraform Service account to be used for creating GCP landing zone = tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
Terraform Backend Storage Bucket: gs://dept-seed-project-guardrails
for Macs - take out sed -i to be just sed
gs://dept-seed-project-guardrails
Creating gs://dept-seed-project-guardrails/...
ServiceException: 409 A Cloud Storage bucket named 'dept-seed-project-guardrails' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.
Replace backend.tf bucketname
terraform {
backend "gcs" {
bucket = "dept-seed-project-guardrails"
prefix = "/orgadmin/seeding/"
}
}Your active configuration is: [michael-clouddevops-dev]
No changes made to gs://dept-seed-project-guardrails/
Updated property [core/project].
audit_data_users="group@email.com"
ssc_broker_users="group@email.com"
org_id="ORG_ID"
terraform_service_account="service-account@email.com"
billing_account="0128F5-0C5308-B9D171"
billing_data_users="group@email.com"
audit_logs_table_delete_contents_on_destroy=true
log_export_storage_force_destroy=true
allowed_regions=["northamerica-northeast1", "northamerica-northeast2"]
bucket_name="guardrails-asset-bkt"
audit_data_users="group@email.com"
ssc_broker_users="group@email.com"
org_id="796553858496"
terraform_service_account="service-account@email.com"
billing_account="BILLING_ACCOUNT"
billing_data_users="group@email.com"
audit_logs_table_delete_contents_on_destroy=true
log_export_storage_force_destroy=true
allowed_regions=["northamerica-northeast1", "northamerica-northeast2"]
bucket_name="guardrails-asset-bkt"
audit_data_users="group@email.com"
ssc_broker_users="group@email.com"
org_id="ORG_ID"
terraform_service_account="tfadmin-dept@dept-seed-project.iam.gserviceaccount.com"
billing_account="BILLING_ACCOUNT"
billing_data_users="group@email.com"
audit_logs_table_delete_contents_on_destroy=true
log_export_storage_force_destroy=true
allowed_regions=["northamerica-northeast1", "northamerica-northeast2"]
bucket_name="guardrails-asset-bkt"
audit_data_users="group@email.com"
ssc_broker_users="group@email.com"
org_id="ORG_ID"
terraform_service_account="service-account@email.com"
billing_account="BILLING_ACCOUNT"
billing_data_users="group@email.com"
audit_logs_table_delete_contents_on_destroy=true
log_export_storage_force_destroy=true
allowed_regions=["northamerica-northeast1", "northamerica-northeast2"]
bucket_name="dept-guardrails-assets"
/*provider "google" {
alias = "gcp-provider"
region = var.default_region
}*/
# https://cloud.google.com/blog/topics/developers-practitioners/using-google-cloud-service-account-impersonation-your-terraform-code
provider "google" {
alias = "impersonate"
scopes = [
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/userinfo.email",
]
}
provider "google-beta" {
alias = "impersonate"
scopes = [
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/userinfo.email",
]
}
provider "google" {
access_token = data.google_service_account_access_token.default.access_token
}
provider "google-beta" {
access_token = data.google_service_account_access_token.default.access_token
}
provider "null" {
}
data "google_service_account_access_token" "default" {
provider = google.impersonate
target_service_account = local.terraform_service_account
scopes = ["userinfo-email", "cloud-platform"]
lifetime = "3600s"
}
# written from bootstrap.sh via tfadmin-dept@dept-seed-project.iam.gserviceaccount.com in form SERVICE_ACCOUNT@PROJECT.iam.gserviceaccount.com
locals { terraform_service_account = "tfadmin-dept@dept-seed-project.iam.gserviceaccount.com" }wrote TF SA to provider.tf and variables.tfvar along with the bucket, billing account and org id - verify them
enabling pubsub identitytoolkit cloudresourcemanager iam cloudbilling on dept-seed-project project
Operation "operations/acat.p2-6631106591-7afb456f-2af5-4026-a383-d983a6a73543" finished successfully.
cloudresourcemanager.googleapis.com Cloud Resource Manager API
identitytoolkit.googleapis.com Identity Toolkit API
pubsub.googleapis.com Cloud Pub/Sub API
cloudbilling.googleapis.com Cloud Billing API
iam.googleapis.com Identity and Access Management (IAM) API
if you get an iam permission on the guardrails-aaaa project - run gcloud services enable iam.googleapis.com --project guardrails-nnnn
Status: 0
GCP seed project created project id: dept-seed-project
Terraform Service account to be used for creating GCP landing zone = tfadmin-dept@dept-seed-project.iam.gserviceaccount.com
Terraform Backend Storage Bucket: gs://dept-seed-project-guardrails
Issue was on the "open in cloud shell" - trust repo - select it
Some clients are not able to authorize the shell or get tokens to run gsutil commands - for these we document installing and using the gcloud SDK and terraform locally
For those that do not see