search

terraform-google-modules / terraform-google-cloud-dns

Creates and manages Cloud DNS public or private zones and their records
https://registry.terraform.io/modules/terraform-google-modules/cloud-dns/google
Apache License 2.0
55 stars 80 forks source link

Fix integration tests #12

Closed bharathkkb closed 1 year ago

bharathkkb commented 4 years ago

Integration tests seem to be failing possible due to not setting network_self_links in private-zone

wilsonfv commented 3 years ago

this is the error i have seen on my local when run integration test 

$make docker_test_prepare
docker run --rm -it \
        -e SERVICE_ACCOUNT_JSON \
        -e TF_VAR_org_id \
        -e TF_VAR_folder_id \
        -e TF_VAR_billing_account \
        -v /Users/junxingmo/dev/github/terraform-google-modules/terraform-google-cloud-dns:/workspace \
        gcr.io/cloud-foundation-cicd/cft/developer-tools:0.1.0 \
        /usr/local/bin/execute_with_credentials.sh prepare_environment
Updated property [core/pass_credentials_to_gsutil].
Updated property [core/pass_credentials_to_gsutil].
Initializing modules...
Downloading terraform-google-modules/project-factory/google 3.3.1 for project...
- project in .terraform/modules/project
- project.gsuite_group in .terraform/modules/project/modules/gsuite_group
- project.project-factory in .terraform/modules/project/modules/core_project_factory

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 3.0.0...
- Downloading plugin for provider "google" (hashicorp/google) 2.13.0...
- Downloading plugin for provider "google-beta" (hashicorp/google-beta) 2.13.0...
- Downloading plugin for provider "null" (hashicorp/null) 3.0.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.null: version = "~> 3.0"
* provider.random: version = "~> 3.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
module.project.module.gsuite_group.data.google_organization.org[0]: Refreshing state...
module.project.module.project-factory.random_id.random_project_id_suffix: Creating...
module.project.module.project-factory.null_resource.preconditions: Creating...
module.project.module.project-factory.null_resource.shared_vpc_subnet_invalid_name[0]: Creating...
module.project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=Cgw]
module.project.module.project-factory.null_resource.preconditions: Provisioning with 'local-exec'...
module.project.module.project-factory.null_resource.shared_vpc_subnet_invalid_name[0]: Creation complete after 0s [id=4674584172891988096]
module.project.module.project-factory.null_resource.preconditions (local-exec): Executing: ["/bin/sh" "-c" ".terraform/modules/project/modules/core_project_factory/scripts/preconditions.sh \\\n    --credentials_path '' \\\n    --impersonate_service_account '' \\\n    --billing_account '014D0A-D8E7B4-975AC3' \\\n    --org_id '387121157408' \\\n    --folder_id '948946194298' \\\n    --shared_vpc ''\n"]
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: google-api-python-client~=1.7 in /usr/lib/python3.7/site-packages (from -r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 1)) (1.7.11)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: google-auth~=1.6 in /usr/lib/python3.7/site-packages (from -r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 2)) (1.6.3)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: httplib2<1dev,>=0.9.2 in /usr/lib/python3.7/site-packages (from google-api-python-client~=1.7->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 1)) (0.13.1)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/lib/python3.7/site-packages (from google-api-python-client~=1.7->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 1)) (0.0.3)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: six<2dev,>=1.6.1 in /usr/lib/python3.7/site-packages (from google-api-python-client~=1.7->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 1)) (1.12.0)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/lib/python3.7/site-packages (from google-api-python-client~=1.7->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 1)) (3.0.0)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: rsa>=3.1.4 in /usr/lib/python3.7/site-packages (from google-auth~=1.6->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 2)) (4.0)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: cachetools>=2.0.0 in /usr/lib/python3.7/site-packages (from google-auth~=1.6->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 2)) (3.1.1)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3.7/site-packages (from google-auth~=1.6->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 2)) (0.2.6)
module.project.module.project-factory.null_resource.preconditions (local-exec): Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3.7/site-packages (from rsa>=3.1.4->google-auth~=1.6->-r .terraform/modules/project/modules/core_project_factory/scripts/preconditions/requirements.txt (line 2)) (0.4.6)
module.project.module.project-factory.null_resource.preconditions (local-exec): You are using pip version 19.0.3, however version 20.3b1 is available.
module.project.module.project-factory.null_resource.preconditions (local-exec): You should consider upgrading via the 'pip install --upgrade pip' command.
module.project.module.project-factory.null_resource.preconditions: Creation complete after 4s [id=7528389536151480954]
module.project.module.project-factory.google_project.main: Creating...
module.project.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [20s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [30s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [40s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [50s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m0s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m10s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m20s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m40s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [1m50s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m0s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m10s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m20s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m30s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m40s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [2m50s elapsed]
module.project.module.project-factory.google_project.main: Still creating... [3m0s elapsed]
module.project.module.project-factory.google_project.main: Creation complete after 3m2s [id=ci-cloud-dns-0a0c]
module.project.module.project-factory.data.null_data_source.default_service_account: Refreshing state...
module.project.module.project-factory.google_service_account.default_service_account: Creating...
module.project.module.project-factory.google_project_service.project_services[0]: Creating...
module.project.module.project-factory.google_service_account.default_service_account: Creation complete after 3s [id=projects/ci-cloud-dns-0a0c/serviceAccounts/project-service-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com]
module.project.module.project-factory.google_project_service.project_services[0]: Still creating... [10s elapsed]
module.project.module.project-factory.google_project_service.project_services[0]: Creation complete after 18s [id=ci-cloud-dns-0a0c/dns.googleapis.com]
google_service_account.int_test: Creating...
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0]: Creating...
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0]: Provisioning with 'local-exec'...
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0] (local-exec): Executing: ["/bin/sh" "-c" ".terraform/modules/project/modules/core_project_factory/scripts/modify-service-account.sh \\\n  --project_id='ci-cloud-dns-0a0c' \\\n  --sa_id='131716288274-compute@developer.gserviceaccount.com' \\\n  --credentials_path='' \\\n  --impersonate-service-account='' \\\n  --action='delete'\n"]
google_service_account.int_test: Creation complete after 3s [id=projects/ci-cloud-dns-0a0c/serviceAccounts/ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com]
google_project_iam_member.int_test[0]: Creating...
google_service_account_key.int_test: Creating...
google_project_iam_member.int_test[1]: Creating...
google_service_account_key.int_test: Creation complete after 2s [id=projects/ci-cloud-dns-0a0c/serviceAccounts/ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com/keys/766205a24dc9f141ab6d6a3f1daf18208a6573e4]
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0] (local-exec): Deleting service account 131716288274-compute@developer.gserviceaccount.com in project ci-cloud-dns-0a0c
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0] (local-exec): deleted service account [131716288274-compute@developer.gserviceaccount.com]
module.project.module.project-factory.null_resource.delete_default_compute_service_account[0]: Creation complete after 8s [id=9042955024865793875]
google_project_iam_member.int_test[0]: Still creating... [10s elapsed]
google_project_iam_member.int_test[1]: Still creating... [10s elapsed]

Error: Batch "iam-project-ci-cloud-dns-0a0c modifyIamPolicy" for request "Create IAM Members roles/dns.admin serviceAccount:ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com for \"project \\\"ci-cloud-dns-0a0c\\\"\"" returned error: Error applying IAM policy for project "ci-cloud-dns-0a0c": Error setting IAM policy for project "ci-cloud-dns-0a0c": googleapi: Error 400: Policy members must be of the form "<type>:<value>"., badRequest

  on iam.tf line 30, in resource "google_project_iam_member" "int_test":
  30: resource "google_project_iam_member" "int_test" {

Error: Batch "iam-project-ci-cloud-dns-0a0c modifyIamPolicy" for request "Create IAM Members roles/owner serviceAccount:ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com for \"project \\\"ci-cloud-dns-0a0c\\\"\"" returned error: Error applying IAM policy for project "ci-cloud-dns-0a0c": Error setting IAM policy for project "ci-cloud-dns-0a0c": googleapi: Error 400: Policy members must be of the form "<type>:<value>"., badRequest

  on iam.tf line 30, in resource "google_project_iam_member" "int_test":
  30: resource "google_project_iam_member" "int_test" {

make: *** [docker_test_prepare] Error 1

when i run docker exec into the container and turn on TF_LOG=DEBUG, i notice it's the terraform module resource google_project_iam_member does not work with the returned IAM policy json which contains "deleted:serviceaccount"

just after make docker_test_prepare is run, a gcp project is created and the google-project-module will delete the default compute service account and this service account will be marked as deleted service account in IAM policy json response

When the terraform module resource google_project_iam_member uses the returned policy json and submit a http pull request to gcp to add IAM role binding, gcp will return the error 

Batch "iam-project-ci-cloud-dns-0a0c modifyIamPolicy" for request "Create IAM Members roles/owner serviceAccount:ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com for \"project \\\"ci-cloud-dns-0a0c\\\"\"" returned error: Error applying IAM policy for project "ci-cloud-dns-0a0c": Error setting IAM policy for project "ci-cloud-dns-0a0c": googleapi: Error 400: Policy members must be of the form "<type>:<value>"., badRequest

A IAM policy json look like this 

{
  "policy": {
    "bindings": [
      {
        "members": [
          "serviceAccount:service-131716288274@compute-system.iam.gserviceaccount.com"
        ],
        "role": "roles/compute.serviceAgent"
      },
      {
        "members": [
          "serviceAccount:131716288274@cloudservices.gserviceaccount.com",
          "deleted:serviceaccount:131716288274-compute@developer.gserviceaccount.com?uid=115988991780471633912"
        ],
        "role": "roles/editor"
      },
      {
        "members": [
          "serviceAccount:ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com",
          "serviceAccount:terraform@gke-eu-1.iam.gserviceaccount.com"
        ],
        "role": "roles/owner"
      },
      {
        "members": [
          "serviceAccount:ci-account@ci-cloud-dns-0a0c.iam.gserviceaccount.com"
        ],
        "role": "roles/dns.admin"
      }
    ],
    "etag": "BwWzlGgY8TA=",
    "version": 1
  },
  "updateMask": "bindings,etag,auditConfigs"
}

I dont have a quick idea what a better fix solution would be the terraform native module resource definitely is causing the problem, it should ignore the "deleted service account" however it may require quite an effort to fix terraform so my quickest idea is to run some gcloud SDK script to add the IAM role rather than using the terraform project iam resource

bharathkkb commented 3 years ago

Hi @wilsonfv Thanks for working on this. I do see that an old version of project factory is being used, perhaps you could try upgrading to a newer version to see if that resolves the issue. https://github.com/terraform-google-modules/terraform-google-cloud-dns/blob/70ee8ee82391b836f6b36b61b29dc0069d454435/test/setup/main.tf#L17

module "project" {
  source  = "terraform-google-modules/project-factory/google"
  version = "~> 9.0"

  name              = "ci-cloud-dns"
  random_project_id = "true"
  org_id            = var.org_id
  folder_id         = var.folder_id
  billing_account   = var.billing_account
  skip_gcloud_download = true

  activate_apis = [
    "dns.googleapis.com"
  ]
}

You may also need to bump the provider version to the latest 3.x which also has some error handling for the deleted:serviceaccount:foo SAs.

Finally you can also upgrade to the latest version of our developer tools image here.

- DOCKER_TAG_VERSION_DEVELOPER_TOOLS := 0.1.0
+ DOCKER_TAG_VERSION_DEVELOPER_TOOLS := 0.12

https://github.com/terraform-google-modules/terraform-google-cloud-dns/blob/70ee8ee82391b836f6b36b61b29dc0069d454435/Makefile#L21

apeabody commented 1 year ago

This appears to have already been resolved.