Integration tests failing...

Items found while testing PR #92.

I have tested from master, with the same symptoms and failures. I can roll in changes for item 1 to my PR if that's appropriate, but have not solved item 2. Also not sure / wondering if CI run failures on the same PR are same / similar to item 2.

Running 'make docker_test_prepare', failure due to bug in provider 2.13.0 for iam role member resource. Updated provider versions in versions.tf (in test/setup dir) to overcome this issue. Also updated the project factory module version, way back at ~>3.0, also due to a bug-relate failed deployment. The deployment is successful with these two updates:

kpeder@kpeder-develop:~/projects/terraform-google-bigquery$ git diff
diff --git a/test/setup/main.tf b/test/setup/main.tf
index eab6c3c..4bcd739 100644
--- a/test/setup/main.tf
+++ b/test/setup/main.tf
@@ -16,7 +16,7 @@

 module "project" {
   source  = "terraform-google-modules/project-factory/google"
-  version = "~> 3.0"
+  version = "~> 9.2.0"

   name              = "ci-bigquery"
   random_project_id = "true"
diff --git a/test/setup/versions.tf b/test/setup/versions.tf
index efbd8ea..a3ffc24 100644
--- a/test/setup/versions.tf
+++ b/test/setup/versions.tf
@@ -19,9 +19,9 @@ terraform {
 }

 provider "google" {
-  version = "~> 2.13.0"
+  version = "~> 3.41.0"
 }

 provider "google-beta" {
-  version = "~> 2.13.0"
+  version = "~> 3.41.0"
 }

[Stuck on this issue]: Running 'make docker_test_integration' returns an empty value for the project_id and executes no tests. I can't find any issue in the inspec.yaml or the controls/big_query.rb in the test/integration/full/ dir... the syntax appears to be correct.

I am pasting the test run here in the hope that it can be resolved by a smart person who knows more about kitchen and ruby than I.

kpeder@kpeder-develop:~/projects/terraform-google-bigquery$ make docker_test_integration 
docker run --rm -it \
    -e SERVICE_ACCOUNT_JSON \
    -v /home/kpeder/projects/terraform-google-bigquery:/workspace \
    gcr.io/cloud-foundation-cicd/cft/developer-tools:0 \
    /usr/local/bin/test_integration.sh
Activated service account credentials for: [terraform-edp@edp-framework.iam.gserviceaccount.com]
Automatically setting inputs from outputs of test/setup
Activated service account credentials for: [ci-account@ci-bigquery-76f4.iam.gserviceaccount.com]
-----> Starting Test Kitchen (v2.7.2)
-----> Creating <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.13.5
       + provider registry.terraform.io/hashicorp/external v2.0.0
       + provider registry.terraform.io/hashicorp/google v3.47.0
       + provider registry.terraform.io/hashicorp/null v3.0.0
       + provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Upgrading modules...
       - example in ../../../examples/multiple_tables
       - example.add_udfs in ../../../modules/udf
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_check_protocol...
       - example.add_udfs.bq_check_protocol in .terraform/modules/example.add_udfs.bq_check_protocol
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_csv_to_struct...
       - example.add_udfs.bq_csv_to_struct in .terraform/modules/example.add_udfs.bq_csv_to_struct
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_find_in_set...
       - example.add_udfs.bq_find_in_set in .terraform/modules/example.add_udfs.bq_find_in_set
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_parse_url...
       - example.add_udfs.bq_parse_url in .terraform/modules/example.add_udfs.bq_parse_url
       - example.bigquery in ../../..

       Initializing the backend...

       Initializing provider plugins...
       - Finding latest version of hashicorp/external...
       - Finding hashicorp/google versions matching "~> 3.0"...
       - Finding latest version of hashicorp/null...
       - Finding latest version of hashicorp/random...
       - Installing hashicorp/google v3.47.0...
       - Installed hashicorp/google v3.47.0 (signed by HashiCorp)
       - Installing hashicorp/null v3.0.0...
       - Installed hashicorp/null v3.0.0 (signed by HashiCorp)
       - Installing hashicorp/random v3.0.0...
       - Installed hashicorp/random v3.0.0 (signed by HashiCorp)
       - Installing hashicorp/external v2.0.0...
       - Installed hashicorp/external v2.0.0 (signed by HashiCorp)

       The following providers do not have any version constraints in configuration,
       so the latest version was installed.

       To prevent automatic upgrades to new major versions that may contain breaking
       changes, we recommend adding version constraints in a required_providers block
       in your configuration, with the constraint strings suggested below.

       * hashicorp/external: version = "~> 2.0.0"
       * hashicorp/null: version = "~> 3.0.0"
       * hashicorp/random: version = "~> 3.0.0"

       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-full-local Terraform workspace...
       Created and switched to workspace "kitchen-terraform-full-local"!

       You're now on a new, empty workspace. Workspaces isolate their state,
       so if you run "terraform plan" Terraform will not see any existing state
       for this configuration.
$$$$$$ Finished creating the kitchen-terraform-full-local Terraform workspace.
       Finished creating <full-local> (0m24.41s).
-----> Test Kitchen is finished. (0m27.69s)
-----> Starting Test Kitchen (v2.7.2)
-----> Converging <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.13.5
       + provider registry.terraform.io/hashicorp/external v2.0.0
       + provider registry.terraform.io/hashicorp/google v3.47.0
       + provider registry.terraform.io/hashicorp/null v3.0.0
       + provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Selecting the kitchen-terraform-full-local Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-full-local Terraform workspace.
$$$$$$ Downloading the modules needed for the Terraform configuration...
       - example in ../../../examples/multiple_tables
       - example.add_udfs in ../../../modules/udf
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_check_protocol...
       - example.add_udfs.bq_check_protocol in .terraform/modules/example.add_udfs.bq_check_protocol
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_csv_to_struct...
       - example.add_udfs.bq_csv_to_struct in .terraform/modules/example.add_udfs.bq_csv_to_struct
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_find_in_set...
       - example.add_udfs.bq_find_in_set in .terraform/modules/example.add_udfs.bq_find_in_set
       Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_parse_url...
       - example.add_udfs.bq_parse_url in .terraform/modules/example.add_udfs.bq_parse_url
       - example.bigquery in ../../..
$$$$$$ Finished downloading the modules needed for the Terraform configuration.
$$$$$$ Validating the Terraform configuration files...
       Success! The configuration is valid.

$$$$$$ Finished validating the Terraform configuration files.
$$$$$$ Building the infrastructure based on the Terraform configuration...
       module.example.module.bigquery.google_bigquery_dataset.main: Creating...
       module.example.module.bigquery.google_bigquery_dataset.main: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo]
       module.example.module.bigquery.google_bigquery_table.main["bar"]: Creating...
       module.example.module.bigquery.google_bigquery_table.main["foo"]: Creating...
       module.example.module.bigquery.google_bigquery_table.main["foo"]: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
       module.example.module.bigquery.google_bigquery_table.main["bar"]: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]

       Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

       Outputs:

       bigquery_dataset = {
         "access" = [
           {
             "domain" = ""
             "group_by_email" = ""
             "role" = "OWNER"
             "special_group" = "projectOwners"
             "user_by_email" = ""
             "view" = []
           },
         ]
         "creation_time" = 1605045037237
         "dataset_id" = "foo"
         "default_encryption_configuration" = []
         "default_partition_expiration_ms" = 0
         "default_table_expiration_ms" = 3600000
         "delete_contents_on_destroy" = false
         "description" = "some description"
         "etag" = "pQEv+/zjV0cJk8J36Qyx0g=="
         "friendly_name" = "foo"
         "id" = "projects/ci-bigquery-76f4/datasets/foo"
         "labels" = {
           "billable" = "true"
           "env" = "dev"
           "owner" = "janesmith"
         }
         "last_modified_time" = 1605045037237
         "location" = "US"
         "project" = "ci-bigquery-76f4"
         "self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo"
       }
       bigquery_tables = {
         "bar" = {
           "clustering" = []
           "creation_time" = 1605045037864
           "dataset_id" = "foo"
           "description" = ""
           "encryption_configuration" = []
           "etag" = "V8yrtCB+HSQbTUvqgA5Y8g=="
           "expiration_time" = 2524604400000
           "external_data_configuration" = []
           "friendly_name" = "bar"
           "id" = "projects/ci-bigquery-76f4/datasets/foo/tables/bar"
           "labels" = {
             "billable" = "true"
             "env" = "devops"
             "owner" = "joedoe"
           }
           "last_modified_time" = 1605045037913
           "location" = "US"
           "materialized_view" = []
           "num_bytes" = 0
           "num_long_term_bytes" = 0
           "num_rows" = 0
           "project" = "ci-bigquery-76f4"
           "range_partitioning" = []
           "schema" = "[{\"description\":\"Full visitor ID\",\"mode\":\"NULLABLE\",\"name\":\"fullVisitorId\",\"type\":\"STRING\"},{\"description\":\"Visit number\",\"mode\":\"NULLABLE\",\"name\":\"visitNumber\",\"type\":\"INTEGER\"},{\"description\":\"Visit ID\",\"mode\":\"NULLABLE\",\"name\":\"visitId\",\"type\":\"INTEGER\"},{\"description\":\"Visit Start Time\",\"mode\":\"NULLABLE\",\"name\":\"visitStartTime\",\"type\":\"INTEGER\"},{\"description\":\"Full Date of Visit\",\"mode\":\"NULLABLE\",\"name\":\"fullDate\",\"type\":\"DATE\"}]"
           "self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo/tables/bar"
           "table_id" = "bar"
           "time_partitioning" = []
           "type" = "TABLE"
           "view" = []
         }
         "foo" = {
           "clustering" = [
             "fullVisitorId",
             "visitId",
           ]
           "creation_time" = 1605045037974
           "dataset_id" = "foo"
           "description" = ""
           "encryption_configuration" = []
           "etag" = "Qx67/A84BaTpDXah2eaO5A=="
           "expiration_time" = 1605048637974
           "external_data_configuration" = []
           "friendly_name" = "foo"
           "id" = "projects/ci-bigquery-76f4/datasets/foo/tables/foo"
           "labels" = {
             "billable" = "true"
             "env" = "dev"
             "owner" = "joedoe"
           }
           "last_modified_time" = 1605045038028
           "location" = "US"
           "materialized_view" = []
           "num_bytes" = 0
           "num_long_term_bytes" = 0
           "num_rows" = 0
           "project" = "ci-bigquery-76f4"
           "range_partitioning" = []
           "schema" = "[{\"description\":\"Full visitor ID\",\"mode\":\"NULLABLE\",\"name\":\"fullVisitorId\",\"type\":\"STRING\"},{\"description\":\"Visit number\",\"mode\":\"NULLABLE\",\"name\":\"visitNumber\",\"type\":\"INTEGER\"},{\"description\":\"Visit ID\",\"mode\":\"NULLABLE\",\"name\":\"visitId\",\"type\":\"INTEGER\"},{\"description\":\"Visit Start Time\",\"mode\":\"NULLABLE\",\"name\":\"visitStartTime\",\"type\":\"INTEGER\"},{\"description\":\"Full Date of Visit\",\"mode\":\"NULLABLE\",\"name\":\"fullDate\",\"type\":\"DATE\"}]"
           "self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo/tables/foo"
           "table_id" = "foo"
           "time_partitioning" = [
             {
        "expiration_ms" = 0
        "field" = ""
        "require_partition_filter" = false
        "type" = "DAY"
             },
           ]
           "type" = "TABLE"
           "view" = []
         }
       }
$$$$$$ Finished building the infrastructure based on the Terraform configuration.
$$$$$$ Reading the output variables from the Terraform state...
$$$$$$ Finished reading the output variables from the Terraform state.
$$$$$$ Parsing the Terraform output variables as JSON...
$$$$$$ Finished parsing the Terraform output variables as JSON.
$$$$$$ Writing the output variables to the Kitchen instance state...
$$$$$$ Finished writing the output varibales to the Kitchen instance state.
$$$$$$ Writing the input variables to the Kitchen instance state...
$$$$$$ Finished writing the input variables to the Kitchen instance state.
       Finished converging <full-local> (0m13.48s).
-----> Test Kitchen is finished. (0m16.68s)
-----> Starting Test Kitchen (v2.7.2)
-----> Setting up <full-local>...
       Finished setting up <full-local> (0m0.00s).
-----> Verifying <full-local>...
$$$$$$ Reading the Terraform input variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform input variables from the Kitchen instance state.
$$$$$$ Reading the Terraform output variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform output varibales from the Kitchen instance state.
$$$$$$ Verifying the systems...
$$$$$$ Verifying the 'system' system...
Failed to load profile local: Failed to load source for controls/big_query.rb: Bad response: {
  "error": {
    "code": 400,
    "message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
    "errors": [
      {
        "message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Profile:         local
Version:         0.1.0
Failure Message: Failed to load source for controls/big_query.rb: Bad response: {
  "error": {
    "code": 400,
    "message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
    "errors": [
      {
        "message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
        "domain": "global",
        "reason": "invalid"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Target:          gcp://ci-account@ci-bigquery-76f4.iam.gserviceaccount.com

     No tests executed.

Profile: Google Cloud Platform Resource Pack (inspec-gcp)
Version: 1.8.1
Target:  gcp://ci-account@ci-bigquery-76f4.iam.gserviceaccount.com

     No tests executed.

Test Summary: 0 successful, 0 failures, 0 skipped
>>>>>> Verifying the 'system' system failed:
    Running InSpec failed:
        Running InSpec failed due to a non-zero exit code of 1.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Verify failed on instance <full-local>.  Please see .kitchen/logs/full-local.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

-----> Starting Test Kitchen (v2.7.2)
-----> Destroying <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
       Terraform v0.13.5
       + provider registry.terraform.io/hashicorp/external v2.0.0
       + provider registry.terraform.io/hashicorp/google v3.47.0
       + provider registry.terraform.io/hashicorp/null v3.0.0
       + provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
       Initializing modules...

       Initializing the backend...

       Initializing provider plugins...
       - Using previously-installed hashicorp/google v3.47.0
       - Using previously-installed hashicorp/null v3.0.0
       - Using previously-installed hashicorp/random v3.0.0
       - Using previously-installed hashicorp/external v2.0.0

       The following providers do not have any version constraints in configuration,
       so the latest version was installed.

       To prevent automatic upgrades to new major versions that may contain breaking
       changes, we recommend adding version constraints in a required_providers block
       in your configuration, with the constraint strings suggested below.

       * hashicorp/external: version = "~> 2.0.0"
       * hashicorp/null: version = "~> 3.0.0"
       * hashicorp/random: version = "~> 3.0.0"

       Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Selecting the kitchen-terraform-full-local Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-full-local Terraform workspace.
$$$$$$ Destroying the Terraform-managed infrastructure...
       module.example.module.bigquery.google_bigquery_dataset.main: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo]
       module.example.module.bigquery.google_bigquery_table.main["foo"]: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
       module.example.module.bigquery.google_bigquery_table.main["bar"]: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]
       module.example.module.bigquery.google_bigquery_table.main["bar"]: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]
       module.example.module.bigquery.google_bigquery_table.main["foo"]: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
       module.example.module.bigquery.google_bigquery_table.main["foo"]: Destruction complete after 0s
       module.example.module.bigquery.google_bigquery_table.main["bar"]: Destruction complete after 0s
       module.example.module.bigquery.google_bigquery_dataset.main: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo]
       module.example.module.bigquery.google_bigquery_dataset.main: Destruction complete after 1s

       Destroy complete! Resources: 3 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
       Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-full-local Terraform workspace...
       Deleted workspace "kitchen-terraform-full-local"!
$$$$$$ Finished deleting the kitchen-terraform-full-local Terraform workspace.
       Finished destroying <full-local> (0m12.25s).
-----> Test Kitchen is finished. (0m15.48s)
Makefile:61: recipe for target 'docker_test_integration' failed
make: *** [docker_test_integration] Error 20

terraform-google-modules / terraform-google-bigquery

Integration tests failing... #93