search

GoogleCloudPlatform / terraform-validator

Terraform Validator is not an officially supported Google product; it is a library for conversion of Terraform plan data to CAI Assets. If you have been using terraform-validator directly in the past, we recommend migrating to `gcloud beta terraform vet`.
https://cloud.google.com/docs/terraform/policy-validation
Apache License 2.0
438 stars 93 forks source link

spec.match.target does not work with projects #182

Closed iam2010 closed 3 years ago

iam2010 commented 3 years ago

I Have multiple folders and projects under organization, I am trying to write policy constraints for each project individually.

I'm having issue similar to #134

My config is as below

Terraform Version Terraform v0.14.4 provider registry.terraform.io/hashicorp/google v3.56.0

Terraform Validator version Build version: 2020-09-24

Policy library from Forseti Security

My Issue:

match:
    target: ["organization/<org_id>/project/<project_number>"]

Does not seem to work. I always get a No Violation Found result.

However,

match:
    target: ["organization/<org_id>/project/*"]

works. Throws proper constraint violations.

when I run terraform-validator validate --policy-path=${POLICY_PATH} ./terraform.tfplan.json --verbose=true

I get the following 

2021/02/16 08:21:01 [INFO] Authenticating using DefaultClient...
2021/02/16 08:21:01 [INFO]   -- Scopes: [https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/ndev.clouddns.readwrite https://www.googleapis.com/auth/devstorage.full_control https://www.googleapis.com/auth/userinfo.email]
2021/02/16 08:21:01 [INFO] Instantiating GCE client for path https://www.googleapis.com/compute/v1/projects/
2021/02/16 08:21:01 [INFO] Instantiating GCE Beta client for path https://www.googleapis.com/compute/beta/projects/
2021/02/16 08:21:01 [INFO] Instantiating GKE client for path https://container.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating GKE Beta client for path https://container.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud DNS client for path https://www.googleapis.com/dns/v1/projects/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud DNS Beta client for path https://www.googleapis.com/dns/v1beta2/projects/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud KMS client for path https://cloudkms.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Stackdriver Logging client for path https://logging.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Storage client for path https://www.googleapis.com/storage/v1/
2021/02/16 08:21:01 [INFO] Instantiating Google SqlAdmin client for path https://sqladmin.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Pubsub client for path https://pubsub.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Dataflow client for path https://dataflow.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud ResourceManager client for path https://cloudresourcemanager.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud ResourceManager V client for path https://cloudresourcemanager.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Runtimeconfig client for path https://runtimeconfig.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud IAM client for path https://iam.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud IAMCredentials client for path https://iamcredentials.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Service Management client for path https://servicemanagement.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Service Usage client for path https://serviceusage.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Billing client for path https://cloudbilling.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Build client for path https://cloudbuild.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud BigQuery client for path https://www.googleapis.com/bigquery/v2/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud CloudFunctions Client for path https://cloudfunctions.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud BigtableAdmin for path https://bigtableadmin.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Source Repo client for path https://sourcerepo.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Spanner client for path https://spanner.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Dataproc client for path https://dataproc.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Dataproc Beta client for path
2021/02/16 08:21:01 [INFO] Instantiating Filestore client for path https://file.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud IoT Core client for path https://cloudiot.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating App Engine client for path https://appengine.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Cloud Composer client for path https://composer.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Service Networking client for path https://servicenetworking.googleapis.com/
2021/02/16 08:21:01 [INFO] Instantiating Google Cloud Storage Transfer client for path https://storagetransfer.googleapis.com/
2021/02/16 08:21:03 [WARN] Failed to retrieve project_id for //cloudresourcemanager.googleapis.com/projects/<valid_project_id> from resource
2021/02/16 08:21:03 [WARN] Failed to retrieve project_id for //cloudresourcemanager.googleapis.com/projects/<valid_project_id> from resource
2021/02/16 08:21:03 [WARN] Failed to retrieve project_id for //cloudresourcemanager.googleapis.com/projects/<valid_project_id> from resource
2021/02/16 08:21:03 [WARN] Failed to retrieve project_id for //cloudresourcemanager.googleapis.com/projects/<valid_project_id> from resource
No violations found.

I am using the credentials of a service account to run, and the service account has the following roles at Organization level

Security Admin
Cloud Asset Viewer
morgante commented 3 years ago

Hi, can you try organization/<org_id>/project/<project-id> for the target?

iam2010 commented 3 years ago

I get the following Error using project-id

Error: validating: FCV: initializing gcv validator: unable to set up GCP Constraint Framework client: failed to add constraint &{map[apiVersion:constraints.gatekeeper.sh/v1alpha1 kind:GCPAlwaysViolatesConstraintV1 metadata:map[annotations:map[description:Testing policy, will always violate. validation.gcp.forsetisecurity.org/originalName:always_violates_all validation.gcp.forsetisecurity.org/yamlpath:/home/chandan/terraform/policy-library/policies/constraints/terraform-test/always_violates.yaml] name:always-violates-all] spec:map[constraintVersion:0.1.0 match:map[target:[organizations/<org-id>/projects/<project-id>]] parameters:map[] severity:high]]}: invalid glob in target: idx: 0: unexpected item <project-id> element 3 in organizations/<org-id>/projects/<project-id>
Usage:
  terraform-validator validate <tfplan> [flags]

Flags:
      --ancestry string      Override the ancestry location of the project when validating resources
  -h, --help                 help for validate
      --offline              Do not connect to GCP API
      --output-json          Print violations as JSON
      --policy-path string   Path to directory containing validation policies
      --project string       Provider project override (override the default project configuration assigned to the google terraform provider when validating resources)

Global Flags:
      --verbose   Log output to stderr
morgante commented 3 years ago

What version of Terraform Validator are you using?

iam2010 commented 3 years ago

Terraform Validator version Build version: 2020-09-24

morgante commented 3 years ago

Can you try downloading the most recent version?

iam2010 commented 3 years ago

Just tried with the latest build, Build version: 2021-01-21

Getting the same error.

iam2010 commented 3 years ago

Hi @morgante, is there anything I can do to help to fix this issue or find a workaround for this?

morgante commented 3 years ago

Can you try running the convert command? The generated asset should reveal what the actual asset looks like (in particular, we're interested in the ancestry path.

iam2010 commented 3 years ago

Here is the full output of convert command

[
    {
        "name": "//cloudresourcemanager.googleapis.com/projects/<project-id-1>",
        "asset_type": "cloudresourcemanager.googleapis.com/Project",
        "ancestry_path": "organization/<org-id>/folder/<folder-id-1>/folder/<folder-id-2>/project/<project-id-1>",
        "iam_policy": {
            "bindings": [
                {
                    "role": "roles/source.admin",
                    "members": [
                        "group:valid-group@googlegroups.com",
                        "user:valid-user@gmail.com"
                    ]
                }
            ]
        }
    },
    {
        "name": "//cloudresourcemanager.googleapis.com/projects/<project-id-2>",
        "asset_type": "cloudresourcemanager.googleapis.com/Project",
        "ancestry_path": "organization/<org-id>/project/<project-id-2>",
        "iam_policy": {
            "bindings": [
                {
                    "role": "roles/source.admin",
                    "members": [
                        "group:valid-group@googlegroups.com",
                        "user:valid-user@gmail.com"
                    ]
                }
            ]
        }
    },
    {
        "name": "//compute.googleapis.com/projects/<project-id-1>/global/firewalls/test-firewall-2",
        "asset_type": "compute.googleapis.com/Firewall",
        "ancestry_path": "organization/<org-id>/folder/<folder-id-1>/folder/<folder-id-2>/project/<project-id-1>",
        "resource": {
            "version": "v1",
            "discovery_document_uri": "https://www.googleapis.com/discovery/v1/apis/compute/v1/rest",
            "discovery_name": "Firewall",
            "parent": "//cloudresourcemanager.googleapis.com/projects/<project-id-1>",
            "data": {
                "allowed": [
                    {
                        "IPProtocol": "icmp"
                    },
                    {
                        "IPProtocol": "tcp",
                        "ports": [
                            "22",
                            "33"
                        ]
                    }
                ],
                "description": "testing firewall scripts",
                "direction": "INGRESS",
                "disabled": false,
                "logConfig": {
                    "enable": false
                },
                "name": "test-firewall-2",
                "network": "projects/<project-id-1>/global/networks/default",
                "priority": 1000,
                "sourceRanges": [
                    "10.0.0.0/15",
                    "192.168.1.0/24"
                ],
                "sourceServiceAccounts": [
                    "terraform-test@<project-id-2>.iam.gserviceaccount.com"
                ]
            }
        }
    },
    {
        "name": "//compute.googleapis.com/projects/<project-id-2>/global/firewalls/test-firewall",
        "asset_type": "compute.googleapis.com/Firewall",
        "ancestry_path": "organization/<org-id>/project/<project-id-2>",
        "resource": {
            "version": "v1",
            "discovery_document_uri": "https://www.googleapis.com/discovery/v1/apis/compute/v1/rest",
            "discovery_name": "Firewall",
            "parent": "//cloudresourcemanager.googleapis.com/projects/<project-id-2>",
            "data": {
                "allowed": [
                    {
                        "IPProtocol": "icmp"
                    },
                    {
                        "IPProtocol": "tcp",
                        "ports": [
                            "22",
                            "33"
                        ]
                    }
                ],
                "description": "testing firewall scripts",
                "direction": "INGRESS",
                "disabled": false,
                "logConfig": {
                    "enable": false
                },
                "name": "test-firewall",
                "network": "projects/<project-id-2>/global/networks/default",
                "priority": 1000,
                "sourceRanges": [
                    "10.0.0.0/15",
                    "192.168.1.0/24"
                ],
                "sourceServiceAccounts": [
                    "terraform-gke-test@<project-id-2>.iam.gserviceaccount.com"
                ]
            }
        }
    },
    {
        "name": "//compute.googleapis.com/projects/<project-id-2>/zones/us-central1-a/instances/test-vm-1",
        "asset_type": "compute.googleapis.com/Instance",
        "ancestry_path": "organization/<org-id>/project/<project-id-2>",
        "resource": {
            "version": "v1",
            "discovery_document_uri": "https://www.googleapis.com/discovery/v1/apis/compute/v1/rest",
            "discovery_name": "Instance",
            "parent": "//cloudresourcemanager.googleapis.com/projects/<project-id-2>",
            "data": {
                "canIpForward": false,
                "deletionProtection": false,
                "description": "Compute Instance",
                "disks": [
                    {
                        "autoDelete": true,
                        "boot": true,
                        "initializeParams": {
                            "sourceImage": "projects/debian-cloud/global/images/family/debian-9"
                        },
                        "mode": "READ_WRITE"
                    }
                ],
                "displayDevice": {
                    "enableDisplay": false
                },
                "machineType": "projects/<project-id-2>/zones/us-central1-a/machineTypes/f1-micro",
                "metadata": {},
                "name": "test-vm-1",
                "networkInterfaces": [
                    {
                        "accessConfigs": [
                            {
                                "type": "ONE_TO_ONE_NAT"
                            }
                        ],
                        "network": "projects/<project-id-2>/global/networks/default"
                    }
                ],
                "scheduling": {
                    "automaticRestart": true
                },
                "shieldedInstanceConfig": {
                    "enableIntegrityMonitoring": false,
                    "enableSecureBoot": false,
                    "enableVtpm": false
                },
                "tags": {}
            }
        }
    }
]
ralsu091 commented 3 years ago

I get the following Error using project-id

Error: validating: FCV: initializing gcv validator: unable to set up GCP Constraint Framework client: failed to add constraint &{map[apiVersion:constraints.gatekeeper.sh/v1alpha1 kind:GCPAlwaysViolatesConstraintV1 metadata:map[annotations:map[description:Testing policy, will always violate. validation.gcp.forsetisecurity.org/originalName:always_violates_all validation.gcp.forsetisecurity.org/yamlpath:/home/chandan/terraform/policy-library/policies/constraints/terraform-test/always_violates.yaml] name:always-violates-all] spec:map[constraintVersion:0.1.0 match:map[target:[organizations/<org-id>/projects/<project-id>]] parameters:map[] severity:high]]}: invalid glob in target: idx: 0: unexpected item <project-id> element 3 in organizations/<org-id>/projects/<project-id>
Usage:
  terraform-validator validate <tfplan> [flags]

Flags:
      --ancestry string      Override the ancestry location of the project when validating resources
  -h, --help                 help for validate
      --offline              Do not connect to GCP API
      --output-json          Print violations as JSON
      --policy-path string   Path to directory containing validation policies
      --project string       Provider project override (override the default project configuration assigned to the google terraform provider when validating resources)

Global Flags:
      --verbose   Log output to stderr

We are seeing the same issue.