Cloud Armor - Config Connector gives Invalid value for field 'resource.rateLimitOptions.enforceOnKeyConfigs'

[henrique-freitas]

Checklist

• [X] I did not find a related open issue.
• [X] I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• [X] If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

We started the task of managing the cloud resources we have using the config connector. The API behaves strangely when creating or updating the resource. in both cases the API rejects the call with Invalid value for field 'resource.rateLimitOptions.enforceOnKeyConfigs, Only one of enforceOnKey and enforceOnKeyConfigs can be specified

even when we create the policy manually in the Google Cloud console and export it using the command config-connector export "//compute.googleapis.com/projects/my-project-id/global/securityPolicies/my-test-policy-2" --output my-test-policy-2.yaml , and trying to apply using kubectl apply we received the same error.

In the documentation ComputeSecurityPolicy there is a optional string field called enforceOnKey In export command this field is not shown in the yaml file, we decided to put it with an empty value enforceOnKey: "" and after that it worked normally without the error above, even though it was described as optional, it was a solution we found to solve the problem for now.

Additional Diagnostic Information

Without the optional field enforceOnKey: "" set, returns :

Message: Update call failed: error applying desired state: summary: Error updating SecurityPolicy "my-project-id": googleapi: Error 400: Invalid value for field 'resource.rateLimitOptions.enforceOnKeyConfigs': ''. Only one of enforceOnKey and enforceOnKeyConfigs can be specified., invalid

Kubernetes Cluster Version

v1.27

Config Connector Version

1.111.0

Config Connector Mode

cluster mode

Log Output

Warning UpdateFailed 5m13s (x4253 over 5d23h) computesecuritypolicy-controller Update call failed: error applying desired state: summary: Error updating SecurityPolicy "my-project-id": googleapi: Error 400: Invalid value for field 'resource.rateLimitOptions.enforceOnKeyConfigs': ''. Only one of enforceOnKey and enforceOnKeyConfigs can be specified., invalid

Steps to reproduce the issue

1. Install Config connector in cluster mode
2. All the required permission is already there
3. All the prerequisite steps are done by following this doc https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall

then kubectl apply -f file.yaml -n "your-namespace"

YAML snippets

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeSecurityPolicy
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: my-project-id
  name: my-test-policy-2
spec:
  resourceID: my-test-policy-1
  rule:
  - action: allow
    description: Default rule, higher priority overrides it
    match:
      config:
        srcIpRanges:
        - '*'
      versionedExpr: SRC_IPS_V1
    priority: 2147483647
  - action: rate_based_ban
    description: Throttle requests if >500 p/1min and ban if >1000 p /2min, IO-15
    match:
      config:
        srcIpRanges:
        - '*'
      versionedExpr: SRC_IPS_V1
    preview: true
    priority: 210000
    rateLimitOptions:
      banDurationSec: 300
      banThreshold:
        count: 1000
        intervalSec: 120
      conformAction: allow
      # enforceOnKey: "" To reproduce the error, leave this line commented out
      enforceOnKeyConfigs:
      - enforceOnKeyType: XFF_IP
      - enforceOnKeyType: HTTP_PATH
      exceedAction: deny(429)
      rateLimitThreshold:
        count: 500
        intervalSec: 60
  type: CLOUD_ARMOR

[diviner524]

This resource is currently based on Terraform. It is possible the default value of enforceOnKey ("ALL") was populated back into spec thus causing the 400 error.

https://github.com/GoogleCloudPlatform/k8s-config-connector/blob/master/third_party/github.com/hashicorp/terraform-provider-google-beta/google-beta/services/compute/resource_compute_security_policy.go#L264C10-L264C10

Could you try adding cnrm.cloud.google.com/state-into-spec: absent and see if it helps?

https://cloud.google.com/config-connector/docs/concepts/ignore-unspecified-fields#skip_populating_unspecified_fields_into_spec

[jingyih]

This behavior is due to the underlying Terraform. They specifically asked users to set "enforce_on_key" to empty string when using "enforce_on_key_configs".

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_security_policy#enforce_on_key_configs

[diviner524]

@jingyih Good spot! In that case I am not sure if the state-into-spec annotation will help. We could potentially patch the Terraform provider to solve this friction.

[henrique-freitas]

@jingyih Good spot! In that case I am not sure if the state-into-spec annotation will help. We could potentially patch the Terraform provider to solve this friction.

i added the annotationcnrm.cloud.google.com/state-into-spec: absent just to test, and the error persist

[jingyih]

It is enforced by the underlying library. So please set "enforece_on_key" to empty string if you are using "enforce_on_key_configs".

The requirement imposed by the underlying library seems counterintuitive. We are currently discussing regarding potential strategies to address and mitigate this issue.