422 - declaration failed - one or more properties must be specified

Setup Details

CIS Version : 2.18.0 Build: f5networks/k8s-bigip-ctlr:2.18.0 BIGIP Version: Big IP 16.1.4.2 Build 0.0.3 Point Release 2 (Virtual Edition) AS3 Version: 3.52.0 Agent Mode: AS3 Orchestration: K8S
Orchestration Version: 1.30.4 Pool Mode: Nodeport
Additional Setup details: Calico is used as CNI

Description

I was giving a first shot at CIS using the VirtualServer CRD and I keep on getting a 422 error with a very generic log one or more properties must be specified. I know there has been many issues regarding the 422 status code in the past but I didn't find a workaround.

Steps To Reproduce

1) Here's my CIS config (deployed using the helm chart)

    - --ingress-class=f5
    - --credentials-directory
    - /tmp/creds
    - --bigip-partition=KubF5
    - --bigip-url=https://<my-lb-lab-endpoint>
    - --custom-resource-mode=true
    - --disable-teems=true
    - --insecure=true
    - --log-as3-response=true
    - --pool-member-type=nodeport
    - --share-nodes=false

2) I use the following VirtualServer definition. Note that the ingress-nginx-controller service is of type NodePort

---
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  name: example
  namespace: my-ns
  labels:
    f5cr: "true"
spec:
  host: www.example.com
  virtualServerAddress: 192.168.94.100
  pools:
  - path: /example
    service: ingress-nginx-controller
    servicePort: 80

Expected Result

After applying the above manifest, I get the following in CIS logs

2024/09/24 15:29:14 [ERROR] [Retry][AS3] Response from BIG-IP: code: 422 --- tenant:KubF5 --- message: declaration failed

Actual Result

Objects (vs, pools, nodes, policies) should be created

Diagnostic Information

As the logs on CIS were quite scarce, I managed to retrieve the payload sent by CIS using tcpdump on the LTM. The following payload is received by AS3

{
  "$schema": "https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/main/schema/3.52.0/as3-schema-3.52.0-5.json",
  "class": "AS3",
  "declaration": {
    "KubF5": {
      "Shared": {
        "class": "Application",
        "template": "shared",
        "ingress_nginx_controller_80_my_ns_www_example_com": {
            "class": "Pool",
            "members": [
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.6"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.13"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.11"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.14"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.12"],
                "servicePort": 30000,
                "shareNodes": true
              }
            ],
            "minimumMonitors": "all"
        },
        "crd_192_168_94_100_80": {
            "class": "Service_HTTP",
          "source": "0.0.0.0/0",
          "translateServerAddress": true,
          "translateServerPort": true,
          "virtualAddresses": ["192.168.94.100"],
          "virtualPort": 80,
          "snat": "auto",
          "policyEndpoint": "/KubF5/Shared/crd_192_168_94_100_80_www_example_com_policy"
        },
        "crd_192_168_94_100_80_www_example_com_policy": {
          "class": "Endpoint_Policy",
          "rules": [
            {
              "name": "vs_www_example_com_example_ingress_nginx_controller_80_my_ns_www_example_com",
              "conditions": [
                {
                  "type": "httpHeader",
                  "name": "host",
                  "event": "request",
                  "all": {
                    "values": ["www.example.com:80", "www.example.com"],
                    "operand": "equals"
                  }
                },
                {
                  "type": "httpUri",
                  "name": "1",
                  "event": "request",
                  "index": 1,
                  "pathSegment": { "values": ["example"], "operand": "equals" }
                }
              ],
              "actions": [
                {
                  "type": "forward",
                  "event": "request",
                  "select": {
                    "pool": {
                      "use": "ingress_nginx_controller_80_my_ns_www_example_com"
                    }
                  }
                }
              ]
            }
          ],
          "strategy": "first-match"
        }
      },
      "class": "Tenant",
      "defaultRouteDomain": 0,
      "label": "KubF5"
    },
    "class": "ADC",
    "controls": { "class": "Controls", "userAgent": "CIS/v2.18.0 K8S/v1.30.4" },
    "id": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d",
    "label": "CIS Declaration",
    "remark": "Auto-generated by CIS",
    "schemaVersion": "3.52.0"
  }
}

The payload seems to be compliant with the AS3 json schema (I ran a check using a small utility), so I don't really understand why I'm getting the error. Again, using tcpdump, here's the response sent by AS3 which is a bit more complete but doesn't really point out what are the properties that are missing

{
  "results": [
    {
      "code": 422,
      "message": "declaration failed",
      "response": "one or more properties must be specified",
      "host": "localhost",
      "tenant": "KubF5",
      "runTime": 4066,
      "declarationId": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d"
    }
  ],
  "declaration": {
    "class": "ADC",
    "controls": {
      "class": "Controls",
      "userAgent": "CIS/v2.18.0 K8S/v1.30.4",
      "archiveTimestamp": "2024-09-24T15:14:17.278Z"
    },
    "id": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d",
    "label": "CIS Declaration",
    "remark": "Auto-generated by CIS",
    "schemaVersion": "3.52.0",
    "updateMode": "selective"
  },
  "code": 422
}

I also manually POSTed the payload while enabling the full AS3 trace (with "traceResponse": true in the payload). Appart from showing more info about the tmsh commands used under the hood, I didn't get useful hints about what's wrong with the payload.

Hi @tmsdce does your virtual server reference to an k8s service is duplicate, have two difference virtual server use same service ?

Hi @nansenat16 No, I have only one VirtualServer declaration in my k8s cluster. The BigIP VE I use is also almost empty as it is used for testing. As said earlier, manually POSTing the AS3 json (so bypassing the CIS completely) also results in the same error.

This could very well be a bug with AS3 itself but I want to clarify if the AS3 json generated by the CIS is indeed OK or do really have some properties missing in some way

Hi @tmsdce I run the yaml on my F5 is work no error, I only add defaultPool setting

CIS Version : 2.18.0 Build: f5networks/k8s-bigip-ctlr:2.18.0 BIGIP Version: Big IP 17.1.1.3-0.0.5 AS3 Version: 3.52.0-5 Agent Mode: AS3 Orchestration: K8S Orchestration Version: k3s-1.30.4

CIS config

--ingress-class=f5 
--credentials-directory /tmp/creds 
--bigip-partition=k3s 
--bigip-url=10.9.55.32 
--custom-resource-mode=true 
--insecure=true 
--pool-member-type=nodeport

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
spec:
  selector:
    app: nginx
  ports:
    - name: http
      nodePort: 30001
      protocol: TCP
      port: 80
      targetPort: 80
  type: NodePort
---
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  name: example
  namespace: default
  labels:
    f5cr: "true"
spec:
  host: www.example.com
  virtualServerAddress: 10.9.55.34
  defaultPool:
    reference: service
    service: ingress-nginx-controller
    serviceNamespace: default
    servicePort: 80
  pools:
  - path: /example
    service: ingress-nginx-controller
    servicePort: 80

Thanks for the suggestion @nansenat16 I just tried but with no luck, same error. Although the generated AS3 json is updated accordingly with the default pool, I'm still getting 422.

I see that you're using a more recent version of LTM (17 vs 16), maybe it could be related. I also need to point out the following :

the IP I'm trying to use (192.168.94.100) in my example is just a dummy IP. The LTM has no network configuration related to this IP or a corresponding VLAN (the same goes for the pool members IPs). As you can create virtual server with any IP address through the web UI, I'm expecting this to be irrelevant when using AS3 as well but maybe I'm mistaken here
Our LTM is configured with failover and synced with another instance, this is managed by BIG-IQ (I'm always targetting the first LTM instance in CIS config). I don't know if it is relevant.

@tmsdce Please try posting the AS3 using any Rest client

{
  "$schema": "https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/main/schema/3.52.0/as3-schema-3.52.0-5.json",
  "class": "AS3",
  "declaration": {
    "KubF5": {
      "Shared": {
        "class": "Application",
        "template": "shared",
        "ingress_nginx_controller_80_my_ns_www_example_com": {
            "class": "Pool",
            "members": [
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.6"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.13"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.11"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.14"],
                "servicePort": 30000,
                "shareNodes": true
              },
              {
                "addressDiscovery": "static",
                "serverAddresses": ["10.1.0.12"],
                "servicePort": 30000,
                "shareNodes": true
              }
            ],
            "minimumMonitors": "all"
        },
        "crd_192_168_94_100_80": {
            "class": "Service_HTTP",
          "source": "0.0.0.0/0",
          "translateServerAddress": true,
          "translateServerPort": true,
          "virtualAddresses": ["192.168.94.100"],
          "virtualPort": 80,
          "snat": "auto",
          "policyEndpoint": "/KubF5/Shared/crd_192_168_94_100_80_www_example_com_policy"
        },
        "crd_192_168_94_100_80_www_example_com_policy": {
          "class": "Endpoint_Policy",
          "rules": [
            {
              "name": "vs_www_example_com_example_ingress_nginx_controller_80_my_ns_www_example_com",
              "conditions": [
                {
                  "type": "httpHeader",
                  "name": "host",
                  "event": "request",
                  "all": {
                    "values": ["www.example.com:80", "www.example.com"],
                    "operand": "equals"
                  }
                },
                {
                  "type": "httpUri",
                  "name": "1",
                  "event": "request",
                  "index": 1,
                  "pathSegment": { "values": ["example"], "operand": "equals" }
                }
              ],
              "actions": [
                {
                  "type": "forward",
                  "event": "request",
                  "select": {
                    "pool": {
                      "use": "ingress_nginx_controller_80_my_ns_www_example_com"
                    }
                  }
                }
              ]
            }
          ],
          "strategy": "first-match"
        }
      },
      "class": "Tenant",
      "defaultRouteDomain": 0,
      "label": "KubF5"
    },
    "class": "ADC",
    "controls": { "class": "Controls", "userAgent": "CIS/v2.18.0 K8S/v1.30.4" },
    "id": "urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d",
    "label": "CIS Declaration",
    "remark": "Auto-generated by CIS",
    "schemaVersion": "3.52.0"
  }
}

Configure CIS with AS3DEBUG and share exact AS3 declaration posted by CIS. Share the info to automation_toolchain_pm at f5 dot com

F5Networks / k8s-bigip-ctlr