"should have required property 'servicePort'" error creating a TransportServer

brunocascio commented 1 year ago

Setup Details

CIS Version : 2.13.0 Build: f5networks/k8s-bigip-ctlr:2.13.0
BIGIP Version: Big IP 15.1.6 AS3 Version: 3.33
Agent Mode: AS3
Orchestration: K8S Orchestration Version: 1.20.12 Pool Mode: Nodeport
Additional Setup details: Antrea / Containerd

Description

Creating a TransportServer for rabbitmq throws a silent (debug) error

2023/06/05 17:36:33 [ERROR] [AS3] Raw response from Big-IP: map[code:422 declarationFullId: errors:[/dev-nonprod-k8s-cis/Shared/rabbitmq_cluster_5672_rabbitmq/members/0: should have required property 'servicePort'] message:declaration is invalid] {"$schema":"https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/master/schema/3.33.0/as3-schema-3.33.0-4.json","class":"AS3","declaration":{"class":"ADC","controls":{"class":"Controls","userAgent":"CIS/v2.13.0 K8S/v1.20.12+vmware.1"},"dev-nonprod-k8s-cis":{"Shared":{"class":"Application","crd_10_x_x_x_5672":{"class":"Service_TCP","virtualAddresses":["10.x.x.x"],"virtualPort":5672,"snat":"auto","pool":"rabbitmq_cluster_5672_rabbitmq","profileL4":"basic"},"rabbitmq_cluster_5672_rabbitmq":{"class":"Pool","members":[{"addressDiscovery":"static","serverAddresses":["172.17.68.51"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.53"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.52"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.62"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.59"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.34"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.39"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.32"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.38"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.57"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.54"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.40"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.41"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.18"],"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.49"],"shareNodes":true}],"monitors":[{"use":"/dev-nonprod-k8s-cis/Shared/rabbitmq_cluster_rabbitmq_tcp_5672"}]},"rabbitmq_cluster_rabbitmq_tcp_5672":{"class":"Monitor","interval":10,"monitorType":"tcp","targetAddress":"","timeout":10,"adaptive":false,"receive":"","send":""},"template":"shared"},"class":"Tenant","defaultRouteDomain":0},"id":"urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d","label":"CIS Declaration","remark":"Auto-generated by CIS","schemaVersion":"3.33.0"}}

Steps To Reproduce

apiVersion: cis.f5.com/v1
kind: TransportServer
metadata:
  name: rabbitmq-ts
  namespace: f5-nonprod
  labels:
    f5cr: "true"
spec:
  virtualServerAddress: "10.x.y.z"
  virtualServerPort: 5672 
  type: tcp
  mode: standard
  snat: auto
  pool:
    service: rabbitmq-cluster
    servicePort: 5672
    monitor:
      type: tcp
      interval: 10
      timeout: 10

2) Open f5 CIS logs

Expected Result

controller should configure bigip with the transport server configuration

Actual Result

Error about schema properties

Observations (if any)

I'm using the same VIP also for an ingress resource, don't know if it could generate problems but would be great to use the same VIP

mdditt2000 commented 1 year ago

@brunocascio message:declaration is invalid. Was this working before 2.13? Or never worked?

mdditt2000 commented 1 year ago

AS3 Json https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/as3.json

AS3 response https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/response.json

Humm why is the "servicePort": missing?

brunocascio commented 1 year ago

@brunocascio message:declaration is invalid. Was this working before 2.13? Or never worked?

This is my first try with TransportServer, so don't know if it works or not with previous versions unfortunately

brunocascio commented 1 year ago

AS3 Json https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/as3.json

AS3 response https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/response.json

Humm why is the "servicePort": missing?

Could it be related to the shared-node: true flag somehow?

mdditt2000 commented 1 year ago

This works

https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/as3-working.json

CIS is not adding the service port. Weird. I need to look into that.

Maybe its Shared Node. Are you going to share Node IP on the BIG-IP?

brunocascio commented 1 year ago

This works

https://github.com/mdditt2000/openshift-4-11/blob/main/GitHub/2914/as3-working.json

CIS is not adding the service port. Weird. I need to look into that.

Maybe its Shared Node. Are you going to share Node IP on the BIG-IP?

Yep, we are sharing nodes in this setup.

Thanks for looking into it! Let me know if I can help

mdditt2000 commented 1 year ago

@brunocascio please can you share the service definition. Please send mail to automation_toolchain_pm@f5.com

brunocascio commented 1 year ago

@brunocascio please can you share the service definition. Please send mail to automation_toolchain_pm@f5.com

When I was copying the service I figured out I didn't set the NodePort type 😆 but now I configured it with NodePort and I get a different error:

2023/06/06 16:21:45 [ERROR] [AS3] Raw response from Big-IP: map[code:422 kind::resterrorresponse message:request failed with null exception referer:172.17.68.63 restOperationId:8.40709713e+08] {"$schema":"https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/master/schema/3.33.0/as3-schema-3.33.0-4.json","class":"AS3","declaration":{"class":"ADC","controls":{"class":"Controls","userAgent":"CIS/v2.13.0 K8S/v1.20.12+vmware.1"},"dev-nonprod-k8s-cis":{"Shared":{"class":"Application","crd_10_12_204_254_5672":{"class":"Service_TCP","virtualAddresses":["10.12.204.254"],"virtualPort":5672,"snat":"auto","pool":"rabbitmq_cluster_nodeport_5672_rabbitmq","profileL4":"basic"},"rabbitmq_cluster_nodeport_5672_rabbitmq":{"class":"Pool","members":[{"addressDiscovery":"static","serverAddresses":["172.17.68.51"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.53"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.52"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.62"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.59"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.34"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.39"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.32"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.38"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.57"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.54"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.40"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.41"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.18"],"servicePort":32680,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.49"],"servicePort":32680,"shareNodes":true}],"monitors":[{"use":"/dev-nonprod-k8s-cis/Shared/rabbitmq_cluster_nodeport_rabbitmq_tcp_5672"}]},"rabbitmq_cluster_nodeport_rabbitmq_tcp_5672":{"class":"Monitor","interval":10,"monitorType":"tcp","targetAddress":"","timeout":10,"adaptive":false,"receive":"","send":""},"template":"shared"},"class":"Tenant","defaultRouteDomain":0},"id":"urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d","label":"CIS Declaration","remark":"Auto-generated by CIS","schemaVersion":"3.33.0"}}

Sure!

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-nodeport
spec:
  type: NodePort
  ports:
    - appProtocol: amqp
      name: amqp
      port: 5672
      protocol: TCP
      targetPort: 5672
  selector:
    app.kubernetes.io/name: rabbitmq-cluster

mdditt2000 commented 1 year ago

@brunocascio I see the As3 declaration composed by CIS is valid Could able to successfully post the declaration to the BIGIP without any issues. I got success as well.

mdditt2000 commented 1 year ago

@brunocascio can i close this issue?

mdditt2000 commented 1 year ago

Note

The last error reported is

[code:422 kind::resterrorresponse message:request failed with null exception referer:172.17.68.63 restOperationId:8.40709713e+08]

This might go away with possible retry post AS3 calls by CIS.

brunocascio commented 1 year ago

This might go away with possible retry post AS3 calls by CIS.

It's retrying every 30s without any success.

Could it be related to the k8s version? I'm running kubernetes 1.20

I'll give it a try with another cluster running k8s 1.24 to see if it works

mdditt2000 commented 1 year ago

@brunocascio K8S dont believe version will make a different. Can you share you application with my team so we can recreate internal. Please email me at automation_toolchain_pm@f5.com or share here

brunocascio commented 1 year ago

@brunocascio K8S dont believe version will make a different. Can you share you application with my team so we can recreate internal. Please email me at automation_toolchain_pm@f5.com or share here

Created same example with a dummy app, getting the same error:

2023/06/08 20:02:31 [ERROR] [AS3] Raw response from Big-IP: map[code:422 kind::resterrorresponse message:request failed with null exception referer:172.17.68.13 restOperationId:8.4848867e+08] {"$schema":"https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/master/schema/3.33.0/as3-schema-3.33.0-4.json","class":"AS3","declaration":{"class":"ADC","controls":{"class":"Controls","userAgent":"CIS/v2.13.0 K8S/v1.20.12+vmware.1"},"dev-nonprod-k8s-cis":{"Shared":{"class":"Application","crd_10_12_204_254_32222":{"class":"Service_TCP","virtualAddresses":["10.12.204.254"],"virtualPort":32222,"snat":"auto","pool":"test_32222_app_devops_techweek_dev","profileL4":"basic"},"template":"shared","test_32222_app_devops_techweek_dev":{"class":"Pool","members":[{"addressDiscovery":"static","serverAddresses":["172.17.68.51"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.53"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.52"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.62"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.59"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.34"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.39"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.32"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.38"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.57"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.54"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.40"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.41"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.18"],"servicePort":32222,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.49"],"servicePort":32222,"shareNodes":true}]}},"class":"Tenant","defaultRouteDomain":0},"id":"urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d","label":"CIS Declaration","remark":"Auto-generated by CIS","schemaVersion":"3.33.0"}}

apiVersion: apps/v1
kind: Deployment
metadata:
  name:  test
spec:
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app:  test
    spec:
      containers:
      - name:  test
        image:  nginx:stable-alpine
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort:  80
          name: nginx-http
---
apiVersion: v1
kind: Service
metadata:
  name: test
spec:
  selector:
    app: test
  type: NodePort
  sessionAffinity: None
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  ports:
  - name: tcp-80
    appProtocol: TCP
    protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 32222
---
apiVersion: cis.f5.com/v1
kind: TransportServer
metadata:
  name: test
  labels:
    f5cr: "true"
spec:
  virtualServerAddress: "10.12.204.254"
  virtualServerPort: 32222 
  type: tcp
  mode: standard
  snat: auto
  pool:
    service: test
    servicePort: 32222

Note: I know nginx is http, but this is just for testing purposes

brunocascio commented 1 year ago

Hey @mdditt2000 do you have any update about this?

trinaths commented 1 year ago

@brunocascio Can you try with AS3 3.44 and CIS 2.13.1 and share your findings.

brunocascio commented 1 year ago

Hi @trinaths

Yesterday was reading the f5-ctrl go code and found that the errors are logged in "debug" log level. Once I enabled logLevel: DEBUG I found this issue:

It helped me to understand that my rabbitmq services was not a NodePort one... So, after putting it as NodePort, that error dissapear, but now I get a new one.

Message

pool (/dev-nonprod-k8s-cis/Shared/ingress_istio_ingress_istio_ingressgateway) is referenced by one or more rules

Full message

2023/07/07 12:25:09 [ERROR] [AS3] Raw response from Big-IP: map[code:422 declaration:map[class:ADC controls:map[archiveTimestamp:2023-07-07T12:25:08.503Z class:Controls userAgent:CIS/v2.13.1 K8S/v1.20.12+vmware.1] id:urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d label:CIS Declaration remark:Auto-generated by CIS schemaVersion:3.33.0 updateMode:selective] results:[map[code:422 host:localhost message:declaration failed response:01070340:3: pool (/dev-nonprod-k8s-cis/Shared/ingress_istio_ingress_istio_ingressgateway) is referenced by one or more rules runTime:26342 tenant:dev-nonprod-k8s-cis]]] {"$schema":"https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/master/schema/3.33.0/as3-schema-3.33.0-4.json","class":"AS3","declaration":{"class":"ADC","controls":{"class":"Controls","userAgent":"CIS/v2.13.1 K8S/v1.20.12+vmware.1"},"dev-nonprod-k8s-cis":{"Shared":{"class":"Application","crd_10_12_204_254_5672":{"class":"Service_TCP","virtualAddresses":["10.12.204.254"],"virtualPort":5672,"snat":"auto","pool":"rabbitmq_cluster_nodeport_amqp_rabbitmq","profileL4":"basic"},"rabbitmq_cluster_nodeport_amqp_rabbitmq":{"class":"Pool","members":[{"addressDiscovery":"static","serverAddresses":["172.17.68.51"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.53"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.52"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.62"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.59"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.34"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.39"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.32"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.38"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.57"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.54"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.40"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.41"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.18"],"servicePort":30457,"shareNodes":true},{"addressDiscovery":"static","serverAddresses":["172.17.68.49"],"servicePort":30457,"shareNodes":true}],"monitors":[{"use":"/dev-nonprod-k8s-cis/Shared/rabbitmq_cluster_nodeport_rabbitmq_tcp_amqp"}]},"rabbitmq_cluster_nodeport_rabbitmq_tcp_amqp":{"class":"Monitor","interval":10,"monitorType":"tcp","targetAddress":"","timeout":3,"adaptive":false,"receive":"","send":""},"template":"shared"},"class":"Tenant","defaultRouteDomain":0},"id":"urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d","label":"CIS Declaration","remark":"Auto-generated by CIS","schemaVersion":"3.33.0"}}

Btw, I'm running 213.1

brunocascio commented 1 year ago

New updates!

I did manage to get it working by using a different partition and a new VIP. For some reason, I can't use the same VIP for an ingress and a TransportServer (even with different ports)

My solution was to use a different partition and a different VIP (because I can't use the same VIP in different partition neither)

trinaths commented 1 year ago

Thanks for the update. Closing this issue.

F5Networks / k8s-bigip-ctlr