OpenShift OVN-Kubernetes using F5 BIG-IP HA with NO Tunnels nothing created in f5

[dsullivarh]

Setup Details

CIS Version : 2.12.0
Build: f5networks/k8s-bigip-ctlr:latest f5networks/cntr-ingress-svcs:
BIGIP Version: Big IP x.x.x
14.1.4.6 AS3 Version: 3.x
Agent Mode: AS3/CCCL default is as3
Orchestration: K8S/OSCP OSCP
Orchestration Version: OSCP 4.12.5
Pool Mode: Cluster/Nodeport Cluster
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc> OVNK Kuberenetes

Description

We don't see any route configuration being added following documentation from [0]

It would be nice to see a working configuration with the versions specified above.

[0] https://github.com/mdditt2000/k8s-bigip-ctlr/tree/main/user_guides/ovn-kubernetes-ha#readme

Steps To Reproduce

See above...

oc get deployment k8s-bigip-ctlr-01-deployment -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:    
  labels:
    app.kubernetes.io/instance: cntr-ingress-svcs
  name: k8s-bigip-ctlr-01-deployment
  namespace: kube-system
 spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: k8s-bigip-ctlr-01-deployment
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: k8s-bigip-ctlr-01-deployment
    spec:
      containers:
      - args:
        - --bigip-username=$(BIGIP_USERNAME)
        - --bigip-password=$(BIGIP_PASSWORD)
        - --bigip-url=XX.XX.XX.XX
        - --bigip-partition=xxxx-k8s-bigip-ctlr---
        - --pool-member-type=cluster
        - --log-level=debug
        - --insecure=true
        - --ipam=false
        - --namespace=cool-project
        - --route-spec-configmap=kube-system/global-cm
        - --controller-mode=openshift
        - --as3-validation=true
        - --log-as3-response=true
        - --disable-teems
        command:
        - /app/bin/k8s-bigip-ctlr
        env:
        - name: BIGIP_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: bigip-login
        - name: BIGIP_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: bigip-login
        - name: TZ
          value: America/New_York
        image: fixme.acme.org/ocpv4/f5networks/cntr-ingress-svcs@sha256:cd810e1f3bc1da85581a6659b44a9bcefd300fd9e231ab44c62a340ccb2c5805
        imagePullPolicy: IfNotPresent
        name: cntr-ingress-svcs
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: bigip-ctlr
      serviceAccountName: bigip-ctlr
      terminationGracePeriodSeconds: 30

[p0ocptcp@p1plvocpm001 ~]$ oc get pods
NAME                                            READY   STATUS    RESTARTS   AGE
k8s-bigip-ctlr-01-deployment-6ff6dd75cb-s9mnl   1/1     Running   0          12m

 oc get cm global-cm -o yaml
apiVersion: v1
data:
  extendedSpec: "extendedRouteSpec: \n- namespace: cool-project\n  vserverAddr: 10.XX.XX.XX\n
    \ vserverName: xxx-wrp-low\n  allowOverride: true\n  policyCR: default/sample-policy\n"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: cntr-ingress-svcs
    f5nr: "true"
  name: global-cm
  namespace: kube-system

route configuration

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    f5nr: "true"
  name: xxxx-wrp-low
  namespace: cool-project 
spec:
  host: blah-blah.acme.org
  port:
    targetPort: 9443
  tls:
    insecureEdgeTerminationPolicy: Redirect
    termination: passthrough
  to:
    kind: Service
    name: xxxx-wrp-low
    weight: 100
  wildcardPolicy: None

Expected Result

No errors in logs we see it talking successfully to f5

But we never see any as3 route post happing

Actual Result

oc logs k8s-bigip-ctlr-01-deployment-6ff6dd75cb-s9mnl | grep -i "as3" 2023/03/15 21:32:59 [DEBUG] [AS3] No certs appended, using only system certs 2023/03/15 21:32:59 [DEBUG] [AS3] posting GET BIGIP AS3 Version request on https://XX.XXX.XX.XX/mgmt/shared/appsvcs/info 2023/03/15 21:33:02 [DEBUG] [AS3] BIGIP is serving with AS3 version: 3.43

oc logs k8s-bigip-ctlr-01-deployment-6ff6dd75cb-s9mnl | grep "Finished syncing" 2023/03/15 21:33:03 [DEBUG] Finished syncing local extended spec configmap: kube-system/global-cm (37.671µs) 2023/03/15 21:33:03 [DEBUG] Finished syncing RouteGroup/Namespace cool-project (118.18µs)

Diagnostic Information

See above

We never see any as3 post information four routing, but you do see it doing a get for as3 version

Observations (if any)

policyCR seems to be a required field although documentations says optional

[trinaths]

@dsullivarh Please share service and deployment manifests to validate this issue.

[trinaths]

Were all the resources in the same namespace CIS is monitoring ?

[dsullivarh]

All the route resources were in the same namespace cis was monitoring

[dsullivarh]

oc get svc isam-wrp-low -o yaml apiVersion: v1 kind: Service metadata: name: xxxx-wrp-low namespace: cool-project spec: clusterIP: XX.XX.XX.XX clusterIPs:

• XX.XX.XX.XX internalTrafficPolicy: Cluster ipFamilies:
• IPv4 ipFamilyPolicy: SingleStack ports:
• name: xxxx-wrp-low-https port: 9443 protocol: TCP targetPort: 9443 selector: app: xxxx-wrp-low sessionAffinity: None type: ClusterIP status: loadBalancer: {}

[dsullivarh]

Are you asking for the deployment of the app, we use statefulset but that shouldn't matter we don't see anything going into the f5 with what we have and we don't see any errors in the big ip controller pod

[mdditt2000]

@dsullivarh we scheduling a meeting for next week. Have a great weekend.

[trinaths]

Created [CONTCNTR-3889] for internal tracking.

[trinaths]

Fixed with PR : https://github.com/F5Networks/k8s-bigip-ctlr/pull/2833 Closing this issue.