CIS+Openshift 4.10 dual stack

[stwhite5]

Setup Details

CIS Version : 2.9.1
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 15.1.3 AS3 Version: 3.18
Agent Mode: AS3
Orchestration: OSCP
Orchestration Version:
Pool Mode: Cluster
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc>

oc get networks cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
spec:
  clusterNetwork:
  - cidr: 192.168.0.0/16
    hostPrefix: 23
  - cidr: fdxx::/48
    hostPrefix: 64
  externalIP:
    policy: {}
  networkType: OVNKubernetes
  serviceNetwork:
  - 198.x.0.0/16
  - fdxx:xxxx:xxxx::/112
status:
  clusterNetwork:
  - cidr: 192.168.0.0/16
    hostPrefix: 23
  - cidr: fdxx::/48
    hostPrefix: 64
  clusterNetworkMTU: 8500
  networkType: OVNKubernetes
  serviceNetwork:
  - 198.x.0.0/16
  - fdxx:xxxx:xxxx::/112

CIS config:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  name: k8s-bigip-0-ctlr-deployment
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: k8s-bigip-0-ctlr
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: k8s-bigip-0-ctlr
      name: k8s-bigip-0-ctlr
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nodeType
                operator: NotIn
                values:
                - loadbalancer
      containers:
      - args:
        - --bigip-username=$(BIGIP_USERNAME)
        - --bigip-password=$(BIGIP_PASSWORD)
        - --bigip-url=[BIGIP_ADDRESS]
        - --bigip-partition=kubernetes
        - --insecure=true
        - --agent=as3
        - --namespace=f5
        - --pool-member-type=cluster
        - --as3-validation=true
        - --manage-ingress=false
        - --manage-routes=false
        - --custom-resource-mode=false
        - --manage-configmaps=true
        - --log-level=DEBUG
        - --log-as3-response=true
        - --openshift-sdn-name=/Common/ingress
        command:
        - /app/bin/k8s-bigip-ctlr
        env:
        - name: BIGIP_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: bigip-login
        - name: BIGIP_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: bigip-login
        image: private_repository/f5/cis/k8s-bigip-ctlr:2.9.1
        imagePullPolicy: IfNotPresent
        name: k8s-bigip-ctlr
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: k8s-bigip-ctlr
      serviceAccountName: k8s-bigip-ctlr
      terminationGracePeriodSeconds: 30

Description

CIS installed in OSCP 4.10 dual stack configuration. k8s-bigip-ctlr cannot create tunnels due to V6 addresses. From the CIS logs:

2022/08/22 14:54:47 [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record: <node ipv6 address>
2022/08/22 14:54:47 [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record: <node ipv6 address>
2022/08/22 14:54:47 [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record: <node ipv6 address>
2022/08/22 14:54:47 [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record: <node ipv6 address>
2022/08/22 14:54:47 [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record:<node ipv6 address>
2022/08/22 14:54:47 [DEBUG] [CCCL] ConfigWriter (0xc000415aa0) writing section name vxlan-fdb
2022/08/22 14:54:47 [DEBUG] [CCCL] ConfigWriter (0xc000415aa0) successfully wrote section (vxlan-fdb)
2022/08/22 14:54:47 [DEBUG] [VxLAN] Vxlan manager (ocp-ingress) wrote config section: [{ <node ipv6 address>} { <node ipv6 address>} { <node ipv6 address>} { <node ipv6 address>} { <node ipv6 address>}]

Steps To Reproduce

1) Install and configure OSCP4.10 2) Install and configure CIS 2.9.1 3) get logs of k8s-bigip-ctlr pod

Expected Result

vxlan tunnels will be created using IPv4 addresses

Actual Result

vxlan tunnel creation fails because CIS submits IPv6 addresses via IPv4 config, failing with bad address format.

Diagnostic Information

Observations (if any)

The current configuration works properly with OSCP 4.7. I have not tested with OSCP 4.10 single-stack V4 because my customer requires dual stack functionality in OSCP/CIS

[vincentmli]

I think CIS could simply check IPV6 node address in for _, addr loop below and continue to bypass it.

                for _, addr := range nodeAddrs {                                 
                        if addr.Type == addrType {                               
                                rec.Endpoint = addr.Address                      
                                // Initially set the name to a fake MAC (for OpenShift use)
                                // For flannel, this will be overwritten with the real MAC
                                rec.Name = ipv4ToMac(addr.Address)               
                        }                                                        
                }

// Convert an IPV4 string to a fake MAC address.                                 
func ipv4ToMac(addr string) string {                                             
        ip := strings.Split(addr, ".")                                           
        if len(ip) != 4 {                                                        
                log.Errorf("[VxLAN] Bad IPv4 address format specified for FDB record: %s", addr)
                return ""                                                        
        }                                                                        
        var intIP [4]int                                                         
        for i, val := range ip {                                                 
                intIP[i], _ = strconv.Atoi(val)                                  
        }                                                                        
        return fmt.Sprintf("0a:0a:%02x:%02x:%02x:%02x", intIP[0], intIP[1], intIP[2], intIP[3])
}                                                                                

[vincentmli]

@stwhite5 I looked briefly through CIS code commits history, I could not find IPV6 FDB entry is ever supported, so this would be a request for enhancement, could you please file a request for enhancement through our support ticket system? support engineer could send you a RFE template to fill it out. in the meantime, we could come up a temporary code fix to bypass IPV6 FDB creation so you won't get the error log [ERROR] [VxLAN] Bad IPv4 address format specified for FDB record: <node ipv6 address>, and CIS could function for IPV4 traffic, but not IPV6, is that an option? if so, I have the test image available from docker hub vli39/cis:noipv6 for you to test in your test environment, please let me know what you think.

[vincentmli]

just post the workaround here

diff --git a/pkg/vxlan/vxlanMgr.go b/pkg/vxlan/vxlanMgr.go
index 39c4b62d..99553d26 100644
--- a/pkg/vxlan/vxlanMgr.go
+++ b/pkg/vxlan/vxlanMgr.go
@@ -128,6 +128,11 @@ func (vxm *VxlanMgr) ProcessNodeUpdate(obj interface{}, err error) {
                nodeAddrs := node.Status.Addresses
                rec := fdbRecord{}
                for _, addr := range nodeAddrs {
+                       ip := strings.Split(addr.Address, ".")
+                       if len(ip) != 4 {
+                               log.Warningf("[VxLAN] IPv6 is not supported for FDB record: %s", addr.Address)
+                               continue
+                       }
                        if addr.Type == addrType {
                                rec.Endpoint = addr.Address
                                // Initially set the name to a fake MAC (for OpenShift use)```

[nandakishorepeddi]

@stwhite5 , Can you try removing argument "--openshift-sdn-name=/Common/ingress" from the CIS deployment config. In case of using OVNKubernetes and just add the static routes in BIGIP to route POD traffic via openshift nodes. OVNKubernetes CNI with iCNI(without tunnels) will help, and CIS will not add any FDB entries on the BIGIP.

[nandakishorepeddi]

Please feel to reopen in case this doesnt work.