BIG-IP VE can not catch up the Pod/Service's change after use namespace-label to isolate namespace

[kylinsoong]

Setup Details

CIS Version : 1.14
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 13.1.3
AS3 Version: 3.19
Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version:
Pool Mode: Cluster/Nodeport
Additional Setup details: Kubernets 1.15.12, flannel host-gw

Description

The pool member can not update automatically once the Pod recreated after running k8s-bigip-ctlr with --namespace-label parameter.

NOTE: if k8s-bigip-ctlr start without --namespace-label parameter, the BIG-IP VE can catch up the underlying K8S Pod change.

Steps To Reproduce

1) Install k8s-bigip-ctlr with --namespace-label parameter(completed yaml refer to Appendix 1) 2) Using configmap with as3 declaration deploy VS to BIG-IP VE, note that the pool members referenced with VS are K8S pods, pool member IP are Pod IP; 3) Deleting some of pods from K8S(related with BIG-IP VE Pool member in above step 2), this will cause new pods created with new IP Address.

NOTE: the namespace has a label cis_scanner_zone=zone_3.

Expected Result

The Pool member should be updated automatically, the pool member should point to new created Pod.

Actual Result

The pool member never updated, the health check failed, the pool and vs in Offline status.

Diagnostic Information

Observations (if any)

Appendix 1

apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-bigip-ctlr
  namespace: kube-system
  labels:
    app: k8s-bigip-ctlr
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bigip-ctlr
  template:
    metadata:
      name: k8s-bigip-ctlr
      labels:
        app: k8s-bigip-ctlr
    spec:
      serviceAccountName: bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr
          image: "f5networks/k8s-bigip-ctlr:latest"
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=https://10.1.10.240:443",
            "--insecure=true",
            "--bigip-partition=k8s",
            "--pool-member-type=cluster",
            "--namespace-label=cis_scanner_zone=zone_3",
            "--vs-snat-pool-name=TEST_SNAT_Pool",
            ]

[kylinsoong]

use --namespace has the same issue.

[kylinsoong]

After verified with latest 2.0.0, this issue be fixed. The issue be fixed in #1145, #1185, #1295

[mdditt2000]

@kylinsoong thank you for the feedback!

[kylinsoong]

After some estimation, the customer still want to use CIS 1.14, rather than 2.0, so I reopen this issue, request to fix this issue in 1.14.

[kylinsoong]

@mdditt2000 The 2.0 enhancement not complete solve the issue. The CIS 2.0 behavior like this:

If enable namespace isolation in CIS 2.0, service auto discover/pod change catch up only works on namespace that configmap resident.

Note that the CIS 2.0 only allow one centralized confgimap, if POD/Service change in namespace that not contain confgimap, the change can not be catched by F5 VE。

---

The following is a reproduce procedure:

CIS 2.0 args like:

   "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=172.16.20.205",
            "--bigip-partition=k8s",
            "--pool-member-type=cluster",
            "--namespace-label=zone=zone2",

set label on as-1, put configmap on as-1, then the POD change can be catched by CIS and VE

# kubectl get ns --show-labels
NAME              STATUS   AGE    LABELS
as3-1             Active   9h     zone=zone2
as3-2             Active   22h    <none>

set label on both as-1 and as-2, put configmap on as-1, then the POD change on as-2 can not be catched by CIS and VE

# kubectl get ns --show-labels
NAME              STATUS   AGE    LABELS
as3-1             Active   9h     zone=zone2
as3-2             Active   22h    zone=zone2

[mdditt2000]

@kylinsoong what is the namespace for both as3-1 and as3-2?

[kylinsoong]

@mdditt2000 as3-1 and as3-2 are the name of namespace in K8S.

In above reproduce steps, one Controller monitor on 2 namesapce(as3-1 and as3-2 ).

[mdditt2000]

Escalation SR 1-6382539511 Internal PM Jira filed for Sprint 43 - CONTCNTR-1948

[mdditt2000]

@kylinsoong i have recreated the issue and see what is going on. CIS 2.0 is only allowing a single configmap. CIS 1.14 does allow you to add multiple configmap.

[mdditt2000]

CIS can monitor the services only for the namespace of configmap or namespace isolation.