F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
357 stars 195 forks source link

Wrong Pool member port configured by CIS 2.12-pre #2767

Closed andreasjacobi closed 1 year ago

andreasjacobi commented 1 year ago

Setup Details

CIS Version : 2.12-pre
Build: nandakishoref5/k8s-bigip-ctlr:VSPoolWafSupport
BIGIP Version: Big IP 16.1.3.2
AS3 Version: 3.36.1
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: 1.23.9
Pool Mode: Cluster
Additional Setup details: Cilium

Description

I have a TransportServer setup consisting of 8 different TS pointing to the same service but with different ports and protocol. Using the new 2.12-pre build they all failed except one and the reason is that all the Pool members port configuration in the F5 changed to one and the same. They all got ports reassigned to 3478, so all, except that specific TS that actually requires that port, failed. Reverting to 2.11.1 solves the issue.

It looks like CIS picks the first configured port in the svc and use that for all Pool members.

Steps To Reproduce

Below config works in 2.11.1 but not in 2.12-pre.

apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-mgmt
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 8443
  virtualServerName: ts-unifi-tcp-mgmt
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 8443
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-devcom
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 8080
  virtualServerName: ts-unifi-tcp-devcom
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 8080
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-shutdown
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 8081
  virtualServerName: ts-unifi-tcp-shutdown
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 8081
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-httpsredir
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 8843
  virtualServerName: ts-unifi-tcp-httpsredir
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 8843
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-httpredir
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 8880
  virtualServerName: ts-unifi-tcp-httpredir
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 8880
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-tcp-speedtest
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 6789
  virtualServerName: ts-unifi-tcp-speedtest
  mode: standard
  type: tcp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 6789
    monitor:
      type: tcp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-udp-stun
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 3478
  virtualServerName: ts-unifi-udp-stun
  mode: standard
  type: udp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 3478
    monitor:
      type: udp
      interval: 10
      timeout: 10
---
apiVersion: "cis.f5.com/v1"
kind: TransportServer
metadata:
   name: ts-unifi-udp-discovery
   labels:
     f5cr: "true"
spec:
  virtualServerAddress: "10.1.1.91"
  virtualServerPort: 10001
  virtualServerName: ts-unifi-udp-discovery
  mode: standard
  type: udp
  snat: auto
  pool:
    service: unifi-controller
    servicePort: 10001
    monitor:
      type: udp
      interval: 10
      timeout: 10

This is the svc:

kubectl describe svc -n unifi
Name:              unifi-controller
Namespace:         unifi
Labels:            app=unifi-controller
                   kustomize.toolkit.fluxcd.io/name=unifi
                   kustomize.toolkit.fluxcd.io/namespace=unifi
Annotations:       <none>
Selector:          app=unifi-controller
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.109.206.12
IPs:               10.109.206.12
External IPs:      10.1.1.91
Port:              3478  3478/UDP
TargetPort:        3478/UDP
Endpoints:         10.245.1.61:3478
Port:              10001  10001/UDP
TargetPort:        10001/UDP
Endpoints:         10.245.1.61:10001
Port:              8080  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.245.1.61:8080
Port:              8081  8081/TCP
TargetPort:        8081/TCP
Endpoints:         10.245.1.61:8081
Port:              8443  8443/TCP
TargetPort:        8443/TCP
Endpoints:         10.245.1.61:8443
Port:              8843  8843/TCP
TargetPort:        8843/TCP
Endpoints:         10.245.1.61:8843
Port:              8880  8880/TCP
TargetPort:        8880/TCP
Endpoints:         10.245.1.61:8880
Port:              6789  6789/TCP
TargetPort:        6789/TCP
Endpoints:         10.245.1.61:6789
Session Affinity:  None
Events:            <none>

In the BigIP the 8 different Pools will have correct names and even monitors but looking at the Pool member port configuration they will all be reverted to that one port (3478 in my case).

Expected Result

CIS should use the configured ports as port for the pool member, not just the first port in the svc list.

Actual Result

CIS uses the first port in the svc for all pools using that same svc.

trinaths commented 1 year ago

Created [CONTCNTR-3800] for internal tracking