After upgrade from 2.16.1 to 2.18, pool pointing to wrong endpoint

jhennf5 commented 1 month ago

Setup Details

CIS Version : 2.16.1 -> 2.18.0
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 16.1.4.3
AS3 Version: 3.x
Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version:
Pool Mode: Cluster
Additional Setup details: OpenShift

Description

F5 Case 00705492 Data sanitized

Service

Port: test-web 8080/TCP TargetPort: 8080/TCP NodePort: test-web 32348/TCP Endpoints: 10.10.10.10:8080 Port: test-web-ssl 443/TCP TargetPort: 8443/TCP NodePort: test-web-ssl 31525/TCP Endpoints: 10.10.10.10:8443 Session Affinity: None

Config Map

        "class": "Pool",
        "monitors": ["https"],
             "members": [
              {
                "servicePort": 8443,
                "serverAddresses": []
              }
          ]
         },

2.16.1 Controller logs

[DEBUG] [CORE] Discovered members for service test/test-console-svc is [{10.10.10.10 8443 443 0} {10.10.10.10 8080 8080 0}]

2.18.0 Controller logs

[DEBUG] [CORE] Discovered members for service test/test-console-svc is [{10.10.10.10 8443 443 enable 0 0} {10.10.10.10 8080 8080 enable 0 0}]

Behavior on upgrade to 2.18.0:

obj_delete { pool_member { pool_member_pool_name "/test-configmap/test/test-console-svc" pool_member_node_name "/test-configmap/10.10.10.10" pool_member_port 8443 } } [Status=Command OK] create_if { pool_member { pool_member_pool_name "/test-configmap/test/test-console-svc" pool_member_node_name "/test-configmap/10.10.10.10" pool_member_port 8080 pool_member_new_session_enable 2 pool_member_monitor_state 3 pool_member_rate_limit 0 pool_member_dynamic_ratio 1 pool_member_priority 0 pool_member_ratio 1 pool_member_conn_limit 0 pool_member_autoscale 0 } } [Status=Command OK]

Behavior on revert to 2.16.1:

obj_delete { pool_member { pool_member_pool_name "/test-configmap/test/test-console-svc" pool_member_node_name "/test-configmap/10.10.10.10" pool_member_port 8080 } } [Status=Command OK] create_if { pool_member { pool_member_pool_name "/test-configmap/test/test-console-svc" pool_member_node_name "/test-configmap/10.10.10.10" pool_member_port 8443 pool_member_new_session_enable 2 pool_member_monitor_state 3 pool_member_rate_limit 0 pool_member_dynamic_ratio 1 pool_member_priority 0 pool_member_ratio 1 pool_member_conn_limit 0 pool_member_autoscale 0 } } [Status=Command OK]

Controller flags:

'--pool-member-type=cluster'
'--insecure=true'
'--log-as3-response=true'
'--openshift-sdn-name=-----'
'--namespace=-----'
'--namespace=test'
'--share-nodes=true'
'--as3-post-delay=30'
'--disable-teems=true'
'--log-level=INFO'

Steps To Reproduce

1) Upgrade/revert controller

Expected Result

Pool member should use port 8443

Actual Result

Pool member uses port 8080

Diagnostic Information

Internal F5 data in case 00705492

trinaths commented 1 month ago

@jhennf5 Please share sample manifest YAML to reproduce this issue.

trinaths commented 1 month ago

Created [CONTCNTR-4947] for internal tracking

jhennf5 commented 1 month ago

@jhennf5 Please share sample manifest YAML to reproduce this issue.

Shared internally

arzzon commented 2 weeks ago

@jhennf5 In this case, Port 8443 is used as the servicePort in the poolMemeber present in the AS3 configmap, where as the following ports are used in the service: servicePort: 443 targetPort: 8443 As per the documentation mentioned below, the servicePort(8443) defined in the AS3 ConfigMap doesn't match the actual servicePort(443). So CIS takes the resource's port on index 0. https://clouddocs.f5.com/containers/latest/userguide/config-map.html#cis-service-discovery

CIS performs service discovery using extracted tenant (Tenant-1), Application (APP1) and Pool (web_pool) details, and fetches service endpoints. If servicePort, defined in the AS3 ConfigMap, matches the actual servicePort in the service we add the members using targetPort in cluster mode. If servicePort, defined in the AS3 ConfigMap, does not match the actual servicePort in the service, we add the service’s corresponding endpoint and the resource’s port on index 0 to support backward compatibility.

Port 8443 was used by CIS 2.16.1 because in case when service ports don't match, CIS uses the port on index 0, however CIS 2.18.1 now maintains a sorted order for the pool members which has been done to fix an issue of multiple posting of declaration. So, the port at index 0 is now changed due to sorting and CIS used the port 8080 instead.

Please use port 443 for the pool member in the AS3 configMap, to ensure the service ports match and port 8443 is used for the pool member.

F5Networks / k8s-bigip-ctlr