In iRule openshift_passthrough_irule the variable "$dflt_pool" could not be set correctly when http/2-profile is linked to VS

joebride commented 4 years ago

Setup Details

CIS Version : 2.1.0
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 15.1.0.3
AS3 Version: 3.20.0-build3 Agent Mode: AS3
Orchestration: K8S/OSCP v1.17.1, OS v4.2 Orchestration Version:
Pool Mode: Cluster Additional Setup details: VS using http/2-profile on client-side

Description

When I added a http/2-profile (clientside only) to my virtual-server the BIG-IP logs EVERY client-request to /var/log/ltm. It looks like the following: Aug 12 16:13:45 debug tmm1[19190]: Rule //Shared/openshift_passthrough_irule : Unable to find pool for

When I removed the referenced http/2-profile from the virtual-server, the BIG-IP logs client-requests ONLY, which do NOT match the LTM-Policy (hostname).

Steps To Reproduce

1) associate http/2-profile (standard one or customized one) to VS 2) start client-request which matches to one of the LTM Policy rule (defined in openshift_secure_routes and openshift_insecure_routes) 3) BIG-IP create entry in /var/log/ltm

Expected Result

valid client-requests should NOT create an entry in /var/log/ltm invalid client-requests should create an entry in /var/log/ltm

Actual Result

valid client-requests create an entry in /var/log/ltm invalid client-requests create an entry in /var/log/ltm

Diagnostic Information

Example-Entry in /var/log/ltm Aug 12 16:13:45 debug tmm1[19190]: Rule //Shared/openshift_passthrough_irule : Unable to find pool for

Excerpt CIS-config image: 'registry.connect.redhat.com/f5networks/cntr-ingress-svcs:2.1.0-rhel7' args:

'--bigip-username=$(BIGIP_USERNAME)'
'--bigip-password=$(BIGIP_PASSWORD)'
'--bigip-url=A.B.C.D'
'--bigip-partition=XXX'
'--pool-member-type=cluster'
'--openshift-sdn-name=/Common/RD_000_TUN_VXL_0001'
'--log-level=INFO'
'--log-as3-response=true'
'--manage-routes=true'
'--manage-ingress=false'
'--route-label=XXX'
'--route-http-vserver=RD_000_VS_RED_XXX_80'
'--route-https-vserver=RD_000_VS_STD_XXX_443'
'--route-vserver-addr=A.B.C.D'
'--agent=as3'
'--insecure=true'
'--tls-version=1.3'
'--cipher-group=/Common/CIPHER_GROUP_AES-GCM_ECDHE_DHE_TLSV12_TLSV13'

Observations (if any)

In iRule openshift_passthrough_irule the variable "$dflt_pool" could not be set correctly when http/2-profile is linked to VS.

mdditt2000 commented 4 years ago

Jira created for internal PM tracking CONTCNTR-2133

mdditt2000 commented 3 years ago

@joebride we are working internally to get 15.1.0.3 + ENG-HF - Without this build HTTP2 does not work correctly.

joebride commented 3 years ago

Issue originally occured in Rel. 15.1.0.3-ENG (0.86.12). But it occures in Rel. 14.1.2.5-ENG (0.101.3), too!

mdditt2000 commented 3 years ago

@joebride can you please confirm. Does 15.1.0.3-ENG (0.86.12) not include the HTTP2 hotfixes that you are currently testing? Maybe to accelerate our testing, could you share the BIP-IP VE build/image or download link where you got the build so that you are using and can reproduce these CIS issues. I want to make sure we are all using the same build.

joebride commented 3 years ago

15.1.0.3-ENG (0.86.12) is a special build for BeWAF (supporting FP) currently used for testing. But as I stated Rel. 14.1.2.5-ENG (0.101.3) is affected, too!

15.1.0.3-ENG (0.67.12) is the SW-version including various http/2-bugs. I don´t have any download link.

nandakishorepeddi commented 3 years ago

Hi @joebride , Could not able to reproduce the issue at our end. Used BIG-IP 15.1.0.3 Build 0.86.12 Engineering Hotfix and applied Http2 profile manually to VS LTM. Observed that no logs are being observed at BIGIP /var/log/ltm for a valid Client request. Only observed for an invalid client request. Tried configuring routes with only edge/None , only edge/Redirect routes and a combination of both.

Could you please provide us more details about the configuration?

My configuration for combination of edge-Redirect and edge-None:(tried even using all routes with edge/None)

centos@close-ewe-bastion:~$ oc get routes
NAME              HOST/PORT                          PATH      SERVICES          PORT    TERMINATION     WILDCARD
svc-as3-route-1   pytest-oss-edge-1.com ... 1 more   /first    svc-as3-route-1   <all>   edge/Redirect   None
svc-as3-route-2   pytest-oss-edge-1.com ... 1 more   /         svc-as3-route-2   <all>   edge/None       None
svc-as3-route-3   pytest-oss-edge-2.com ... 1 more   /third    svc-as3-route-3   <all>   edge/None       None
svc-as3-route-4   pytest-unsecured.com ... 1 more    /fourth   svc-as3-route-4   <all>                   None

Traffic Sent:

centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  12482      0 --:--:-- --:--:-- --:--:-- 12595
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  14556      0 --:--:-- --:--:-- --:--:-- 14601
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/dummy -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) TCP connection reset by peer
1 centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  14558      0 --:--:-- --:--:-- --:--:-- 14473
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:443:172.16.3.22 https://pytest-oss-edge-1.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1304  100  1304    0     0  10853      0 --:--:-- --:--:-- --:--:-- 10866
service_name=svc-as3-route-2
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:443:172.16.3.22 https://pytest-oss-edge-1.com/first -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   962  100   962    0     0   8312      0 --:--:-- --:--:-- --:--:--  8365
service_name=svc-as3-route-1
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:80:172.16.3.22 http://pytest-oss-edge-1.com/first -kv

BIGIP /var/log/ltm Logs:

Nov 19 03:57:02 bigip.h1.com warning httpd[11053]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:08 bigip.h1.com warning httpd[10902]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:11 bigip debug tmm1[10876]: Rule /test/Shared/openshift_passthrough_irule <CLIENTSSL_DATA>: Unable to find pool for pytest-oss-edge-2.com
Nov 19 03:57:16 bigip.h1.com warning httpd[11208]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:21 bigip.h1.com warning httpd[12232]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:30 bigip.h1.com warning httpd[10965]: 0118000a:4: The Service Check Date check was skipped.

CIS controller parameters:

containers:
      - args:
        - --bigip-partition
        - test
        - --bigip-url
        - <x.y.z.a>
        - --bigip-username
        - <username>
        - --bigip-password
        - <password>
        - --verify-interval
        - "2"
        - --node-poll-interval
        - "1"
        - --log-level
        - DEBUG
        - --as3-validation=false
        - --insecure
        - --manage-routes=true
        - --route-vserver-addr=172.16.3.22
        - --route-label=systest
        - --openshift-sdn-name
        - /test/vxlan-tunnel-mp
        - --pool-member-type
        - cluster
        - --tls-version=1.3
        - --cipher-group=/Common/CIPHER_GROUP_AES-GCM_ECDHE_DHE_TLSV12_TLSV13
        - --log-as3-response
        command:
        - /app/bin/k8s-bigip-ctlr
        image: f5networks/k8s-bigip-ctlr:2.1.0

joebride commented 3 years ago

To reproduce the issue you must have reachable PODs With latest CISv2.2.0 I do not see entries anymore in /var/log/ltm. Maybe iRule "openshift_passthrough_irule" was optimized to make messages invisible.

When activating http2-profile on client-side of VS, the client cannot connect when set to http2 (curl --http2 ...) ! TCPDUMP shows me TCP-RST from BIG-IP with rst_cause >>> "[0x2a34fea:6605] iRule execution (reject command)"

When forcing client to use http/1.1 (curl --http1.1 ...) the connection is successful!!! When removing http2-profile on client-side of VS, the client can connect succesfully.

+++ I use A/B-testing on a second OS-Route. For testing purposes I removed it. weight: 20 alternateBackends:

kind: Service name: httpd-b-example weight: 80

But no improvement occurs, still the same error!

+++ Here are the configured OS-Routes: C:\Users\A307017\OneDrive - Deutsche Telekom AG\Downloads_Portable>oc4 get routes -n f5test NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD httpd-b-example httpd-b-example-f5test.apps.tc06.otc.t-internal.com ... 1 more httpd-b-example 8080-tcp edge/Redirect None httpd-example httpd-example-f5test.apps.tc06.otc.t-internal.com ... 2 more httpd-example(20%),httpd-b-example(80%) 8080-tcp edge/Redirect None

+++ C:\Users\A307017\OneDrive - Deutsche Telekom AG\Downloads_Portable>oc4 get routes -n juice NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD juice-external juice.t-lefonica.de ... 2 more juice-shop 3000-tcp edge/Redirect None juice-shop juice-shop-juice.apps.tc06.otc.t-internal.com ... 1 more juice-shop 3000-tcp None

+++ VS-config ltm virtual RD_000_VS_STD_XXX_443 { auto-lasthop disabled description Shared destination /XXX/ ip-protocol tcp mask 255.255.255.255 partition XXX persist { /Common/PROF_PERSIST_COOKIE_SESSION_VSYYY { default yes } } policies { openshift_secure_routes { } } profiles { /Common/ASM_RD_000_WAF_BASIC_OWASP10 { } /Common/PROF_ANALYTICS_HTTP_CUSTOMIZED { } /Common/PROF_HTTP2_INSERTHEADER { context clientside } /Common/PROF_HTTP_REDIRECT_HSTS_XFF { } /Common/PROF_TCP_LANOPTIMIZED_KEEPALIVE_120 { context serverside } /Common/PROF_TCP_WANOPTIMIZED_KEEPALIVE_120_NONAGLE { context clientside } /Common/PROF_WAF_BOT_DEFENSE_RELAXED { } /Common/PROF_WAF_DOS_BA_TLSFP_BLOCKING { } /Common/websecurity { } RD_000_VS_STD_XXX_443_tls_server { context clientside } RD_000_VS_STD_XXX_443_tls_server-1- { context clientside } } rules { /Common/RD_000_RULE_SNAT_GENERIC /Common/RD_000_RULE_HTTP_XFORWARDEDHOST openshift_passthrough_irule } security-log-profiles { /Common/PROF_LOG_ASM_ALL_BOT_ALL_DOS_ALL } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { pool /Common/RD_000_SNATPOOL_GENERIC type snat } translate-address enabled translate-port enabled vlans { /Common/RD_000_VLAN_EXT } vlans-enabled }

+++ CIS-config spec: containers:

resources: {} terminationMessagePath: /dev/termination-log name: k8s-bigip-ctlr command:
- /app/bin/k8s-bigip-ctlr env:
- name: BIGIP_USERNAME valueFrom: secretKeyRef: name: bigip-login key: username
- name: BIGIP_PASSWORD valueFrom: secretKeyRef: name: bigip-login key: password imagePullPolicy: IfNotPresent terminationMessagePolicy: File image: 'registry.connect.redhat.com/f5networks/cntr-ingress-svcs:2.2.0-rhel7' args:
- '--bigip-username=$(BIGIP_USERNAME)'
- '--bigip-password=$(BIGIP_PASSWORD)'
- '--bigip-url='
- '--bigip-partition=XXX'
- '--pool-member-type=cluster'
- '--openshift-sdn-name=/Common/RD_000_TUN_VXL_0001'
- '--log-level=INFO'
- '--log-as3-response=true'
- '--manage-routes=true'
- '--manage-ingress=false'
- '--namespace=f5test'
- '--namespace=juice'
- '--route-label='
- '--route-http-vserver=RD_000_VS_RED_XXX_80'
- '--route-https-vserver=RD_000_VS_STD_XXX_443'
- '--route-vserver-addr='
- '--agent=as3'
- '--insecure=true'
- '--tls-version=1.3'
- - --cipher-group=/Common/CIPHER_GROUP_AES-GCM_ECDHE_DHE_TLSV12_TLSV13

trinaths commented 3 years ago

This issue is fixed with CIS v2.3

F5Networks / k8s-bigip-ctlr