search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
359 stars 195 forks source link

Each Client request will be logged on BIG-IP when http2-profile is associated to VS #1457

Closed joebride closed 3 years ago

joebride commented 4 years ago

Setup Details

CIS Version : 2.1.0
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 15.1.0.3
AS3 Version: 3.20.0-build3 Agent Mode: AS3
Orchestration: K8S/OSCP v1.17.1, OS v4.2 Orchestration Version:
Pool Mode: Cluster Additional Setup details: VS using http/2-profile on client-side

Description

When I added a http/2-profile (clientside only) to my virtual-server the BIG-IP logs EVERY client-request to /var/log/ltm. It looks like the following: Aug 12 16:13:45 debug tmm1[19190]: Rule //Shared/openshift_passthrough_irule : Unable to find pool for

When I removed the referenced http/2-profile from the virtual-server, the BIG-IP logs client-requests ONLY, which do NOT match the LTM-Policy (hostname).

Steps To Reproduce

1) associate http/2-profile (standard one or customized one) to VS 2) start client-request which matches to one of the LTM Policy rule (defined in openshift_secure_routes and openshift_insecure_routes) 3) BIG-IP create entry in /var/log/ltm

Expected Result

valid client-requests should NOT create an entry in /var/log/ltm invalid client-requests should create an entry in /var/log/ltm

Actual Result

valid client-requests create an entry in /var/log/ltm invalid client-requests create an entry in /var/log/ltm

Diagnostic Information

Example-Entry in /var/log/ltm Aug 12 16:13:45 debug tmm1[19190]: Rule //Shared/openshift_passthrough_irule : Unable to find pool for

Excerpt CIS-config image: 'registry.connect.redhat.com/f5networks/cntr-ingress-svcs:2.1.0-rhel7' args:

Observations (if any)

In iRule openshift_passthrough_irule the variable "$dflt_pool" could not be set correctly when http/2-profile is linked to VS.

cisbotctlr commented 4 years ago

cisbot will assign the issue to one of the devs. @devs, use /jira for internal tracking.

mdditt2000 commented 4 years ago

/jira

iam-veeramalla commented 4 years ago

/jira - CONTCNTR-2072 created for internal tracking

mdditt2000 commented 4 years ago

@joebride please could you open another Github issue for the following

Observations (if any) In iRule openshift_passthrough_irule the variable "$dflt_pool" could not be set correctly when http/2-profile is linked to VS.

Would like to track this separately. Thanks Mark

mdditt2000 commented 4 years ago

This needs to be tested with HTTP2 profile using CURL on BIG-IP 15.1.0.3

joebride commented 4 years ago

issue originally occured in Rel. 15.1.0.3-ENG (0.86.12). But it also occures in Rel. 14.1.2.5-ENG (0.101.3)

nandakishorepeddi commented 3 years ago

Hi @joebride , Could not able to reproduce the issue at our end. Used BIG-IP 15.1.0.3 Build 0.86.12 Engineering Hotfix and applied Http2 profile manually to VS LTM. Observed that no logs are being observed at BIGIP /var/log/ltm for a valid Client request. Only observed for an invalid client request. Tried configuring routes with only edge/None , only edge/Redirect routes and a combination of both.

Could you please provide us more details about the configuration?

My configuration for combination of edge-Redirect and edge-None:(tried even using all routes with edge/None)

centos@close-ewe-bastion:~$ oc get routes
NAME              HOST/PORT                          PATH      SERVICES          PORT    TERMINATION     WILDCARD
svc-as3-route-1   pytest-oss-edge-1.com ... 1 more   /first    svc-as3-route-1   <all>   edge/Redirect   None
svc-as3-route-2   pytest-oss-edge-1.com ... 1 more   /         svc-as3-route-2   <all>   edge/None       None
svc-as3-route-3   pytest-oss-edge-2.com ... 1 more   /third    svc-as3-route-3   <all>   edge/None       None
svc-as3-route-4   pytest-unsecured.com ... 1 more    /fourth   svc-as3-route-4   <all>                   None

Traffic Sent:

centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  12482      0 --:--:-- --:--:-- --:--:-- 12595
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  14556      0 --:--:-- --:--:-- --:--:-- 14601
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/dummy -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (56) TCP connection reset by peer
1 centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-2.com:443:172.16.3.22 https://pytest-oss-edge-2.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1650  100  1650    0     0  14558      0 --:--:-- --:--:-- --:--:-- 14473
service_name=svc-as3-route-3
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:443:172.16.3.22 https://pytest-oss-edge-1.com/third -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1304  100  1304    0     0  10853      0 --:--:-- --:--:-- --:--:-- 10866
service_name=svc-as3-route-2
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:443:172.16.3.22 https://pytest-oss-edge-1.com/first -k |grep service_name
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   962  100   962    0     0   8312      0 --:--:-- --:--:-- --:--:--  8365
service_name=svc-as3-route-1
centos@close-ewe-bastion:~/h2/edns$ curl --resolve pytest-oss-edge-1.com:80:172.16.3.22 http://pytest-oss-edge-1.com/first -kv

BIGIP /var/log/ltm Logs:

Nov 19 03:57:02 bigip.h1.com warning httpd[11053]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:08 bigip.h1.com warning httpd[10902]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:11 bigip debug tmm1[10876]: Rule /test/Shared/openshift_passthrough_irule <CLIENTSSL_DATA>: Unable to find pool for pytest-oss-edge-2.com
Nov 19 03:57:16 bigip.h1.com warning httpd[11208]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:21 bigip.h1.com warning httpd[12232]: 0118000a:4: The Service Check Date check was skipped.
Nov 19 03:57:30 bigip.h1.com warning httpd[10965]: 0118000a:4: The Service Check Date check was skipped.

CIS controller parameters:

containers:
      - args:
        - --bigip-partition
        - test
        - --bigip-url
        - <x.y.z.a>
        - --bigip-username
        - <username>
        - --bigip-password
        - <password>
        - --verify-interval
        - "2"
        - --node-poll-interval
        - "1"
        - --log-level
        - DEBUG
        - --as3-validation=false
        - --insecure
        - --manage-routes=true
        - --route-vserver-addr=172.16.3.22
        - --route-label=systest
        - --openshift-sdn-name
        - /test/vxlan-tunnel-mp
        - --pool-member-type
        - cluster
        - --tls-version=1.3
        - --cipher-group=/Common/CIPHER_GROUP_AES-GCM_ECDHE_DHE_TLSV12_TLSV13
        - --log-as3-response
        command:
        - /app/bin/k8s-bigip-ctlr
        image: f5networks/k8s-bigip-ctlr:2.1.0
joebride commented 3 years ago

To reproduce the issue you must have reachable PODs With latest CISv2.2.0 I do not see entries anymore in /var/log/ltm. Maybe iRule "openshift_passthrough_irule" was optimized to make messages invisible.

When activating http2-profile on client-side of VS, the client cannot connect when set to http2 (curl --http2 ...) ! TCPDUMP shows me TCP-RST from BIG-IP with rst_cause >>> "[0x2a34fea:6605] iRule execution (reject command)"

When forcing client to use http/1.1 (curl --http1.1 ...) the connection is successful!!! When removing http2-profile on client-side of VS, the client can connect succesfully.

+++ I use A/B-testing on a second OS-Route. For testing purposes I removed it. weight: 20 alternateBackends:

But no improvement occurs, still the same error!

+++ Here are the configured OS-Routes: C:\Users\A307017\OneDrive - Deutsche Telekom AG\Downloads_Portable>oc4 get routes -n f5test NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD httpd-b-example httpd-b-example-f5test.apps.tc06.otc.t-internal.com ... 1 more httpd-b-example 8080-tcp edge/Redirect None httpd-example httpd-example-f5test.apps.tc06.otc.t-internal.com ... 2 more httpd-example(20%),httpd-b-example(80%) 8080-tcp edge/Redirect None

+++ C:\Users\A307017\OneDrive - Deutsche Telekom AG\Downloads_Portable>oc4 get routes -n juice NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD juice-external juice.t-lefonica.de ... 2 more juice-shop 3000-tcp edge/Redirect None juice-shop juice-shop-juice.apps.tc06.otc.t-internal.com ... 1 more juice-shop 3000-tcp None

+++ VS-config ltm virtual RD_000_VS_STD_XXX_443 { auto-lasthop disabled description Shared destination /XXX/ ip-protocol tcp mask 255.255.255.255 partition XXX persist { /Common/PROF_PERSIST_COOKIE_SESSION_VSYYY { default yes } } policies { openshift_secure_routes { } } profiles { /Common/ASM_RD_000_WAF_BASIC_OWASP10 { } /Common/PROF_ANALYTICS_HTTP_CUSTOMIZED { } /Common/PROF_HTTP_REDIRECT_HSTS_XFF { } /Common/PROF_TCP_LANOPTIMIZED_KEEPALIVE_120 { context serverside } /Common/PROF_TCP_WANOPTIMIZED_KEEPALIVE_120_NONAGLE { context clientside } /Common/PROF_WAF_BOT_DEFENSE_RELAXED { } /Common/PROF_WAF_DOS_BA_TLSFP_BLOCKING { } /Common/websecurity { } RD_000_VS_STD_XXX_443_tls_server { context clientside } RD_000_VS_STD_XXX_443_tls_server-1- { context clientside } } rules { /Common/RD_000_RULE_SNAT_GENERIC /Common/RD_000_RULE_HTTP_XFORWARDEDHOST openshift_passthrough_irule } security-log-profiles { /Common/PROF_LOG_ASM_ALL_BOT_ALL_DOS_ALL } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { pool /Common/RD_000_SNATPOOL_GENERIC type snat } translate-address enabled translate-port enabled vlans { /Common/RD_000_VLAN_EXT } vlans-enabled }

+++ CIS-config spec: containers:

joebride commented 3 years ago

I missed config of http2-profile modify ltm virtual RD_000_VS_STD_XXX_443 profiles add { /Common/PROF_HTTP2_INSERTHEADER { context clientside } }

mdditt2000 commented 3 years ago

Created another story for fixing this : CONTCNTR-2271

Github reported:

1: https://github.com/F5Networks/k8s-bigip-ctlr/issues/1498

2: https://github.com/F5Networks/k8s-bigip-ctlr/issues/1457

Could able to reproduce the issue now with CIS 2.1.0 version where HTTP/2 traffic is allowed after applying/not-applying http2-profile at the client side of VS.

But on CIS 2.2.0 latest version, observed that HTTP/2 traffic is blocked when http-2 profile is applied at client side of LTM VS. And logs are not observed as traffic is not honored.

This needs a fix. Adding to milestone 2.2.2

joebride commented 3 years ago

@mdditt2000 / @trinaths : I don´t see the fix for this issue in CIS v2.2.2. What is the reason? When it will be fixed?

trinaths commented 3 years ago

@joebride - we have taken this in 2.2.2 to reproduce the issue at our end so that the exact issue is understood. We have created CONTCNTR-2271 for internal tracking of this issue.

trinaths commented 3 years ago

This issue is fixed with CIS v2.3