search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
355 stars 193 forks source link

Cannot reproduce ingresslink example from user guide #1929

Closed x599123 closed 3 years ago

x599123 commented 3 years ago

Setup Details

CIS Version : 2.3.0
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 15.1.3.1
AS3 Version: 3.29.0 Agent Mode: AS3/CCCL
Orchestration: Openshift/OSCP
Orchestration Version: Client Version: openshift-clients-4.5.0-202006231303.p0-4-gb66f2d3a6 Server Version: 4.5.6 Kubernetes Version: v1.18.3+002a51f
Pool Mode: Cluster
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc>

Description

I want to reproduce ingresslink lab,but there is some problem. It can't connect to pool member(pods)

Steps To Reproduce

1)https://clouddocs.f5.com/containers/latest/userguide/openshift/#installing-cis-manually 2)https://clouddocs.f5.com/containers/latest/userguide/ingresslink/

Expected Result

pool member should be green

Actual Result

pool member dead

Diagnostic Information

CIS log

2021/07/29 12:56:29 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.3.0, BuildInfo: azure-65-f3c176bb7132859516810114ec3547b75df7c37a
2021/07/29 12:56:29 [DEBUG] [AS3] No certs appended, using only system certs
2021/07/29 12:56:29 [INFO] ConfigWriter started: 0xc0007e52f0
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) writing section name global
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) successfully wrote section (global)
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) writing section name bigip
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) successfully wrote section (bigip)
2021/07/29 12:56:29 [INFO] Started config driver sub-process at pid: 14
2021/07/29 12:56:29 [DEBUG] Custom Resource Manager Created
2021/07/29 12:56:29 [INFO] [CORE] NodePoller (0xc000524360) registering new listener: 0x1361ea0
2021/07/29 12:56:29 [INFO] [CORE] NodePoller (0xc000524360) registering new listener: 0x1361f20
2021/07/29 12:56:29 [INFO] Posting GET BIGIP AS3 Version request on https://192.168.51.180/mgmt/shared/appsvcs/info
2021/07/29 12:56:29 [INFO] Starting Custom Resource Manager
2021/07/29 12:56:29 [INFO] Starting IngressLink Informer
2021/07/29 12:56:29 [DEBUG] Client Created
2021/07/29 12:56:29 [DEBUG] Creating Informers for Namespace kube-system
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller object created: 0xc000524360
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) caching listener 0x1361ea0, poller is not running
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) caching listener 0x1361f20, poller is not running
2021/07/29 12:56:29 [DEBUG] [VxLAN] Vxlan Manager waiting for pod events from appManager.
I0729 12:56:29.256026       1 shared_informer.go:197] Waiting for caches to sync for F5 CIS CRD Controller
2021/07/29 12:56:29 [INFO] Enqueueing IngressLink: &{{IngressLink cis.f5.com/v1} {vs-ingresslink  kube-system /apis/cis.f5.com/v1/namespaces/kube-system/ingresslinks/vs-ingresslink 051f3319-71bd-4273-8629-134b8b27b422 986210 1 2021-06-03 17:42:27 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"cis.f5.com/v1","kind":"IngressLink","metadata":{"annotations":{},"name":"vs-ingresslink","namespace":"kube-system"},"spec":{"iRules":["/Common/Proxy_Protocol_iRule"],"selector":{"matchLabels":{"app":"nginx-ingress"}},"virtualServerAddress":"192.168.51.181"}}
] [] []  [{kubectl-create Update cis.f5.com/v1 2021-06-03 17:42:27 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 105 82 117 108 101 115 34 58 123 125 44 34 102 58 115 101 108 101 99 116 111 114 34 58 123 34 46 34 58 123 125 44 34 102 58 109 97 116 99 104 76 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 125 125 44 34 102 58 118 105 114 116 117 97 108 83 101 114 118 101 114 65 100 100 114 101 115 115 34 58 123 125 125 125],}} {kubectl-client-side-apply Update cis.f5.com/v1 2021-06-05 11:44:35 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 117 98 101 99 116 108 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 97 115 116 45 97 112 112 108 105 101 100 45 99 111 110 102 105 103 117 114 97 116 105 111 110 34 58 123 125 125 125 125],}}]} {192.168.51.181 &LabelSelector{MatchLabels:map[string]string{app: nginx-ingress,},MatchExpressions:[]LabelSelectorRequirement{},} [/Common/Proxy_Protocol_iRule]}}
I0729 12:56:29.357249       1 shared_informer.go:204] Caches are synced for F5 CIS CRD Controller 
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) registering cached listener: 0x1361ea0

2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) poller goroutine started
2021/07/29 12:56:29 [INFO] [CORE] NodePoller started: (0xc000524360)
2021/07/29 12:56:29 [INFO] Worker got IngressLink: &{{IngressLink cis.f5.com/v1} {vs-ingresslink  kube-system /apis/cis.f5.com/v1/namespaces/kube-system/ingresslinks/vs-ingresslink 051f3319-71bd-4273-8629-134b8b27b422 986210 1 2021-06-03 17:42:27 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"cis.f5.com/v1","kind":"IngressLink","metadata":{"annotations":{},"name":"vs-ingresslink","namespace":"kube-system"},"spec":{"iRules":["/Common/Proxy_Protocol_iRule"],"selector":{"matchLabels":{"app":"nginx-ingress"}},"virtualServerAddress":"192.168.51.181"}}
] [] []  [{kubectl-create Update cis.f5.com/v1 2021-06-03 17:42:27 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 105 82 117 108 101 115 34 58 123 125 44 34 102 58 115 101 108 101 99 116 111 114 34 58 123 34 46 34 58 123 125 44 34 102 58 109 97 116 99 104 76 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 125 125 44 34 102 58 118 105 114 116 117 97 108 83 101 114 118 101 114 65 100 100 114 101 115 115 34 58 123 125 125 125],}} {kubectl-client-side-apply Update cis.f5.com/v1 2021-06-05 11:44:35 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 117 98 101 99 116 108 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 97 115 116 45 97 112 112 108 105 101 100 45 99 111 110 102 105 103 117 114 97 116 105 111 110 34 58 123 125 125 125 125],}}]} {192.168.51.181 &LabelSelector{MatchLabels:map[string]string{app: nginx-ingress,},MatchExpressions:[]LabelSelectorRequirement{},} [/Common/Proxy_Protocol_iRule]}}

2021/07/29 12:56:29 [INFO] IngressLink Selector: &LabelSelector{MatchLabels:map[string]string{app: nginx-ingress,},MatchExpressions:[]LabelSelectorRequirement{},}

2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) poller goroutine adding listener: {l:0xc0000467e0 s:0xc000046840}
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) registering cached listener: 0x1361f20

2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener goroutine started: 0x1361ea0
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener callback - num items: 8 err: <nil>
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener add wake up - next poll in 29.999901988s

2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) poller goroutine adding listener: {l:0xc000100d80 s:0xc000100e40}
2021/07/29 12:56:29 [DEBUG] Starting Custom Resource Worker
2021/07/29 12:56:29 [DEBUG] Processing Key: &{kube-system IngressLink vs-ingresslink 0xc00055f600 false}
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener goroutine started: 0x1361f20
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener callback - num items: 8 err: <nil>
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) writing section name vxlan-fdb
2021/07/29 12:56:29 [DEBUG] [CORE] NodePoller (0xc000524360) listener add wake up - next poll in 29.997233972s

2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) successfully wrote section (vxlan-fdb)
2021/07/29 12:56:29 [DEBUG] [VxLAN] Vxlan manager (okd-tunnel) wrote config section: [{0a:0a:0a:ff:53:36 10.255.83.54} {0a:0a:0a:ff:53:37 10.255.83.55} {0a:0a:0a:ff:53:5d 10.255.83.93} {0a:0a:0a:ff:53:5e 10.255.83.94} {0a:0a:0a:ff:53:5f 10.255.83.95} {0a:0a:0a:ff:53:33 10.255.83.51} {0a:0a:0a:ff:53:34 10.255.83.52} {0a:0a:0a:ff:53:35 10.255.83.53}]
2021/07/29 12:56:29 [DEBUG] Found endpoints for backend kube-system/nginx-ingress-ingresslink: [{10.128.2.12 80 0 user-enabled} {10.131.0.22 80 0 user-enabled} {10.131.0.26 80 0 user-enabled}]
2021/07/29 12:56:29 [DEBUG] Found endpoints for backend kube-system/nginx-ingress-ingresslink: []
2021/07/29 12:56:29 [DEBUG] Found endpoints for backend kube-system/nginx-ingress-ingresslink: []
2021/07/29 12:56:29 [DEBUG] Found endpoints for backend kube-system/nginx-ingress-ingresslink: [{10.128.2.12 443 0 user-enabled} {10.131.0.22 443 0 user-enabled} {10.131.0.26 443 0 user-enabled}]
2021/07/29 12:56:29 [DEBUG] Finished syncing Ingress Links &{TypeMeta:{Kind:IngressLink APIVersion:cis.f5.com/v1} ObjectMeta:{Name:vs-ingresslink GenerateName: Namespace:kube-system SelfLink:/apis/cis.f5.com/v1/namespaces/kube-system/ingresslinks/vs-ingresslink UID:051f3319-71bd-4273-8629-134b8b27b422 ResourceVersion:986210 Generation:1 CreationTimestamp:2021-06-03 17:42:27 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"cis.f5.com/v1","kind":"IngressLink","metadata":{"annotations":{},"name":"vs-ingresslink","namespace":"kube-system"},"spec":{"iRules":["/Common/Proxy_Protocol_iRule"],"selector":{"matchLabels":{"app":"nginx-ingress"}},"virtualServerAddress":"192.168.51.181"}}
] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:kubectl-create Operation:Update APIVersion:cis.f5.com/v1 Time:2021-06-03 17:42:27 +0000 UTC FieldsType:FieldsV1 FieldsV1:&FieldsV1{Raw:*[123 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 105 82 117 108 101 115 34 58 123 125 44 34 102 58 115 101 108 101 99 116 111 114 34 58 123 34 46 34 58 123 125 44 34 102 58 109 97 116 99 104 76 97 98 101 108 115 34 58 123 34 46 34 58 123 125 44 34 102 58 97 112 112 34 58 123 125 125 125 44 34 102 58 118 105 114 116 117 97 108 83 101 114 118 101 114 65 100 100 114 101 115 115 34 58 123 125 125 125],}} {Manager:kubectl-client-side-apply Operation:Update APIVersion:cis.f5.com/v1 Time:2021-06-05 11:44:35 +0000 UTC FieldsType:FieldsV1 FieldsV1:&FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 117 98 101 99 116 108 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 97 115 116 45 97 112 112 108 105 101 100 45 99 111 110 102 105 103 117 114 97 116 105 111 110 34 58 123 125 125 125 125],}}]} Spec:{VirtualServerAddress:192.168.51.181 Selector:&LabelSelector{MatchLabels:map[string]string{app: nginx-ingress,},MatchExpressions:[]LabelSelectorRequirement{},} IRules:[/Common/Proxy_Protocol_iRule]}} (5.876821ms)
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) writing section name gtm
2021/07/29 12:56:29 [DEBUG] [CCCL] ConfigWriter (0xc0007e52f0) successfully wrote section (gtm)
2021/07/29 12:56:29 [DEBUG] Wrote gtm config section: map[]
2021/07/29 12:56:29 [DEBUG] [AS3] PostManager Accepted the configuration
2021/07/29 12:56:29 [DEBUG] Custom Resource Manager wrote endpoints to VxlanMgr
2021/07/29 12:56:29 [DEBUG] [AS3] posting request to https://192.168.51.180/mgmt/shared/appsvcs/declare/
2021/07/29 12:56:29 [ERROR] [VxLAN] Vxlan manager could not get VtepMac for 10.128.2.12's node.
2021/07/29 12:56:30 [INFO] BIGIP is serving with AS3 version : 3.29.0-3 
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,452 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/sys/ AND suffix:  AND kwargs: {}
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,455 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): 192.168.51.180:443
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,768 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "POST /mgmt/shared/authn/login HTTP/1.1" 200 723
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,771 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): 192.168.51.180:443
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,831 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/sys/ HTTP/1.1" 200 4048
2021/07/29 12:56:30 [INFO] Text: '{"kind":"tm:sys:syscollectionstate","selfLink":"https://localhost/mgmt/tm/sys?ver=15.1.3.1","items":[{"reference":{"link":"https://localhost/mgmt/tm/sys/application?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/crypto?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-log-settings?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/diags?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/disk?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dynad?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ecm?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/file?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/fpga?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icall?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ipfix?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-config?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/pfman?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sflow?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/software?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/turboflex?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/url-db?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/aom?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/autoscale-group?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/cluster?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/config?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/core?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-ha?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/datastor?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/db?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dns?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/feature-module?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/folder?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/global-settings?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ha-group?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/httpd?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icontrol-soap?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/internal-proxy?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-rotate?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-dhcp?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ip?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ovsdb?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-proxy-config?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-route?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ntp?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/outbound-smtp?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/provision?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/scriptd?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/service?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/smtp-server?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/snmp?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sshd?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/state-mirroring?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/syslog?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/telemd?ver=15.1.3.1"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ucs?ver=15.1.3.1"}}]}'
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,832 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,853 f5_cccl DEBUG] F5CloudServiceManager initialize
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,853 f5_cccl.bigip DEBUG] BigIPProxy.__init__()
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,871 __main__ DEBUG] config handler thread start
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,871 __main__ DEBUG] config handler woken for reset
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,872 __main__ DEBUG] loaded configuration file successfully
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,872 __main__ DEBUG] NET Config: {"userFdbTunnels": [{"name": "okd-tunnel", "records": [{"name": "0a:0a:0a:ff:53:36", "endpoint": "10.255.83.54"}, {"name": "0a:0a:0a:ff:53:37", "endpoint": "10.255.83.55"}, {"name": "0a:0a:0a:ff:53:5d", "endpoint": "10.255.83.93"}, {"name": "0a:0a:0a:ff:53:5e", "endpoint": "10.255.83.94"}, {"name": "0a:0a:0a:ff:53:5f", "endpoint": "10.255.83.95"}, {"name": "0a:0a:0a:ff:53:33", "endpoint": "10.255.83.51"}, {"name": "0a:0a:0a:ff:53:34", "endpoint": "10.255.83.52"}, {"name": "0a:0a:0a:ff:53:35", "endpoint": "10.255.83.53"}]}]}
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,872 f5_cccl.service.manager DEBUG] apply_net_config start
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,872 f5_cccl.service.validation DEBUG] Validating desired config against CCCL API schema.
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,872 f5_cccl.service.validation DEBUG] validate start
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,873 f5_cccl.service.validation DEBUG] validate took 0.00081 seconds.
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,874 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/auth/partition/Common AND suffix:  AND kwargs: {}
2021/07/29 12:56:30 [INFO] [2021-07-29 12:56:30,880 __main__ INFO] entering inotify loop to watch /tmp/k8s-bigip-ctlr.config688999502/config.json
2021/07/29 12:56:30 [INFO] Text: '{"kind":"tm:auth:partition:partitionstate","name":"Common","fullPath":"Common","generation":0,"selfLink":"https://localhost/mgmt/tm/auth/partition/Common?ver=15.1.3.1","defaultRouteDomain":0,"description":"Repository for system objects and shared objects."}'
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,981 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/auth/partition/Common HTTP/1.1" 200 257
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,995 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,996 f5_cccl.bigip DEBUG] Refreshing the BIG-IP net cached state...
2021/07/29 12:56:30 [DEBUG] [2021-07-29 12:56:30,997 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/auth/partition/Common AND suffix:  AND kwargs: {}
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,020 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/auth/partition/Common HTTP/1.1" 200 257
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:auth:partition:partitionstate","name":"Common","fullPath":"Common","generation":0,"selfLink":"https://localhost/mgmt/tm/auth/partition/Common?ver=15.1.3.1","defaultRouteDomain":0,"description":"Repository for system objects and shared objects."}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,021 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,022 f5_cccl.bigip DEBUG] Retrieving arps from BIG-IP /Common...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,022 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/net/arp/ AND suffix:  AND kwargs: {'params': '$filter=partition+eq+Common'}
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:net:arp:arpcollectionstate","selfLink":"https://localhost/mgmt/tm/net/arp?$filter=partition+eq+Common&ver=15.1.3.1","items":[]}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,048 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/net/arp/?$filter=partition+eq+Common HTTP/1.1" 200 139
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,049 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,049 f5_cccl.bigip DEBUG] Retrieving fdb tunnels from BIG-IP /Common...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,049 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/net/fdb/tunnel/ AND suffix:  AND kwargs: {}
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:net:fdb:tunnel:tunnelcollectionstate","selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel?ver=15.1.3.1","items":[{"kind":"tm:net:fdb:tunnel:tunnelstate","name":"http-tunnel","partition":"Common","fullPath":"/Common/http-tunnel","generation":1,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel?ver=15.1.3.1","recordsReference":{"link":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel/records?ver=15.1.3.1","isSubcollection":true}},{"kind":"tm:net:fdb:tunnel:tunnelstate","name":"okd-tunnel","partition":"Common","fullPath":"/Common/okd-tunnel","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel?ver=15.1.3.1","recordsReference":{"link":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records?ver=15.1.3.1","isSubcollection":true}},{"kind":"tm:net:fdb:tunnel:tunnelstate","name":"socks-tunnel","partition":"Common","fullPath":"/Common/socks-tunnel","generation":1,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel?ver=15.1.3.1","recordsReference":{"link":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel/records?ver=15.1.3.1","isSubcollection":true}}]}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,082 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/net/fdb/tunnel/ HTTP/1.1" 200 1188
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,083 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,085 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel/records/ AND suffix:  AND kwargs: {}
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:net:fdb:tunnel:records:recordscollectionstate","selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel/records?ver=15.1.3.1","items":[]}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,108 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/net/fdb/tunnel/~Common~http-tunnel/records/ HTTP/1.1" 200 165
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,108 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,109 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/ AND suffix:  AND kwargs: {}
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,123 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/ HTTP/1.1" 200 2283
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,124 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:net:fdb:tunnel:records:recordscollectionstate","selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records?ver=15.1.3.1","items":[{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:5d","fullPath":"0a:0a:0a:ff:53:5d","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:5d?ver=15.1.3.1","endpoint":"10.255.83.93%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:5e","fullPath":"0a:0a:0a:ff:53:5e","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:5e?ver=15.1.3.1","endpoint":"10.255.83.94%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:5f","fullPath":"0a:0a:0a:ff:53:5f","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:5f?ver=15.1.3.1","endpoint":"10.255.83.95%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:33","fullPath":"0a:0a:0a:ff:53:33","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:33?ver=15.1.3.1","endpoint":"10.255.83.51%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:34","fullPath":"0a:0a:0a:ff:53:34","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:34?ver=15.1.3.1","endpoint":"10.255.83.52%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:35","fullPath":"0a:0a:0a:ff:53:35","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:35?ver=15.1.3.1","endpoint":"10.255.83.53%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:36","fullPath":"0a:0a:0a:ff:53:36","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:36?ver=15.1.3.1","endpoint":"10.255.83.54%0"},{"kind":"tm:net:fdb:tunnel:records:recordsstate","name":"0a:0a:0a:ff:53:37","fullPath":"0a:0a:0a:ff:53:37","generation":107,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~okd-tunnel/records/0a:0a:0a:ff:53:37?ver=15.1.3.1","endpoint":"10.255.83.55%0"}]}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,126 icontrol.session DEBUG] get WITH uri: https://192.168.51.180:443/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel/records/ AND suffix:  AND kwargs: {}
2021/07/29 12:56:31 [INFO] Text: '{"kind":"tm:net:fdb:tunnel:records:recordscollectionstate","selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel/records?ver=15.1.3.1","items":[]}'
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,139 urllib3.connectionpool DEBUG] https://192.168.51.180:443 "GET /mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel/records/ HTTP/1.1" 200 166
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,140 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json; charset=UTF-8 Content-Encoding: None
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,140 f5_cccl.bigip DEBUG] BIG-IP net refresh took 0.14397 seconds.
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,140 f5_cccl.service.manager DEBUG] Getting arp tasks...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,140 f5_cccl.service.manager DEBUG] Getting tunnel tasks...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,140 f5_cccl.service.manager DEBUG] Getting pre-existing tunnel update tasks...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,141 f5_cccl.service.manager DEBUG] Building task lists...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,141 f5_cccl.service.manager DEBUG] Service task queue length: 0
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,141 f5_cccl.service.manager DEBUG] Creating 0 resources...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,141 f5_cccl.service.manager DEBUG] Updating 0 resources...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,141 f5_cccl.service.manager DEBUG] Deleting 0 resources...
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,142 f5_cccl.service.manager DEBUG] apply_net_config took 0.26913 seconds.
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,142 __main__ DEBUG] loaded configuration file successfully
2021/07/29 12:56:31 [DEBUG] [2021-07-29 12:56:31,143 __main__ DEBUG] updating tasks finished, took 0.27157044410705566 seconds
2021/07/29 12:56:32 [DEBUG] [AS3] Response from BIG-IP: code: 200 --- tenant:okd --- message: no change

<Configuration files, error messages, logs>

#CIS_deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-bigip-ctlr
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k8s-bigip-ctlr
  template:
    metadata:
      name: k8s-bigip-ctlr
      labels:
        app: k8s-bigip-ctlr
    spec:
      serviceAccountName: k8s-bigip-ctlr
      containers:
        - name: k8s-bigip-ctlr
          image: "f5networks/k8s-bigip-ctlr:2.3.0"
          imagePullPolicy: IfNotPresent
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=https://192.168.51.180",
            "--insecure",
            "--bigip-partition=okd",
            "--pool-member-type=cluster",
            "--openshift-sdn-name=/Common/okd-tunnel",
            "--custom-resource-mode=true",
            "--ingress-link-mode=true",
            "--namespace=kube-system",
            "--log-level=DEBUG",
            "--log-as3-response=true"
          ]
#openshift_node_deploy.yaml
apiVersion: v1
kind: HostSubnet
metadata:
  name: openshfit-f5-node
  annotations:
    pod.network.openshift.io/fixed-vnid-host: "0"
    pod.network.openshift.io/assign-subnet: "true"
# provide a name for the node that will serve as BIG-IP's entry into the cluster
host: openshfit-f5-node
# The hostIP address will be the BIG-IP interface address routable to the
# OpenShift Origin nodes.
# This address is the BIG-IP VTEP in the SDN's VXLAN.
hostIP: 192.168.51.180
subnet: "10.131.0.0/23"
#nginx_deploy.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: NginxIngressController
metadata:
  name: my-nginx-ingress-controller
  namespace: default
spec:
  type: deployment
  image:
    repository: nginx/nginx-ingress
    tag: 1.12.0
    pullPolicy: Always
  serviceType: NodePort
  nginxPlus: False
nginx_ingress_controller_service_deploy.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-ingresslink
  namespace: kube-system
  labels:
    app: nginx-ingress
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
    - port: 443
      targetPort: 443
      protocol: TCP
      name: https
  selector:
    app: nginx-ingress-controller
  type: NodePort
#ingresslink_deploy.yaml
apiVersion: "cis.f5.com/v1"
kind: IngressLink
metadata:
  name: vs-ingresslink
  namespace: kube-system
spec:
  virtualServerAddress: "192.168.51.181"
  iRules:
    - /Common/Proxy_Protocol_iRule
  selector:
    matchLabels:
      app: nginx-ingress
net tunnels tunnel okd-tunnel {
    app-service none
    auto-lasthop default
    description none
    idle-timeout 300
    if-index 160
    key 1
    local-address 192.168.51.180
    mode bidirectional
    mtu 0
    partition Common
    profile okd-vxlan
    remote-address any
    secondary-address any
    tos preserve
    traffic-group none
    transparent disabled
    use-pmtu enabled
}

net fdb tunnel okd-tunnel {
    records {
        0a:0a:0a:ff:53:5d {
            endpoint 10.255.83.93%0
        }
        0a:0a:0a:ff:53:5e {
            endpoint 10.255.83.94%0
        }
        0a:0a:0a:ff:53:5f {
            endpoint 10.255.83.95%0
        }
        0a:0a:0a:ff:53:33 {
            endpoint 10.255.83.51%0
        }
        0a:0a:0a:ff:53:34 {
            endpoint 10.255.83.52%0
        }
        0a:0a:0a:ff:53:35 {
            endpoint 10.255.83.53%0
        }
        0a:0a:0a:ff:53:36 {
            endpoint 10.255.83.54%0
        }
        0a:0a:0a:ff:53:37 {
            endpoint 10.255.83.55%0
        }
    }
}

net self okd-tunnel_sip {
    address 10.128.8.0/14
    allow-service all
    traffic-group traffic-group-local-only
    vlan okd-tunnel
}
net self aaa {
    address 192.168.51.180/24
    allow-service all
    traffic-group traffic-group-local-only
    vlan aaa
}
[root@localhost:Active:Standalone] config # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.51.253  0.0.0.0         UG    0      0        0 aaa
10.128.0.0      0.0.0.0         255.252.0.0     U     0      0        0 okd-tunnel
10.131.0.0      0.0.0.0         255.255.0.0     U     0      0        0 okd-tunnel
127.1.1.0       0.0.0.0         255.255.255.0   U     0      0        0 tmm
127.7.0.0       tmm-shared      255.255.0.0     UG    0      0        0 tmm
127.20.0.0      0.0.0.0         255.255.0.0     U     0      0        0 tmm_bp
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 mgmt
192.168.51.0    0.0.0.0         255.255.255.0   U     0      0        0 aaa

Observations (if any)

圖片 圖片

trinaths commented 3 years ago

@x599123 - IngressLink is not supported on OCP yet. Please try the same on k8s. Closing this issue.