search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
355 stars 193 forks source link

Unable to update the FDB entries on BIG IP 16 using F5 SDK #1488

Closed chen23 closed 3 years ago

chen23 commented 4 years ago

Setup Details

CIS Version : 2.1.1 Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 16
AS3 Version: 3.21 Agent Mode: AS3/CCCL
Orchestration: OSCP 4.3 Orchestration Version:
Pool Mode: Cluster/Nodeport
Additional Setup details: Using OVS

Description

When you use BIG-IP 16.0.0 you get an error when trying to make fdb updates. This results in not being able to connect via VXLAN.

Steps To Reproduce

Deploy CIS 2.1.1 in OCP 4.3 and try and use BIG-IP 16

Expected Result

It works

Actual Result

/
2020/08/28 18:57:28 [INFO] Text: '{"code":400,"message":"Version 11.5.0 is not supported.","referer":"10.1.1.9","restOperationId":1310282,"kind":":rest
2020/08/28 18:57:28 [ERROR] [2020-08-28 18:57:28,319 __main__ ERROR] Unexpected error
2020/08/28 18:57:28 [ERROR] [2020-08-28 18:57:28,319 __main__ ERROR] Error applying config, will try again in 16 seconds
2020/08/28 18:57:44 [INFO] Traceback (most recent call last):
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 325, in _do_reset
2020/08/28 18:57:44 [INFO]     incomplete = self._update_cccl(config)
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 397, in _update_cccl
2020/08/28 18:57:44 [INFO]     incomplete += mgr._apply_net_config(cfg_net)
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 121, in _apply_net_config
2020/08/28 18:57:44 [INFO]     return self._cccl.apply_net_config(config)
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-cccl/f5_cccl/api.py", line 102, in apply_net_config
2020/08/28 18:57:44 [INFO]     return self._service_manager.apply_net_config(services)
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-cccl/f5_cccl/service/manager.py", line 705, in apply_net_config
2020/08/28 18:57:44 [INFO]     retval = self._service_deployer.deploy_net(desired_config)
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-cccl/f5_cccl/service/manager.py", line 470, in deploy_net
2020/08/28 18:57:44 [INFO]     self._bigip.refresh_net()
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-cccl/f5_cccl/bigip.py", line 146, in refresh_net
2020/08/28 18:57:44 [INFO]     self._refresh_net()
2020/08/28 18:57:44 [INFO]   File "/app/src/f5-cccl/f5_cccl/bigip.py", line 406, in _refresh_net
2020/08/28 18:57:44 [INFO]     tunnels = self._bigip.tm.net.fdb.tunnels.get_collection()
2020/08/28 18:57:44 [INFO]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 781, in get_collection
2020/08/28 18:57:44 [INFO]     self.refresh(**kwargs)
2020/08/28 18:57:44 [INFO]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 651, in refresh
2020/08/28 18:57:44 [INFO]     self._refresh(**kwargs)
2020/08/28 18:57:44 [INFO]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 634, in _refresh
2020/08/28 18:57:44 [INFO]     response = refresh_session.get(uri, **requests_params)
2020/08/28 18:57:44 [INFO]   File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/icontrol/session.py", line 271, in wrapper
2020/08/28 18:57:44 [INFO]     raise iControlUnexpectedHTTPError(error_message, response=response)
2020/08/28 18:57:44 [INFO] icontrol.exceptions.iControlUnexpectedHTTPError: 400 Unexpected Error: Bad Request for uri: https://10.1.20.240:443/mgmt/tm/
2020/08/28 18:57:44 [INFO] Text: '{"code":400,"message":"Version 11.5.0 is not supported.","referer":"10.1.1.9","restOperationId":1428170,"kind":":rest
2020/08/28 18:57:44 [ERROR] [2020-08-28 18:57:44,458 __main__ ERROR] Unexpected error
2020/08/28 18:57:44 [ERROR] [2020-08-28 18:57:44,459 __main__ ERROR] Error applying config, will try again in 32 seconds

comparing 16.0.0 to 12.1.x

16:

curl https://10.1.20.240:443/mgmt/tm/net/fdb/tunnel/?ver=11.5.0 -u admin:admin -k
{"code":400,"message":"Version 11.5.0 is not supported.","referer":"10.1.1.4","restOperationId":5506285,"kind":":resterrorresponse"}

12.1.x

~$ curl https://192.168.122.58:443/mgmt/tm/net/fdb/tunnel/?ver=11.5.0 -u admin:admin -k
{"kind":"tm:net:fdb:tunnel:tunnelcollectionstate","selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel?ver=11.5.0","items":[{"kind":"tm:net:fdb:tunnel:tunnelstate","name":"http-tunnel","partition":"Common","fullPath":"/Common/http-tunnel","generation":1,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel?ver=11.5.0"},{"kind":"tm:net:fdb:tunnel:tunnelstate","name":"socks-tunnel","partition":"Common","fullPath":"/Common/socks-tunnel","generation":1,"selfLink":"https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel?ver=11.5.0"}]}
mdditt2000 commented 4 years ago

Jira CONTCNTR-2034 for PM tracking. CIS planning to validate BIG-IP for the CIS 2.2 release. This item is one of the acceptance criterias

mdditt2000 commented 4 years ago

@vklohiya started validation of BIGv16 for standalone.

ARP Validated.

Able to create static arp using F5-SDK -->.

bigip._conn.tm.net.arps.arp.create(partition="test", name="s1", ipAddress="10.244.1.252", macAddress="16:eb:5e:6a:67:17")

mdditt2000 commented 3 years ago

Workaround solution:

bigip._conn.tm.net.fdb.tunnels.raw['_meta_data']['icontrol_version']=bigip._conn.tmos_version

chen23 commented 3 years ago

@mdditt2000 there's some overhead with specifying a schema (at least in older versions?). I would recommend omitting a specific version string or check for the version and handle different responses (if any exist)

mdditt2000 commented 3 years ago

Will check @vklohiya on the proposed solution tonight and update you. Most likely updating the schema version in CIS

vklohiya commented 3 years ago

@chen23 , We are not putting any version check we are just updating the icontrol version to the bigip version which is fetched while creating the connection, There is no conditional version checks added for fix. Please use the following image for testing with TMOS 16: vklohiya1/k8s-bigip-ctlr:big16-7

mdditt2000 commented 3 years ago

@chen23 On further debugging found that F5 SDK is not able to get all the FDB records under tunnel object. @vklohiya will be submitting a BZ

mdditt2000 commented 3 years ago

@chen23 here is the BZ 953885. CIS is blocked due to SDK

chen23 commented 3 years ago

I believe this is related to the version of the f5-sdk that is pinned for cccl.
https://github.com/f5devcentral/f5-cccl/blob/master/setup_requirements.txt#L3

https://github.com/F5Networks/f5-common-python/blob/19f3a3d1b5a1f7eb99b8b034f876c7109bd8bbd4/f5/bigip/tm/net/fdb.py#L36

It looks like newer versions of the f5-sdk do not lock to 11.5.0: https://github.com/F5Networks/f5-common-python/blob/development/f5/bigip/tm/net/fdb.py#L41

chen23 commented 3 years ago

If you access iControl REST directly w/out CCCL you can see the response. @mdditt2000 curl https://10.1.20.240:443/mgmt/tm/net/fdb/tunnel -u admin:admin -k|jq .

$ curl https://10.1.20.240:443/mgmt/tm/net/fdb/tunnel -u admin:admin -k|jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1585  100  1585    0     0  19812      0 --:--:-- --:--:-- --:--:-- 19812
{
  "kind": "tm:net:fdb:tunnel:tunnelcollectionstate",
  "selfLink": "https://localhost/mgmt/tm/net/fdb/tunnel?ver=16.0.0",
  "items": [
    {
      "kind": "tm:net:fdb:tunnel:tunnelstate",
      "name": "http-tunnel",
      "partition": "Common",
      "fullPath": "/Common/http-tunnel",
      "generation": 1,
      "selfLink": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel?ver=16.0.0",
      "recordsReference": {
        "link": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~http-tunnel/records?ver=16.0.0",
        "isSubcollection": true
      }
    },
    {
      "kind": "tm:net:fdb:tunnel:tunnelstate",
      "name": "openshift-connectivity",
      "partition": "Common",
      "fullPath": "/Common/openshift-connectivity",
      "generation": 1,
      "selfLink": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~openshift-connectivity?ver=16.0.0",
      "recordsReference": {
        "link": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~openshift-connectivity/records?ver=16.0.0",
        "isSubcollection": true
      }
    },
    {
      "kind": "tm:net:fdb:tunnel:tunnelstate",
      "name": "openshift_vxlan",
      "partition": "Common",
      "fullPath": "/Common/openshift_vxlan",
      "generation": 1,
      "selfLink": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~openshift_vxlan?ver=16.0.0",
      "recordsReference": {
        "link": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~openshift_vxlan/records?ver=16.0.0",
        "isSubcollection": true
      }
    },
    {
      "kind": "tm:net:fdb:tunnel:tunnelstate",
      "name": "socks-tunnel",
      "partition": "Common",
      "fullPath": "/Common/socks-tunnel",
      "generation": 1,
      "selfLink": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel?ver=16.0.0",
      "recordsReference": {
        "link": "https://localhost/mgmt/tm/net/fdb/tunnel/~Common~socks-tunnel/records?ver=16.0.0",
        "isSubcollection": true
      }
    }
  ]
}
pfischer8989 commented 3 years ago

Any progress on this one? I am seeing this with CIS 2.2 as well.

2020/11/06 21:18:03 [DEBUG] [2020-11-06 21:18:03,624 urllib3.connectionpool DEBUG] https://x.x.x.x:443 "GET /mgmt/tm/net/fdb/tunnel/?ver=11.5.0 HTTP/1.1" 400 138 2020/11/06 21:18:03 [INFO] Traceback (most recent call last): 2020/11/06 21:18:03 [INFO] File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 325, in _do_reset 2020/11/06 21:18:03 [INFO] incomplete = self._update_cccl(config) 2020/11/06 21:18:03 [INFO] File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 397, in _update_cccl 2020/11/06 21:18:03 [INFO] incomplete += mgr._apply_net_config(cfg_net) 2020/11/06 21:18:03 [INFO] File "/app/src/f5-ctlr-agent/f5_ctlr_agent/bigipconfigdriver.py", line 121, in _apply_net_config 2020/11/06 21:18:03 [INFO] return self._cccl.apply_net_config(config) 2020/11/06 21:18:03 [INFO] File "/app/src/f5-cccl/f5_cccl/api.py", line 102, in apply_net_config 2020/11/06 21:18:03 [INFO] return self._service_manager.apply_net_config(services) 2020/11/06 21:18:03 [INFO] File "/app/src/f5-cccl/f5_cccl/service/manager.py", line 705, in apply_net_config 2020/11/06 21:18:03 [INFO] retval = self._service_deployer.deploy_net(desired_config) 2020/11/06 21:18:03 [INFO] File "/app/src/f5-cccl/f5_cccl/service/manager.py", line 470, in deploy_net 2020/11/06 21:18:03 [INFO] self._bigip.refresh_net() 2020/11/06 21:18:03 [INFO] File "/app/src/f5-cccl/f5_cccl/bigip.py", line 146, in refresh_net 2020/11/06 21:18:03 [INFO] self._refresh_net() 2020/11/06 21:18:03 [INFO] File "/app/src/f5-cccl/f5_cccl/bigip.py", line 406, in _refresh_net 2020/11/06 21:18:03 [INFO] tunnels = self._bigip.tm.net.fdb.tunnels.get_collection() 2020/11/06 21:18:03 [INFO] File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 781, in get_collection 2020/11/06 21:18:03 [INFO] self.refresh(kwargs) 2020/11/06 21:18:03 [INFO] File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 651, in refresh 2020/11/06 21:18:03 [INFO] self._refresh(kwargs) 2020/11/06 21:18:03 [INFO] File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/f5/bigip/resource.py", line 634, in _refresh 2020/11/06 21:18:03 [INFO] response = refresh_session.get(uri, **requests_params) 2020/11/06 21:18:03 [INFO] File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/icontrol/session.py", line 271, in wrapper 2020/11/06 21:18:03 [INFO] raise iControlUnexpectedHTTPError(error_message, response=response) 2020/11/06 21:18:03 [INFO] icontrol.exceptions.iControlUnexpectedHTTPError: 400 Unexpected Error: Bad Request for uri: https://35.193.230.48:443/mgmt/tm/net/fdb/tunnel/?ver=11.5.0 2020/11/06 21:18:03 [INFO] Text: '{"code":400,"message":"Version 11.5.0 is not supported.","referer":"x.x.x.x","restOperationId":6593537,"kind":":resterrorresponse"}' 2020/11/06 21:18:03 [ERROR] [2020-11-06 21:18:03,626 main ERROR] Unexpected error

mdditt2000 commented 3 years ago

@pfischer8989 please can you open a SR. Here is the Bugzilla – Bug 953885 for reference. I just asked the CORE BIG-IP folks for a update. -- | --

mdditt2000 commented 3 years ago

Merge request https://github.com/f5devcentral/f5-cccl/pull/259/files

mdditt2000 commented 3 years ago

This issue has been resolved in CIS 2.2.1 and validated in CIS 2.2.2

mdditt2000 commented 3 years ago

Closing issue. CIS 2.2.2 is released