search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
357 stars 195 forks source link

Authorization errors and unexpected 503 HTTP return code inside F5 BIGIP controller version 2.16-WIP #3335

Closed wheestermans31 closed 5 months ago

wheestermans31 commented 6 months ago

Setup Details

CIS Version : 2.16-WIP, version I received for testing new persistence arguments inside VirtualServer CRD
Build: f5networks/k8s-bigip-ctlr:2.156-WIP (test one) BIGIP Version: Big IP 14.1.4.6 AS3 Version: 3.24.0
Agent Mode: AS3/CCCL
Orchestration: K8S OVN
Orchestration Version: OpenShift Pool Mode: Cluster Additional Setup details: OVNKubernetes networking

Description

When the BIGIP controller is running, the F5 configuration is updated, but after some time I see auhtorization issues

2024/03/12 19:48:25 [ERROR] [2024-03-12 19:48:25,107 f5_cccl.bigip ERROR] F5 SDK Error: 401 Unexpected Error: F5 Authorization Required for uri: [https://150.45.88.251:443/mgmt/tm/auth/partition/AS3-OpenShift-ocp-behoi-01](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F150.45.88.251%2Fmgmt%2Ftm%2Fauth%2Fpartition%2FAS3-OpenShift-ocp-behoi-01&data=05%7C02%7Cwalter.heestermans%40external.toyota-europe.com%7C59c54a3652d34e4793b508dc4342b813%7C52b742d13dc247acbf03609c83d9df9f%7C0%7C0%7C638459201073064736%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=XLUG%2Bs7EjsZaAlHvLJVwaK1t98ODXdw2neg7J%2BiuwBI%3D&reserved=0) Text: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn\'t understand how to supply\nthe credentials required.</p>\n</body></html>\n'
2024/03/12 19:48:25 [ERROR] [2024-03-12 19:48:25,107 __main__ ERROR] CCCL Error: The requested partition AS3-OpenShift-ocp-behoi-01 was not found.
2024/03/12 19:48:25 [ERROR] [2024-03-12 19:48:25,107 __main__ ERROR] Error applying config, will try again in 1 seconds
2024/03/12 19:48:25 [ERROR] [2024-03-12 19:48:25,113 f5_cccl.bigip ERROR] F5 SDK Error: 401 Unexpected Error: F5 Authorization Required for uri: [https://150.45.88.251:443/mgmt/tm/auth/partition/AS3-OpenShift-ocp-behoi-01](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F150.45.88.251%2Fmgmt%2Ftm%2Fauth%2Fpartition%2FAS3-OpenShift-ocp-behoi-01&data=05%7C02%7Cwalter.heestermans%40external.toyota-europe.com%7C59c54a3652d34e4793b508dc4342b813%7C52b742d13dc247acbf03609c83d9df9f%7C0%7C0%7C638459201073074551%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Jj8mgS7dpZoDDBM3egkQCmA61YYrisr0CLZCuk7Ori8%3D&reserved=0) Text: '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn\'t understand how to supply\nthe credentials required.</p>\n</body></html>\n'
2024/03/12 19:48:25 [ERROR] [2024-03-12 19:48:25,113 __main__ ERROR] CCCL Error: The requested partition AS3-OpenShift-ocp-behoi-01 was not found.

But the configuration updates are still working.

I also see regularly 503 HTTP codes, like this:

2024/03/13 10:18:12 [DEBUG] [2024-03-13 10:18:12,916 urllib3.connectionpool DEBUG] https://150.45.88.251:443 "GET /mgmt/tm/auth/partition/AS3-OpenShift-ocp-behoi-01 HTTP/1.1" 503 108
2024/03/13 10:18:12 [DEBUG] [2024-03-13 10:18:12,917 icontrol.session DEBUG] RESPONSE::STATUS: 503 Content-Type: application/json; charset=UTF-8 Content-Encoding: None Text: '{"code":503,"message":"There is an active asynchronous task executing.","errorStack":[],"apiError":32964609}'
2024/03/13 10:18:12 [ERROR] [2024-03-13 10:18:12,917 f5_cccl.bigip ERROR] F5 SDK Error: 503 Unexpected Error: Service Unavailable for uri: https://150.45.88.251:443/mgmt/tm/auth/partition/AS3-OpenShift-ocp-behoi-01 Text: '{"code":503,"message":"There is an active asynchronous task executing.","errorStack":[],"apiError":32964609}'
2024/03/13 10:18:12 [ERROR] [2024-03-13 10:18:12,917 __main__ ERROR] CCCL Error: The requested partition AS3-OpenShift-ocp-behoi-01 was not found.

But as said the updates are being pushed to the F5, even with these errors.

This is all happening inside out production F5 environment. I have a similar test environment, where I don't see none of these. The only difference is the BIGIP version 1.15.1.10 and AS version 3.49.0.

I'm currently running a log in debug mode until I have captured all errors. Attached already a log where you can see some 503 errors, but not yet the Authorization issue.

[Uploading bigip.log…]()

wheestermans31 commented 6 months ago

Here logs with the 401 error f5.zip

wheestermans31 commented 6 months ago

Maybe here is some issue with the connectionpool logic: urllib3.connectionpool

I see many of these connectionpool processing inside the logs, but sometimes when connection is used, I have that 401 one. Is this connectionpool configurable inside the CIS or internal to the controller. Maybe some connections are getting closed, but still in the pool and resulting inside the 401 errors.

trinaths commented 6 months ago

Created [CONTCNTR-4648] for internal tracking.

trinaths commented 5 months ago

Fixed in 2.16.1