AS3 deployment fails - Error 500 AsyncContext timeout

oakh1 commented 10 months ago

Environment

Application Services Version: f5-appsvcs-3.43.0-2.noarch
BIG-IP Version: BIG-IP 15.1.9.1 Build 0.0.5 Point Release 1

Summary

AS3 declaration deployment fails with config save failure / timeout.

Steps To Reproduce

Steps to reproduce the behavior:

Submit any declaration

Observe the following error response:

{
"code": 500,
"declarationFullId": "48aa4a3e-9af1-45b9-8e28-70dc2293077c",
"message": "failed to save BIG-IP config (GET ***localhost:8100/mgmt/tm/task/sys/config/1692867713201884 checking config save status response=500 body=<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\n<title>Error 500 AsyncContext timeout</title>\n</head>\n<body><h2>HTTP ERROR 500 AsyncContext timeout</h2>\n<table>\n<tr><th>URI:</th><td>/mgmt/tm/task/sys/config/1692867713201884</td></tr>\n<tr><th>STATUS:</th><td>500</td></tr>\n<tr><th>MESSAGE:</th><td>AsyncContext timeout</td></tr>\n<tr><th>SERVLET:</th><td>com.f5.rest.app.RestServerServlet-11f39f9</td></tr>\n</table>\n<hr/><a href=\"[https://eclipse.org/jetty\](https://eclipse.org/jetty/)">Powered by Jetty:// 9.4.49.v20220914</a><hr/>\n\n</body>\n</html>\n)"
}

Expected Behavior

The declaration to be deployed and saved successfully

Actual Behavior

The declaration is indeed deployed (but we don't know for sure without checking the actual config), but the config save fails which makes our pipieline fail as well, including that of course it's not saved which could be a big issue incase reboot occurs.

The failure is intermittent and can work one day, while the next day it doesn't. Now lately it's issues all the time. For those declarations it doesn't even take long to deploy (1-2 minutes) and they're not complicated at all. It makes no difference on what's deployed.

Before minor upgrade to 15.1.9.1 we had the same issue but the error message was different (not HTML with that "AsyncContext timeout").

sunitharonan commented 10 months ago

Thanks @oakh1 for reaching out. In order to prioritize and understand the issue better, please reach out to us at automation_toolchain_pm@f5.com

amolari commented 10 months ago

@sunitharonan I have the same/very similar issue (on v16.1.4, AS3 3.47.0rc) and opened some days ago a Support case 00460665. Note: the timeout settings (sys db) on the system follow the AS3 best-practices document => the 3 values have been changed from 60 to 180

[{'code': 422, 'message': 'declaration failed', 'response': 'GET http://admin:XXXXXX@localhost:8100/mgmt/tm/task/cli/script/1693226757164034 checking cli script status response=500 body=<html>\\n<head>\\n<meta http-equiv=\"Content-Type\" content=\"text/html;charset=ISO-8859-1\"/>\\n<title>Error 500 AsyncContext timeout</title>\\n</head>\\n<body><h2>HTTP ERROR 500 AsyncContext timeout</h2>\\n<table>\\n<tr><th>URI:</th><td>/mgmt/tm/task/cli/script/1693226757164034</td></tr>\\n<tr><th>STATUS:</th><td>500</td></tr>\\n<tr><th>MESSAGE:</th><td>AsyncContext timeout</td></tr>\\n<tr><th>SERVLET:</th><td>com.f5.rest.app.RestServerServlet-51cdd8a</td></tr>\\n</table>\\n<hr/><a href=\"https://eclipse.org/jetty\">Powered by Jetty:// 9.4.49.v20220914</a><hr/>\\n\\n</body>\\n</html>\\n', 'host': 'localhost', 'tenant': 'APIm', 'runTime': 59686}]"

sunitharonan commented 9 months ago

Hi, we digged a bit more into this issue and even increasing timeout wouldn't work in this case as we have already tried it. We believe it is a Big Ip product side issue as it is working fine on 15.1.8. We referred this case to Big IP team as AS3 is working as intended. Closing this case.

F5Networks / f5-appsvcs-extension