search

F5Networks / f5-appsvcs-extension

F5 BIG-IP Application Services 3 Extension
Apache License 2.0
164 stars 53 forks source link

HTTP ERROR 500 AsyncContext timeout when sending AS3 Declaration #780

Closed thepowercoders closed 10 months ago

thepowercoders commented 10 months ago

Environment

Summary

When trying to update the bigip VE device using AS3, the declaration is failing with the following error:

HTTP ERROR 500 AsyncContext timeout
URI:    /mgmt/shared/appsvcs/declare
STATUS: 500
MESSAGE:    AsyncContext timeout
SERVLET:    com.f5.rest.app.RestServerServlet-7f13d6e
Powered by Jetty:// 9.4.49.v20220914

AS3 declarations are failing with http 500 error (Server Error). It is not clear if the declaration is actually successful or not as no response is given.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit a large declaration - NOTE! This appears to only be an issue for asynchronous mode. Even if you have asynchronous mode set to false, after 45 seconds BIG-IP AS3 sets asynchronous mode to true (API swap), and returns an async response.
  2. Observe the above error response.

Expected Behavior

For a big declaration, BIG-IP normally returns a 202 Accepted which contains the job ID. This can then be queried with the following API call: https://<device>/mgmt/shared/appsvcs/task/<id> ..which eventually returns a 200 OK when the declaration is completed.

Actual Behavior

As above. Instead of getting the 202 Accepted, the API crashes with the error.

thepowercoders commented 10 months ago

note - this has only started since upgrading to big-ip 16.1.4.1 + Hotfix-BIGIP-16.1.4.1.0.50.5-ENG as a response to CVE-2023-467467and CVE-2023-46748. It does not occur in the 16.1.3.1 software.

dstokesf5 commented 10 months ago

That error does not appear to be coming from AS3. Do you see any errors in /var/log/restnoded/restnoded.log?

thepowercoders commented 10 months ago

No errors seen there. Due to the urgency of this (cannot provision) I have raised a Support Case (00512084) - this git issue was raised in parallel as advised by my CSM. Issue only occurs with a big declaration and error is returned always after around 30s.

nino-reger commented 10 months ago

We are experiencing the same Error, also raising a Support Case via our Reseller

thepowercoders commented 10 months ago

Tried different versions of AS3:

v3.46.1 - still get 500 AsyncContext timeout v3.47 - still get 500 AsyncContext timeout v3.48 - weirdly this version of AS3 is complaining about the presence of the property: "transparent": false within: "class": "Monitor","monitorType": "external", object. Error is: <object> should NOT have additional properties. (The versions above are fine with it). If I remove this, I then still get the 500 AsyncContext timeout

mdditt2000 commented 10 months ago

Jira AUTOTOOL-4089 added to the next sprint - Looks like a change on the BIG-IP side causing a timeout issue with AS3 API. Please contact me automation_toolchain_pm@f5.com so i can add a customer name to the Jira task. Cannot add a priority to an anonymous task

thepowercoders commented 10 months ago

@nino-reger - I've got a workaround for this - if you add the "async" parameter to the URL query string - like this: https://<URL_OF_BIGIP>/mgmt/shared/appsvcs/declare?async=true

..then it forces AS3 to return a 202 Accepted and a job ID. You can then do a GET on the ID to check for completion status.

nino-reger commented 10 months ago

@thepowercoders thanks for the hint, we already found that workaround, but as we are using f5networks ansible modules this issue still breaks our deployment pipeline.

@mdditt2000 our case number is 00516194

thepowercoders commented 10 months ago

you're welcome @nino-reger - we are using terraform provider which also doesn't have this, so also our pipeline is broken.

nino-reger commented 10 months ago

Not an appsvcs topic, but we are experiencing 500 AsyncContext timeout problems on several endpoints of the REST API

thepowercoders commented 10 months ago

@nino-reger - I tested Hotfix-BIGIP-16.1.4.1.0.117.5-ENG last night which fixed this issue on my bigips. Please ask F5 for this and hopefully it will work for you too. This hotfix includes ENG50 which patches the CVE issues.

mdditt2000 commented 10 months ago

Folks, this issue is related to BIG-IP and not AS3 - REST requests timeout after 30 seconds regardless of sys db variables setting - bugzilla id=1325685

Looks like 15.x and now 16.x is patched. Please open a SR to get a hotfix for your BIG-IP version. Please reference issue bugzilla id=1325685. This issue will timeout the AS3 request after 30 seconds for the POST to complete. Glad this issue is resolved in BIG-IP. Closing Github issue.

mdditt2000 commented 10 months ago

AS3 Team investigation Jira Tasks AUTOTOOL-4005, AUTOTOOL-4089 and AUTOTOOL-3980