[BUG] FUNCTION_ERROR - Acquire operation took longer than the configured maximum time.

[mildmojo]

In the past few days, I've gotten two workflow error emails with this text, which sounds like an upstream AWS issue:

FUNCTION_ERROR — Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.

This sounds like an infrastructure error that's not actionable for me as a Pipedream user; perhaps this one should be suppressed and maybe used by internal staff instead? Or maybe it could be caught and rewritten with Pipedream-specific advice?

[dylburger]

@mildmojo I just wanted to acknowledge this and let you know that we're aware of the issue - I apologize for the spurious errors. You're correct that it's an internal error and isn't actionable.

We're investigating a way to both:

• Suppress the error string and note that it's a system error, and
• Reduce the frequency of this specific error

[dylburger]

@mildmojo we shipped a fix for this last week and have been letting it marinate to confirm the issue didn't arise again. I believe it should now be fixed. Closing this ticket but let me know if you see any related issues.