[FEATURE] Retry requests if "tls handshake eof" error detected

[tomas-zijdemans-vipps]

I have a workflow that runs daily, haven't made changes to it for some time. Got this error today. Maybe a hiccup on the backend?

2023-11-13 08:00:50 [info] [Fn060X3USBNZ] (Trace=Tr0657KMM4GN) Function output:
        Caught error from user supplied module: TypeError: error sending request for url (https://slack.com/api/functions.completeSuccess): error trying to connect: tls handshake eof
            at async mainFetch (ext:deno_fetch/26_fetch.js:277:12)
            at async fetch (ext:deno_fetch/26_fetch.js:504:7)
            at async BaseSlackAPIClient.apiCall (file:///opt/runtime.js:44:22)
            at async Object.RunFunction [as function_executed] (file:///opt/runtime.js:1709:29)
            at async DispatchPayload (file:///opt/runtime.js:1790:16)
            at async run (file:///opt/runtime.js:1857:18)
            at async serveHttp (file:///opt/runtime.js:1885:38)
        Error responding HTTP 500 to webapp request: Http: connection closed before message completed

[WilliamBergamin]

Hi @tomas-zijdemans-vipps thanks for writing in 💯

Was this error happening consistently?

As this was a tls handshake issue I don't think a proper connection was made with the Slack API this could be related to a one off network issue

[filmaj]

Hey @tomas-zijdemans-vipps I transferred this issue from the deno-slack-sdk repo to this one, deno-slack-runtime, which houses the specific code your stack trace logged out.

Can you confirm what time zone your logs are based in? I don't see any obvious problems with the functions.completeSuccess API this week, but there may be further areas of failure on our hosting side that I can investigate.

[filmaj]

And one more question @tomas-zijdemans-vipps : how many times did you see this happen? If only a handful of times, perhaps we should consider some manner of fallback or retry behaviour if we can catch this specific error... what do you think @WilliamBergamin ?

[WilliamBergamin]

If this happens enough a retry behavior may be worth while 💯

[tomas-zijdemans-vipps]

@filmaj I am on Oslo time (GMT+1. 6h behind you I reckon) This a function used in several workflows, so it typically runs 2-3 times per day without any problems. Looking back, as far as slack logs would let me, I could find 2 more of these (so that's 3 times in 1 month).

2023-10-30 08:02:03 [info] [Fn060X3USBNZ] (Trace=Tr063BK3AQCA) Function output:
        Caught error from user supplied module: TypeError: error sending request for url (https://slack.com/api/functions.completeSuccess): error trying to connect: tls handshake eof
            at async mainFetch (ext:deno_fetch/26_fetch.js:277:12)
            at async fetch (ext:deno_fetch/26_fetch.js:504:7)
            at async BaseSlackAPIClient.apiCall (file:///opt/runtime.js:44:22)
            at async Object.RunFunction [as function_executed] (file:///opt/runtime.js:1709:29)
            at async DispatchPayload (file:///opt/runtime.js:1790:16)
            at async run (file:///opt/runtime.js:1857:18)
            at async serveHttp (file:///opt/runtime.js:1885:38)
        Error responding HTTP 500 to webapp request: Http: connection closed before message completed

and

2023-10-13 10:15:57 [info] [Fn060X3USBNZ] (Trace=Tr061D158Q0Z) Function output:
        Caught error from user supplied module: InternalServerErrorError: 500: 
            at createHttpError (file:///var/task/functions/send-kam-report.js:1410:12)
            at BaseSlackAPIClient.createHttpError (file:///var/task/functions/send-kam-report.js:1492:16)
            at eventLoopTick (ext:core/01_core.js:183:11)
            at async BaseSlackAPIClient.apiCall (file:///var/task/functions/send-kam-report.js:1472:19)
            at async AsyncFunction.<anonymous> (file:///var/task/functions/send-kam-report.js:3079:28)
            at async Object.RunFunction [as function_executed] (file:///opt/runtime.js:1683:56)
            at async DispatchPayload (file:///opt/runtime.js:1790:16)
            at async run (file:///opt/runtime.js:1857:18)
            at async serveHttp (file:///opt/runtime.js:1885:38)

[filmaj]

I did one more transfer for this issue 😅 I moved it to deno-slack-api, the repo responsible for making HTTP calls to Slack APIs in deno. After some discussion internally, we are going to add a retry feature to this project so that such negotiation connection errors as the one identified in this issue are retried. We have a few more reports for this error elsewhere at Slack, and the general feedback from our infrastructure team is 'these issues are extremely hard to isolate and very rare' and the recommendation has been to implement a retry mechanism.

[tomas-zijdemans-vipps]

Sounds good! I case it's useful: Deno has a retry function available in the standard library :)