Test Replay: [Upload] Update timeout logic to fail faster on catastrophic timeouts

[ryanthemanuel]

What would you like?

Currently, the logic for timing out on uploading test replay data is set at 2 minutes. We also have retrying logic that attempts to retry 8 times on failures. This ends up being over 16 minutes of waiting if the upload URL will never be reachable. To resolve this, I propose:

• Reducing the upload time based on some maximum upload times we've seen in the wild
  • node-fetch may not support all that we want to do here, but we can likely tap into our agent to listen to the appropriate events
  • We should mimic what @cypress/request does and provide timeouts in multiple scenarios (https://github.com/cypress-io/request/blob/master/request.js#L799-L851):
    1. socket connect (to upstream)
    2. response sent (headers/status code)
    3. data flowing
    4. response complete (possibly this one should be very high)
• Detect when a timeout has occurred more than once and fail immediately (other retries would work as they do today)

More context:

• https://cypressio.slack.com/archives/C02FKC77SR1/p1707414373773239
• https://cypressio.slack.com/archives/C045K9KUWR2/p1707342711145369

Why is this needed?

No response

Other

No response

[cacieprins]

I think we can simplify this a little - copying what @cypress/request does may not be the best course.

cypress/request connection timeouts have a maximum of the OS-wide TCP connection timeout. Increasing this won't have any effect, and reducing this would probably only cause a higher error rate. I'd prefer to keep this at the OS-specified value; extending @packages/network Agent will be expensive to implement, and likely won't have a high rate of return.

Status code available sent is only important in error cases I think, which will simply reject the promise.

Data flowing is a great metric for timing out an upload, because it's what's important to the success of the upload.

I suggest:

• Timeout on "Upload Begin": the interval between fetch being called and the first .write to an intermediate .pipe from the artifact's ReadableStream. This handles: socket connect, http status/headers available (if sent prior to upload beginning). This is to detect initial connection errors.
• Timeout on "Upload Progress": the time between each .write to an intermediate .pipe from the artifact's ReadableStream. This is to detect stalled uploads.
• No timeout on upload complete: the upload duration is too variable to choose a good number, and the error case (a stalled upload) is already handled by the progress timeout.

What do you think?