[files] Large uploads timing out

[mateimarica]

When uploading large files (~2GB and above), the connection will often die after several minutes of uploading.

Progress bar stops moving:

Then, a minute or two later, this error is printed to the console. No extra info in the network tab.

Failed to load resource: net::ERR_CONNECTION_RESET

This is on Chrome, not tested on Firefox.

---

I suspect it has to do with the server timing-out the request. Needs more investigation.

• https://stackoverflow.com/questions/34892240/node-js-connection-reset-on-file-upload-with-multer
• https://stackoverflow.com/questions/63500097/err-connection-reset-when-upload-large-file-nodejs-multer
• https://stackoverflow.com/questions/21708208/express-js-response-timeout
• https://stackoverflow.com/questions/32882507/nodejs-and-express-server-closes-connection-after-2-minutes

The documentation for server.keepAliveTimeout:

Timeout in milliseconds. Default: 5000 (5 seconds). The number of milliseconds of inactivity a server needs to wait for additional incoming data, after it has finished writing the last response, before a socket will be destroyed. If the server receives new data before the keep-alive timeout has fired, it will reset the regular inactivity timeout, i.e., server.timeout.

It could be that if the client doesn't send a chunk of data for 5 seconds upload, the server kills the connection. This is more likely to happen during large (and long) uploads. In addition, there might be some network throttling (from the browser) occuring if the tab is not in focus during upload. This could play a role in the client not sending data frequently enough.

Read this article too, server.headersTimeout may need to be set too.

If possible, it would be preferable to increase this timeout only for the /upload endpoint.

[mateimarica]

Problem

It turns out the issue is with the new default node http.server options as of Node 18: (see docs)

• requestTimeout , default is 300000 ms (5 min)

  Sets the timeout value for receiving the complete HTTP headers from the client

• connectionsCheckingInterval , default is 30000 ms (30 secs)

  Sets the interval value to check for request and headers timeout in incomplete requests

So, the server will terminate any long-running request after 5 to 5:30 minutes, depending on when the connectionsCheckingInterval checker runs.

Desired Outcome

• Any request to the /uploads endpoint should run for as long as it takes. For example, it would take ~12 hours to upload a 5 GB file with a 1 Mbps upload rate.

• Only the /uploads endpoint should use a long request timeout. Other endpoints should stay with the default.

• A long request timeout should only be set AFTER the user is authenticated, otherwise it is more vulnerable to DDoS attacks and is just not efficient in general.

Solution

Now the problem is that requestTimeout seems to only be settable server-wide, rather than on a per-endpoint basis. Increasing the server-wide requestTimeout is a bad idea.

Googling this problem will give you people saying to add middleware to set timeouts directly to the req object. This doesn't work:

• req.setTimeout(ms) is an alias for req.socket.setTimeout(x)
• req.socket.setTimeout(ms) sets the SOCKET timeout

  A socket timeout is the timeout when waiting for individual packets. It's a common misconception that a socket timeout is the timeout to receive the full response

• req.connection.setTimeout(ms) sets the max time for a connection to be SET UP

  A connection timeout is the maximum amount of time that the program is willing to wait to setup a connection to another process. You aren't getting or posting any application data at this point, just establishing the connection, itself.

_source

I tested the methods above on both the req and res objects, there's mixed answers on which one works. Either way, they do not affect the total timeout of a request.

Since I can only see requestTimeout being set server-wide, here are two bad solutions:

• Make a new subdomain (eg filesupload.mateimarica.dev) for uploading. This would require a virtual network for a second IP address and would have to be run as a separate node server.
• Run a second node server on a separate port (eg 444) just for uploading. Would have to update the CSP header too.

Both of these options are terrible. More research must be done.