Resumable File Uploads

[Acconut]

Inside IETF's HTTP working group, we are currently discussing a draft for resumable file uploads over HTTP: https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/01/. The draft itself is rather new and might undergo severe changes, but I would like to bring this draft to the WHATWG for two reasons:

1) Ask you as experienced HTTP users for general feedback on the draft and its approach to resumable uploads, and 2) discuss a possible inclusion of resumable file uploads in the Fetch API.

In the remaining text here, I would like to briefly outline the current draft (version -01) and provide an initial idea how it could interact with the Fetch API:

Resumable Uploads over HTTP

Traditional file uploads using POST or PUT requests are not resumable, i.e. if the connection is interrupted during the upload, the entire upload has to be retried and some data might have to be retransmitted. For larger files or users with poor connectivity, this can be problematic. Many applications have solved this by implementing a proprietary solution for providing resumability. With our draft, we try to standardize an approach to resumable uploads, so that compatible software implementations are possible. Maybe resumable uploads capabilities can be implemented directly into browsers and other platforms directly.

The draft has been designed that it builds upon "traditional" file uploads and allows extending them with resumablity capabilities if supported by the client and server. If a client wants to upload a file using a POST request, the client can indicate its interest in resumablity by including a special header in the request (currently, this is the Upload-Incomplete header, but this name might change and is not too relevant for the broader picture here).

If the server also supports resumability, it can respond with an informational 104 Upload Resumption Supported response. It includes a Location header, which points to a newly created upload resource. This resource can be used to resume the upload, if it gets interrupted.

The request we are currently talking about is the normal POST/PUT request whose body is the file that should be uploaded. So while the file is being transferred the server might send this informational response.

If this request succeeds and the upload is finished successful, the upload resource is never used and everything is fine.

If the request gets interrupted or the connectivity is lost, the upload is interrupted and can be resumed using the upload resource. For this, the client first sends a HEAD request to the upload resource to fetch the offset (i.e. the number of bytes that the server successfully received and stored). After this, the client can upload the remaining data after the offset using a PATCH request to the upload resource. This dance is repeated until the entire file has been transferred without the need of retransmitting any chunks.

Interaction with Fetch API

As mentioned before, this draft attempts to extend traditional file uploads with resumability. If a server does not support resumability, the file upload should just be treated as any other HTTP request.

If the developer wants to upload a file resumably, it could indicate this interest to the server using an option resumable:

  const response = await fetch(url, {
    method: "POST"
    body: file,
    resumable: true,
  });

When this option is set, the client should into the special header and monitor the connection for the 104 Upload Resumption Supported informational response. Based on whether the client received it, multiple outcomes are possible:

• the request completes -> the response is provided as for any normal fetch invocation
• the request is interrupted and no 104 Upload Resumption Supported was received -> an error is raised as for any normal fetch invocation
• the request is interrupted and a 104 Upload Resumption Supported was received -> the client initiates a resumption and either raises an error or provides a response depending on the outcome

This way, resumable file uploads are provided by the Fetch API and can easily be used by developers without the inclusion of an additional library.

I hope this provides a quick overview of our current approach to resumable uploads. We are interested in any feedback on the approach itself, but also if there is interest in integrating it into Fetch.

[jimmywarting]

kind of interesting... should the request happen in the background then like using background fetch? what if someone wants to manually stop/pause and resume by themself?

You would have to create a Request first before you could do the actual request.

req = new Request(url, { resumable: true, ... })
fetch(req)
req.pause() || req.stop()
req.resume() // initiate a new request.

[Acconut]

what if someone wants to manually stop/pause

With resumable uploads, a pause is just an aborting of the currently running HTTP request. Therefore, one option would be to reuse the abort controller that is already available in the Fetch API:

const controller = new AbortController();
const signal = controller.signal;

fetch(url, {
    method: "POST"
    body: file,
    resumable: true,
    signal
});

// Stop the upload
controller.abort();

You would have to create a Request first before you could do the actual request.

I am not sure if there is interest in using a method on the Request type to trigger a new HTTP request. Another option is to not reuse the previous fetch instance and instead create a new request that carries over some resumption information from the previous request. That's similar how Apple implemented resumable uploads in URLSession on iOS:

try {
  await fetch(url, {
    method: "POST"
    body: file,
    resumable: true,
  });
} catch (err) {
  const resumeData = err.getResumeData()
  if (resumeData != null) {
    await fetch(url, {
      method: "POST"
      resumeData,
    });
  }
}

err.getResumeData() could return some opaque data type that carries information about the upload URL and the file to be uploaded. By passing this information from the previous fetch call to the next one, enough information is available to resume the upload.