multipart/form-data parser

cyco130 commented 2 years ago

We should implement a multipart/form-data parser.

The problem

Real-world server server-side applications have to have a way of parsing multipart/form-data, including file uploads. The fetch standard provides request.formData but the API is not adequate for server-side use: multipart/form-data is a streaming format but formData is a "random access" one. You're supposed to be able to access uploaded files at any time. That is, you can call formData.get("file1") or formData.get("file2") at any time. This requires buffering the whole request which is prohibitively expensive in many environments.

To add to the inadequacy of the API itself, existing implementations are also lacking:

Node's native fetch doesn't support multipart/form-data at all (calling request.formData throws).
Cloudflare Workers' multipart parser treats the file bodies as UTF-8 strings, causing the file contents to be corrupt.

node-fetch does a good job within the limitations of the API by buffering the entire request in memory.

Suggested solution

We can implement a streaming parser for multipart/* as an async iterator (or equivalent interface):

for await (const part of multipartParser(request)) {
  // part.body is a ReadableStream
  // part.headers is a Headers object
  //
  // If the user doesn't drain the stream before
  // requesting the next part, the library will.
  // It is not allowed to save the stream for later
  // use as it would be very inefficient. The user
  // can implement their own buffering though.
}

On top of this, we can implement an easier-to-use interface specifically for multipart/form-data and on a third layer, we can implement file stores for Node FS, Deno FS, Cloudflare Workers KV, AWS S3, etc.

Impact

Such a solution would be very useful for non-HatTip environments too.

TODO

[ ] Investigate how Deno handles this.

Prior art and references

CanRau commented 2 years ago

Just stumbled upon hattip and really like the concept. Maybe Remix implementation might be inspirational as well https://remix.run/docs/en/v1/api/remix#unstable_parsemultipartformdata

cyco130 commented 2 years ago

Maybe Remix implementation might be inspirational as well

Thanks, I've already dug it up of course :D Progress report: I have a mostly working implementation, I'll clean it up and release it after I'm done with the CLI.

CanRau commented 2 years ago

Amazing 😍 already curious how I'd be used in rakkas 😁

hattipjs / hattip