search

cheatcode / joystick

A full-stack JavaScript framework for building stable, easy-to-maintain apps and websites.
https://cheatcode.co/joystick
Other
209 stars 11 forks source link

Upgrade multer dependency so we can leverage streams #349

Closed rglover closed 2 months ago

rglover commented 1 year ago

Noticed that uploads can be slow and after digging found out that the latest version of Multer switched to using streams for uploads. Just need to refactor the uploads API and make sure we keep the shape of the returned data consistent (e.g., req.file or req.files still exists).

Another option is to just use Busboy directly (the core dependency Multer uses to parse uploads). Technically we don't need a bunch of fancy features—we just want the file data.

rglover commented 1 year ago

Just did a test. Only "gotcha" is making sure we handle the write to disk from the stream for local uploads and then pass the stream off to S3 for uploads there: https://stackoverflow.com/questions/37336050/pipe-a-stream-to-s3-upload.

rglover commented 1 year ago

Got it. See examples in your /cheatcode/experiments/multer2 directory.

rglover commented 2 months ago

I ended up trying this but the latest releases completely change data structures and the streaming doesn't work how you'd think. It basically just writes the file to a temp folder on disk and then creates a stream from that. Not worth it compared to the current version.