Handle file uploads

[Rich-Harris]

Currently, the body parser bails on multipart forms that include files

[benmccann]

It looks like the way you'd do this in Sapper is to add something like Multer or Formidable as a middleware which would parse the incoming request and stick the files on the request object. Probably the way to do this would be to expose a way to add middleware, so that users can utilize these packages, so I wonder if this is essentially a duplicate of https://github.com/sveltejs/kit/issues/334. (side note, I wonder if we should have adapter-polka and adapter-express instead of adapter-node)

I was trying to figure out how Netlify, Lambda, etc. handle this. All the examples I see directly upload to S3 from the browser. I couldn't find how you'd handle a form upload inside a Lambda, so I guess that's something that's generally not supported

[thgh]

Here is an example of file uploads to lambda: https://github.com/mpxr/lambda-file-upload/blob/master/handler.js

I may be biased, but I think serverless APIs typically don't use global middleware and instead have several awaits (one per middleware) at the start of the function. See https://nextjs.org/docs/api-routes/api-middlewares#connectexpress-middleware-support

[asos-tomp]

For what it's worth, confirmed that this was working fine with sapper + Formidable but not working with sveltekit.

[axmad386]

I think hooks is enough for handling Middleware, except the request (ServerRequest) is little bit different from polka or express. Multer require request.pipe and unpipe to handle File Upload. Formidable require request.on which is missing from request object on handle hooks. Cmiiw

[benmccann]

You'd have to convert from SvelteKit request to Express request first before calling the middleware

[axmad386]

You'd have to convert from SvelteKit request to Express request first before calling the middleware

That's exactly I want, but how. Any references?

[vonadz]

Would it be possible to comment out this section https://github.com/sveltejs/kit/blob/e7f6fefe6756b4df24100c48e389fbde2ccecab4/packages/kit/src/runtime/server/parse_body/index.js#L92-L95 for now? This will give users the chance to implement their own solutions in the handle hook.

EDIT: If anyone is trying to find a solution for this, or a work around, here is what I did for an app that will work with the adapter-node:

1. Comment out the throw Error part of the code I linked above in your node_module @sveltejs/kit (I found this by just using grep -rin 'File upload is not yet implemented' ./node_modules/@sveltejs/)
2. Install busboy
3. Add the code below to your hooks file

import busboy from 'busboy'
import type { Handle } from '@sveltejs/kit'
export const handle: Handle = async ({ request, render }) => {
  console.log('in handle')
  console.log(request)
  if (request.path === 'your/path') {
    const files = {}
    await new Promise((resolve, reject) => {
      const bb = new busboy({
        headers: { 'content-type': request.headers['content-type'] },
      })
      bb.on('file', (fieldname, file, filename, encoding, mimetype) => {
        console.log(
          'File [%s]: filename=%j; encoding=%j; mimetype=%j',
          fieldname,
          filename,
          encoding,
          mimetype
        )
        console.log(file)
        const buffers = []
        files[fieldname] = {
          filename,
          encoding,
          mimetype,
        }
        file
          .on('data', (fileData) => {
            console.log('File [%s] got %d bytes', fieldname, fileData.length)
            buffers.push(fileData) //this adds files as buffers, which can be hard on memory. You can also write to a file using streams here.
          })
          .on('end', () => {
            console.log('File [%s] Finished', fieldname)
            const buffer = Buffer.concat(buffers)
            files[fieldname]['file'] = buffer
          })
      })
        .on('field', (fieldname, val) => {
          console.log('Field [%s]: value: %j', fieldname, val)
          request.body[fieldname] = val
        })
        .on('finish', () => {
          console.log('Done parsing form!')
          request.body['files'] = files
          resolve(request.body)
        })
        .on('error', (err) => {
          console.log('failed', err)
          reject(err)
        })

      bb.end(request.rawBody)
    })
  }
  const response = await render(request)
  return {
    ...response,
    headers: {
      ...response.headers,
    },
  }
}

EDIT 2: so I can't seem to get the above code to actually write nonbroken files. I've tried a couple of different methods, including substituting busboy for something like aws-lambda-multipart-parser, which gave files that were the correct size, but still corrupted or something. Maybe there is an issue in encoding/decoding processes for the rawBody that happens here:

https://github.com/sveltejs/kit/blob/f3ef93d16eed098fc2c247279e44e560bf46f5fe/packages/kit/src/core/http/index.js#L18-L29

EDIT 3: so changing the Buffer type for content-type multipart/form-data in the above code to 'ascii' gives me files of the exact size that are sent, but they're still corrupted for some reason. Anyone know what I might be missing here?

[MTyson]

I just did this in Sapper, and I'll throw out my thoughts in case it might help.

I used formidable and filepond. Filepond sends the chunks, and I used formidable to write a temp file to disk (via node fs). Upon submitting the form, a unique id is used to retrieve the file and write it to its permanent home (mongodb in this case).

In a pure serverless environment, you could override formidable and handle the chunks yourself, writing them to memory, and do the same process. (This assumes we can import formidable into sveltekit).

In the end, there should be a simple way to interact with the filesystem if there is one at hand.

[vonadz]

I just did this in Sapper, and I'll throw out my thoughts in case it might help.

I used formidable and filepond. Filepond sends the chunks, and I used formidable to write a temp file to disk (via node fs). Upon submitting the form, a unique id is used to retrieve the file and write it to its permanent home (mongodb in this case).

In a pure serverless environment, you could override formidable and handle the chunks yourself, writing them to memory, and do the same process. (This assumes we can import formidable into sveltekit).

In the end, there should be a simple way to interact with the filesystem if there is one at hand.

Yeah I played around with using formidable, but I don't think it's a very nice solution. From my understanding, the handle hook would be the natural place for handling fileupload. The issue is that the request in the handle hook isn't a stream of chunks, like is normally expected for something like formidable or express-fileupload, but an object with a rawBody that is the complete stream in string format. The reason I was using busboy is because it can parse out form data out of a string, both fields and files.

[MTyson]

handle hook isn't a stream of chunks

Interesting. So something in sveltekit itself is handling the chunks and marshalling them into a string of bits?

I wonder if that could pose a limitation in cases where one wanted to stream the chunks directly to a datastore...?

[juanmait]

handle hook isn't a stream of chunks

Interesting. So something in sveltekit itself is handling the chunks and marshalling them into a string of bits?

I wonder if that could pose a limitation in cases where one wanted to stream the chunks directly to a datastore...?

I'm just looking for that. I'm working with streaming in the browser side and now reached the point of wanting to stream the result of the client pipeline to the server. Seems like there is no way to access the raw request in sveltekit endpoints. I was planning to use cloudflare workers but first giving it a try using nodejs.

I would vote for having a way to access the plain raw request object on whatever the platform you use. I guess this would be the job of every adapter.

[avhust]

plain/text Content-Type allows to send data (e.g base64 coded image). I use this code with SvelteKit to upload images:

// <input type="file" bind:files>
const getBase64 = (image) => {
    const reader = new FileReader();
    reader.onload = (e) => {
        uploadImage(e.target.result);
    };
    reader.readAsDataURL(image);
};
const uploadFunction = async (imgBase64) => {
    const imgData = imgBase64.split(',');
    await fetch(`/api/upload`, {
        method: 'POST',
        headers: {
            'Content-Type': 'text/plain',
            Accept: 'application/json'
        },
        body: imgData[1]
    });
};
getBase64(file[0]);

and in hook.ts to capture the image and process with sharp:

export const handle: Handle = async ({ request, render }) => {
    //...
    // Upload Image
    if (request.path === '/api/upload' && request.method === 'POST') {
        const imageName = 'some.webp';
        try {
            await sharp(Buffer.from(request.rawBody, 'base64'))
                .rotate()
                .resize({ width: 200, height: 200 })
                .webp()
                .toFile(`./${imageName}`);
        } catch (err) {
            console.log(err);
        }
        return {
            status: 201,
            body: JSON.stringify({
                src: imageName
            }),
            headers: {
                'Content-Type': 'application/json'
            }
        };
    }
    // ...
};

[juanmait]

@avhust It's a good solution for relatively small uploads. But if you're dealing with sizes in the range of the gigabytes its like you cannot afford to fill the whole rawBody especially in an environment like vercel or cloudflare. Streaming allows you to start processing the data as soon as it start coming in and you can kind of control the memory usage and also even pause the stream to also control the CPU usage. I guess I could use some sort of proxy to redirect uploads to something outside sveltekit, or use CORS. But is an inconvenience :sweat_smile:

[benmccann]

There's an issue about streaming here: https://github.com/sveltejs/kit/issues/1563

[MTyson]

It would seem strange to have the client-side content type dictated by the framework. Feels like a workaround right out the box.

[geoidesic]

https://github.com/mscdex/busboy ftw! @vonadz any luck with this yet?

[vonadz]

https://github.com/mscdex/busboy ftw! @vonadz any luck with this yet?

Nope. I just decided to use an external service (transloadit) to handle my file uploads for me. It made more sense for my use case anyway.

[geoidesic]

@avhust I tried your work-around but I still get the dreaded Error: File upload is not yet implemented response. Doesn't seem to trigger any console.logs in the hooks.ts file, so I'm not convinced it's even getting that far.

export const handle: Handle = async ({ request, resolve }) => {
  console.log('handle hooks.ts')       //<---- this does get output on a normal request but not on a file upload
  const cookies = cookie.parse(request.headers.cookie || '');
  request.locals.userid = cookies.userid || uuid();

  // TODO https://github.com/sveltejs/kit/issues/1046
  if (request.query.has('_method')) {
    request.method = request.query.get('_method').toUpperCase();
  }

  // Upload Image
    if (request.path === '/examples/api/upload' && request.method === 'POST') {
    console.log('yo')         //<---- this does NOT get output on the route
    ...

[avhust]

@geoidesic Looks like you doesn't use 'Content-Type': 'text/plain' in your POST request.

[UltraCakeBakery]

As I am writing this I'm filling up my production svelte-kit server's error log budget... Svelte kit throws / logs an error, even after build. Anyone can simply add an empty file to their request and this causes the endpoint to break.

It is also annoying to work around the endpoint forcing a 500 error.

Can't instead a development environment only warning be thrown instead with the form data entry being set to a value of null? I have some security concerns and shitty developer experience because of the current implementation..

[joelhickok]

I'm using the solution provided by @avhust for now. Can't wait until sveltejs/kit supports better server-side customizations and middleware. It's kind of a handicapped tool at this point, powerful as it is. Yes, I understand it's still under development.

My example uses .xlsx instead of an image. This works fine without using the text/plain Content-Type.

// client side code
const reader = new FileReader()

reader.onload = async (e) => {
    const file = e.target.result.split(',')

    const response = await request.post('/upload.json', file[1], {
        params: {
            filename: encodeURIComponent('file.xlsx'), // just to pass in the actual filename to use
        },
    }).catch(err => err)
}

reader.readAsDataURL(fileInput.files[0])

// hooks.js or .ts
export async function handle({ request, resolve }) {
    if (request.path === '/upload.json' && request.method === 'POST') {
        const filename = request.query.get('filename')
        try {
            // this example show use for a file such as .xlsx instead of the previous example
            // using sharp for an image
            fs.writeFileSync(`static/uploads/${filename}`, request.rawBody, { encoding: 'base64' })
            // file now available for use in local upload path, so create additional logic elsewhere
        } catch (err) {
            console.log(err)
        }
    }

    const response = await resolve(request)

    return {
        ...response,
        headers: {
            ...response.headers,
        }
    }

}

// upload.json.js
export async function post(req) {
    console.log(`HANDLER: ${req.path}`)
    try {
        // do something else, or just return a response letting
        // the client know the upload went okay

        return {
            body: {
                success: true,
                filename: req.query.get('filename'),
            },
        }
    } catch (err) {
        return {
            body: err.message
        }
    }
}

[benmccann]

We're soon going to have the ability to add middleware in adapter-node (https://github.com/sveltejs/kit/pull/2051). Users can also do file uploads to S3, etc. on the serverless platforms. That seems like it should basically solve this

[benbender]

@benmccann I'm sorry, but I tend to disagree. Uploads are an essential part of the web - and of HTML-Spec. If we consider this issue closed, I'm basically forced to do either one of two things. In even mild usecases where the endpoint-concept of sveltekit might by totally fine otherwise:

• replicate my db-setup, form-handling, etc outside of sveltekit in a middleware as I normally need to handle/save the resulting filename, path, etc somewhere
• pay a service to do it externally for me with possibly attached concerns about cost, privacy, gdpr etc (yaeh I know, most of them start for free, but you get the point...)

Tbh, I don't get the notion transported in handling this issue. If I argue that external, specialized services do things better and I, as a dev, shouldn't do them therefor myself, we could skip many of the features of sveltekit and basically provide a linklist to 3rd-party, paid, specialized services in many cases...

To me, the handling of the request/response-objects in Sveltekit feels like an unnecessary cutback of possibilities the web offers me as a developer. I get the point that you are striving to support the whole serverless-idea, but I don't get why I have to sacrifice on adapter-node, where the apis are right in front of me but I'm not allowed to grab them. Imho nextjs does a much better job in abstracting this problem and I would love to see a change of mind by the sveltekit core team!

Edit: As I think more about this, I have to realize, that I'm especially confused about the handling here because it feels almost "non-svelte". Why? Because Svelte gives me almost direct-access to all the fun, native apis in the browser - without many of the indirections and abstractions introduced by other frameworks. Thats an incredibly powerful approach and removes so much overhead. But on the server-side of things sveltekit doesn't follow this approach - even if it would perfectly possible on adapter-node.

To me, it should expose as much of http.ClientRequest/http.ServerResponse as the platform, sveltekit runs on, offers - at least as an optIn (in cost of interoperability - but if I need it, I am able to leverage the power which is given to me by the platform) in addition to a guaranteed subset of interoperable apis.

That all said, thanks for your ongoing, incredible work! Love it otherwise! 👍

[benmccann]

I don't get why I have to sacrifice on adapter-node, where the apis are right in front of me but I'm not allowed to grab them To me, it should expose as much of http.ClientRequest/http.ServerResponse as the platform, sveltekit runs on, offers - at least as an optIn (in cost of interoperability - but if I need it, I am able to leverage the power which is given to me by the platform) in addition to a guaranteed subset of interoperable apis.

I'm not sure what you mean by this. We will be exposing adapter-node so that you can customize it 100% including any middleware to do things like handle file uploads (https://github.com/sveltejs/kit/pull/2051). That sounds to me like exactly what you're asking for. Or is there something else you mean?

Though I did realize that multer will stick the uploaded files on request.files, so we'll need to make sure there's a way to get at that before 1.0

[joelhickok]

Is the middleware in adapter-node only going to work in builds, or will it work in dev mode?

[benbender]

I'm not sure what you mean by this. We will be exposing adapter-node so that you can customize it 100% including any middleware to do things like handle file uploads (#2051). That sounds to me like exactly what you're asking for. Or is there something else you mean?

Though I did realize that multer will stick the uploaded files on request.files, so we'll need to make sure there's a way to get at that before 1.0

Your example pinpoints my direct problem quite exactly. As of now I expected that those middlewares will be basically living in there own silo and I won't have access to the (modified) "real" request/response-objects within hooks/endpoints. So even if I have full access to request/response in the middleware, there is no proprietary "communication-channel" into my app. Or am I missing something?

But even if there is such a concept to deliver possible changes from a middleware into my app the "cutback of possibilities" because of the missing access to clientRequst/serverResponse still exists and cripples the power of adapter-node quite a bit...

[benmccann]

Is the middleware in adapter-node only going to work in builds, or will it work in dev mode?

dev mode middleware is documented here: https://kit.svelte.dev/faq#integrations

I won't have access to the (modified) "real" request/response-objects within hooks/endpoints

Assuming that by "real" you mean the Express versions, I was basically agreeing that we will need a way to handle that in my last comment

[benbender]

Assuming that by "real" you mean the Express versions, I was basically agreeing that we will need a way to handle that in my last comment

More like the ones from https://nodejs.org/api/http.html - but 🥳 😄

[adam314315]

@vonadz @geoidesic Hello,

Finally succeeded to implement your method with the help of this post https://stackoverflow.com/questions/66807052/utf-8-encoded-string-to-buffer-node-js on issue generated with UTF-8 encoding. You need to cancel utf8 encoding in adapter-node as below. Then it will fix the corrupt file issue.

in @sveltejs/adapter-node/files/index.js

 if (isContentTypeTextual(type)) {
         const encoding = h['content-encoding'] || 'utf-8';
         return fulfil(data);
         //return fulfil(new TextDecoder(encoding).decode(data));
   }

[meticoeus]

@benmccann Chiming in with a serverless use case. I'm currently building a specialized app that will be running on Vercel (serverless) but I would still like to handle file uploads directly. This app will have only a small user base and has a strict allowed file uploads policy and I don't want to add the complexity of using S3 for this (Files are stored in a service that isn't very upload-from-client friendly). Uploads are limited to less than 1MB on the client so I don't expect any problems for the lambda runtime. Direct to S3 makes plenty of sense if you need to support large files. It just adds a lot of hassle if you don't really need it.

It would be great if sveltekit considered providing a well documented mechanism for gaining access to the raw request/response of whatever platform it is running on or some other way to handle raw requests at least for those of us with non-standard use cases.

For now I'm going to be uploading base64 strings but this bloats processing time and bandwidth transfer.

Also, really loving Svelte development compared to React so far. It is so much easier to deal with most things so far without a ton of annoying performance workarounds.

[benmccann]

@meticoeus I have a PR pending to allow access to the raw request of whatever platform is being used. See https://github.com/sveltejs/kit/pull/2359

[meticoeus]

@benmccann With the recent close of #2359 and the new discussion #2426 to add props to locals, is support for getting access to the raw request body in anything other that adapter-node still being considered?

[benmccann]

Whatever we do will work for all platforms and not just Node. I still need to sync up with Rich about design of https://github.com/sveltejs/kit/issues/2426

[higoka]

Are there any examples how to implement file uploads? Or are they not supported yet?

[benmccann]

Not supported yet

[higoka]

Not supported yet

I also tried the workaround with busboy posted above but this doesn't seem to work anymore. So was wondering if there is other workaround for now until it's implemented? Its stopping me from finishing my app :(

[ShahrukhAhmed89]

The workaround would be using express with sveltekit acting as a middleware (node-adapter). Express will handle the file upload while you can do the rest with sveltekit.

https://github.com/sveltejs/kit/tree/master/packages/adapter-node

[higoka]

The workaround would be using express with sveltekit acting as a middleware (node-adapter). Express will handle the file upload while you can do the rest with sveltekit.

https://github.com/sveltejs/kit/tree/master/packages/adapter-node

I see. Will try that for now, thx!

[benmccann]

That probably doesn't work yet either. Most Node middleware adds the files to the request and there's not a way to grab them off the request yet. I will discuss a solution to that with Rich when he dives back in a couple weeks from now

[ShahrukhAhmed89]

More like this

router.post("/upload-logo", async (req, res) => {
   //use multer to handle upload
  res.send()
})

//sveltekit for the rest 
router.use(assetsMiddleware, prerenderedMiddleware, kitMiddleware);

This is working for me for now. Yes, it would be great is we could also grab things passed via express session middleware.

[mithlesh425]

Community, any update on this issue / any working workaround as of now? Please let us know!

[jack-y]

My solution, for those who use adapter-node.

First, adds an upload plugin in the vite options of the svelte.config.js file:

// vite options
vite: {
    plugins: [{
        name: 'upload-middleware',
        configureServer (server) {
            server.middlewares.use(((req, res, next) => {
                if (req.url.startsWith('/upload')) {
                    /* Uploads the files here.
                     In this example, the "uploadFiles" function returns a Promise.
                     To retrieve the separate parts of the FormData,
                     the "uploadFiles" function uses the "busboy" package.
                     If you need, I can give you a full example.
                     */
                    uploadFiles(req)
                    .then(() => {
                        res.writeHead(200, {
                            'Content-Type': 'text/plain'
                        });
                        return res.end('ok');
                    })
                    .catch(err => {
                        res.writeHead(500, {
                            'Content-Type': 'text/plain'
                        });
                        return res.end('upload plugin error: ' + err);
                    });
                } else {
                    next();
                }
            }));
    }],
    ...
},

Then, in the application:

/* Assuming the "fileToUpload" variable is the File object of the multipart form */
const filepath = "your-file-path-and-name-on-disk";
/* Creates the form data */
let formData = new FormData();
/* Size must be the first part */
formData.append('size', fileToUpload.size);
/* Others parts */
formData.append('file', fileToUpload);
formData.append('filepath', filepath);
formData.append('mimeType', fileToUpload.mimeType); // could be used in a type control server side
/* Calls the process on the server */
fetch('upload', {
    method: 'POST',
    body: formData
    /* Don't set the Content-Type header.
     See: https://muffinman.io/blog/uploading-files-using-fetch-multipart-form-data/ 
     */
})
/* Handles the response */
.then(res => {
    if (!res.ok) { // the response has a bad status (500..)
        throw new Error('upload error status ' + res.status + ', status text: ' + res.statusText);
    } else {
        /* Do something when the file is uploaded */
    }
})
.catch(err => {
    /* Do something with the error */
});

Hope it helps.

[sebamon]

jack-y can u show the upload.js or ts file? i cant upload file to the server side. Thanks u!

[jack-y]

OK, here is a full example repository: svelte-kit-file-upload.
Have fun!

[yanick]

OK, here is a full example repository: svelte-kit-file-upload. Have fun!

Awesome! For giggles I cloned the repo and made a variation that uses multer and Uppy. So far, it works pretty darn good -- https://github.com/yanick/svelte-kit-file-upload

[sebamon]

Thank both! we can upload files now!

[vonadz]

Do the above solutions actually work when you export the project for production though? I vaguely remember when trying to solve this issue, I ran into the problem that the adapter-node and/or other adapters actually had to be adjusted as well.

[yanick]

Do the above solutions actually work when you export the project for production though?

Darn it, it doesn't. :-( dev mode is all peachy, but in the build mode I still get the 'File Upload is not yet implemented'. Bummer.

[vonadz]

Yeah, I figured. You'll have to make appropriate adjustments in the adapter-node package as well, which isn't that bad. The reason I stopped pursuing this though is because you'd also have to make adjustments in the other adapters, and I wasn't comfortable with serverless envs at the time.

[jack-y]

Arghhh!
@vonadz Please, what adjustments are necessary in the adapter-node package?