File upload doesn't work as expected

Bug Report

Current behavior

on uploading file, value of file is undefined

Input Code

    @Post('upload')
    @UseInterceptors(FileInterceptor('file'))
    uploadFile(@UploadedFile() file) {
        console.log(file);
    }

POST /digital-verification-dev/us-central1/api/session/upload HTTP/1.1
Host: localhost:5000
Content-Length: 207
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="/C:/Users/Downloads/picture.jpg"
Content-Type: image/jpeg

(data)
----WebKitFormBoundary7MA4YWxkTrZu0gW

I added this config in my module as well, it creates the folder but nothing will be uploaded there (although I need the file in memory not on disk)

        MulterModule.register({
            dest: './upload',
        }),

Expected behavior

file has value

Possible Solution

Environment


Nest version: 7.5.1


For Tooling issues:
- Node version: 14.15.0
- Platform:  Windows 10

Others:

Please provide a minimum reproduction Repository

@jmcdo29 Thanks for fast response, Here is the repo https://github.com/RezaRahmati/nest-file-upload

I realized this issue doesn't happen on nestjs running alone. When adding firebase to nestjs this happens, the above is the minimum project nestjs + firebase function.

To run it you need to npm install -g firebase-tools and then run firebase login and after login change the project name in .firebaserec file and in the package.json set:env command

@jmcdo29 I think this might be the root cause of the issue https://stackoverflow.com/a/58506868/2279488

and here is in cloud documentation about handling the file upload https://cloud.google.com/functions/docs/writing/http#multipart_data

This issue does not seem to be related to NestJS. As suggested in the issue you've linked, I'd recommend using the mentioned multer fork or busboy. Let's add a warning in the docs https://github.com/nestjs/docs.nestjs.com/issues/1614

@kamilmysliwiec Thanks for response, would you please also update docs about how to create custom provider (Interceptors) for file upload with other third parties

In case somebody comes here for solution:

npm i busboy

file-info.ts

export interface FileInfo {
    fieldName: string;
    fileName: string;
    encoding: string;
    mimeType: string;
}

file-interceptor-service

import { ForbiddenException, Injectable } from '@nestjs/common';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { v4 as uuidV4 } from 'uuid';

import { FileInfo } from './file-info';

@Injectable()
export class FileInterceptorService {
    interceptRequest(request, options?: {
        onFile?: () => void,
        headers?: any,
        fileUniqueId?: string,
    }): Promise<{ fields: { [key: string]: any }, files?: Array<FileInfo> }> {
        options = options || {};
        options.headers = options.headers || request.headers;
        const customOnFile = typeof options.onFile === 'function' ? options.onFile : false;
        const fileUniqueId = options.fileUniqueId || uuidV4();
        delete options.onFile;

        const Busboy = require('busboy');
        const busboy = new Busboy({
            ...options,
            limits: {
                // Cloud functions impose this restriction anyway
                fileSize: 10 * 1024 * 1024,
            },
        });

        return new Promise<{ fields: { [key: string]: any }, files?: Array<FileInfo> }>((resolve, reject) => {
            const fields = {};
            const filePromises: Array<Promise<FileInfo>> = [];

            const cleanup = () => {

                busboy.removeListener('field', this.onField);
                busboy.removeListener('file', customOnFile || this.onFile);
                busboy.removeListener('close', cleanup);
                busboy.removeListener('end', cleanup);
                busboy.removeListener('error', onEnd);
                busboy.removeListener('partsLimit', onEnd);
                busboy.removeListener('filesLimit', onEnd);
                busboy.removeListener('fieldsLimit', onEnd);
                busboy.removeListener('finish', onEnd);
            };

            const onError = (err) => {
                cleanup();
                return reject(err);
            };

            const onEnd = (err) => {
                if (err) {
                    return reject(err);
                }
                if (customOnFile) {
                    cleanup();
                    resolve({ fields: fields });
                } else {
                    Promise.all(filePromises)
                        .then((files) => {
                            cleanup();
                            resolve({ fields: fields, files: files });
                        })
                        .catch(reject);
                }
            };

            request.on('close', cleanup.bind(this));

            busboy
                .on('field', this.onField.bind(this, fields))
                .on('file', customOnFile || this.onFile.bind(this, filePromises, fileUniqueId))
                .on('close', cleanup.bind(this))
                .on('error', onError.bind(this))
                .on('end', onEnd.bind(this))
                .on('finish', onEnd.bind(this));

            busboy.on('partsLimit', () => {
                const err = new ForbiddenException('Reach parts limit');
                onError(err);
            });

            busboy.on('filesLimit', () => {
                const err = new ForbiddenException('Reach files limit');
                onError(err);
            });

            busboy.on('fieldsLimit', () => {

                const err = new ForbiddenException('Reach fields limit');
                onError(err);
            });

            request.pipe(busboy);

            busboy.end(request.body);

        });
    }

    deleteFiles(files: Array<FileInfo>): void {
        files.forEach(file => {
            fs.unlinkSync(file.fileName);
        });
    }

    private onField(fields, name: string, val: any, fieldNameTruncated, valTruncated) {
        if (name.indexOf('[') > -1) {
            const obj = this.objectFromBluePrint(this.extractFormData(name), val);
            this.reconcile(obj, fields);

        } else {
            if (fields.hasOwnProperty(name)) {
                if (Array.isArray(fields[name])) {
                    fields[name].push(val);
                } else {
                    fields[name] = [fields[name], val];
                }
            } else {
                fields[name] = val;
            }
        }
    }

    private onFile(
        filePromises: Array<Promise<FileInfo>>,
        fileUniqueId: string,
        fieldName: string,
        file: NodeJS.ReadableStream,
        fileName: string,
        encoding: string,
        mimeType: string,
    ) {
        const tmpName = `${fileUniqueId}-${path.basename(fileName)}`;
        const saveTo = path.join(os.tmpdir(), path.basename(tmpName));
        const writeStream = fs.createWriteStream(saveTo);

        const filePromise = new Promise<FileInfo>((resolve, reject) => writeStream
            .on('open', () => file
                .pipe(writeStream)
                .on('error', reject)
                .on('finish', () => {
                    const fileInfo = {
                        fieldName: fieldName,
                        fileName: saveTo,
                        encoding: encoding,
                        mimeType: mimeType,
                    };
                    resolve(fileInfo);
                }),
            )
            .on('error', (err) => {
                file
                    .resume()
                    .on('error', reject);
                reject(err);
            }),
        );

        filePromises.push(filePromise);
    }

    private extractFormData = (str: string) => {
        const arr = str.split('[');
        const first = arr.shift();
        const res = arr.map(v => v.split(']')[0]);
        res.unshift(first);
        return res;
    }

    private objectFromBluePrint = (arr, value) => {
        return arr
            .reverse()
            .reduce((acc, next) => {
                if (Number(next).toString() === 'NaN') {
                    return { [next]: acc };
                } else {
                    const newAcc = [];
                    newAcc[Number(next)] = acc;
                    return newAcc;
                }
            }, value);
    }

    private reconcile = (obj, target) => {
        const key = Object.keys(obj)[0];
        const val = obj[key];

        if (target.hasOwnProperty(key)) {
            return this.reconcile(val, target[key]);
        } else {
            return target[key] = val;
        }

    }
}

file-saver.service.ts

import { Injectable } from '@nestjs/common';

import * as admin from 'firebase-admin';
import { v4 as uuidV4 } from 'uuid';

@Injectable()
export class FileSaverService {
    async saveFile(fileName: string, mimeType: string, destinationFolder: string) {
        const path = require('path');

        const bucket = admin.storage().bucket();

        const [file, meta] = await bucket.upload(fileName, {
            destination: `${destinationFolder}/${path.basename(fileName)}`,
            resumable: false,
            public: true,
            metadata: {
                contentType: mimeType,
                metadata: {
                    firebaseStorageDownloadTokens: uuidV4(),
                },
            },
        });

    }
}

in controller


import { Post, Req, Request } from '@nestjs/common';

    constructor(
        private fileSaverService: FileSaverService,
        private fileInterceptorService: FileInterceptorService,
    ) { }

    @Post('upload')
    async uploadFile(@Req() req: Request) {

        const { files, fields } = await this.fileInterceptorService.interceptRequest(req);

        const folder: string = `path/to/store`;

        await this.asyncForEach(files, async (file) => {
            this.fileSaverService.saveFile(file.fileName, file.mimeType, folder);
        });

        this.fileInterceptorService.deleteFiles(files);
    }

    async asyncForEach(array: Array<any>, callback: (item: any, index: number, array: Array<any>) => void): Promise<void> {
        for (let index = 0; index < array.length; index++) {
            await callback(array[index], index, array);
        }
    }

I wasted hours and hours thinking my code was buggy or my multer declaration was wrong. Finally found that it was cloud functions, it was really really hard for me to get to this thread, also didn't find anything about this in documentation.

Just in case someone wants just a simple workaround for now:

Install await-busboy fork from my github: npm i shahkeyur/await-busboy

Create file.interceptor.ts

import {
  CallHandler,
  ExecutionContext,
  Injectable,
  NestInterceptor,
} from '@nestjs/common';
import * as busboy from 'await-busboy';
import * as fs from 'fs';
import { tmpdir } from 'os';
import { extname, join } from 'path';
import { Observable } from 'rxjs';

@Injectable()
export class FileInterceptor implements NestInterceptor {
  editFileName = file => {
    const name = file.filename.split('.')[0];
    const fileExtName = extname(file.filename);
    const randomName = Array(4)
      .fill(null)
      .map(() => Math.round(Math.random() * 16).toString(16))
      .join('');

    return `${name}-${randomName}${fileExtName}`;
  };

  async intercept(
    context: ExecutionContext,
    next: CallHandler<any>,
  ): Promise<Observable<any>> {
    const request: any = context.switchToHttp().getRequest();

    if (!request.is('multipart/*')) return next.handle();

    const parts = busboy(request, { autoFields: true });

    try {
      let part;
      while ((part = await parts)) {
        const name = join(tmpdir(), this.editFileName(part));
        request.file = {
         // path where image is stored temporarily on api
          path: name,
        };
        // otherwise, it's a stream
        part.pipe(fs.createWriteStream(name));
      }
    } catch (err) {
      return request.throw(err);
    }

    request.body = parts.field;

    return next.handle();
  }
}

And you're ready to import and use: @UseInterceptors(new FileInterceptor())

Don't forget to unlink the image after use. Otherwise you may end up with all these images on api consuming gigs.

  unlink(avatar.path, () => {
        console.log('File deleted');
      });

It's not the perfectt interceptor, you might need to change it for your needs. But it may give you a starting point.

@kamilmysliwiec Thanks for response, would you please also update docs about how to create custom provider (Interceptors) for file upload with other third parties

Yeah, I encountered this issue today. Is there a chance you show us an example?

"@nestjs/core": "^8.4.3"

Like others did, let me leave the following solution for whom it might benefit.

The solutions of others are very okay. But I wanted something very close to what NestJS provides. At the same time, I wanted to be so close to the solution that Google Cloud or Firebase provided. I also didn't want to be storing the files on disk. I just wanted them to be available in the request body.

You will first need some FileInfo interface that is kind of equivalent to Express.Multer.File (you will understand if you are coming from NestJS's Docs)

// file-info.ts
export interface FileInfo {
  fieldName: string;
  originalName: string;
  encoding: string;
  mimeType: string;
  size: number;
  buffer: Buffer;
}

The following FileInterceptor decorator closely resembles what the docs above specify. It is modeled against the source code of the Interceptor exported from the @nestjs/common package.

Note that this Interceptor doesn't take any fieldName parameter like the docs specify. This interceptor processes all files and adds the first file on the request body for the UploadedFile decorator (if you will use it). It also adds all files for the UploadedFiles decorator. The difference is in the s.

One last thing. The following stores the file directly in memory (as a buffer). It doesn't store to disk like other examples. This was what I needed. Tweak to your convenience.

// file.interceptor.ts
import {
  CallHandler,
  ExecutionContext,
  NestInterceptor,
  Type,
  mixin,
} from '@nestjs/common';
import Busboy from 'busboy';
import { Observable } from 'rxjs';

export function FileInterceptor(): Type<NestInterceptor> {
  class MixinInterceptor implements NestInterceptor {
    async intercept(
      context: ExecutionContext,
      next: CallHandler,
    ): Promise<Observable<any>> {
      const request = context.switchToHttp().getRequest();

      if (!request.is('multipart/*')) return next.handle();

      const busboy = Busboy({ headers: request.headers });
      const fields = {};
      const files = [];

      return new Promise((resolve, reject) => {
        busboy.on(
          'file',
          (fieldname, file, { filename: originalName, encoding, mimeType }) => {
            const fileInfo = { fieldname, originalName, encoding, mimeType };
            const buffer = [];
            file
              .on('error', reject)
              .on('data', (data) => buffer.push(data))
              .on('close', () => {
                fileInfo['buffer'] = Buffer.concat(buffer);
                fileInfo['size'] = fileInfo['buffer'].length;
                files.push(fileInfo);
              });
          },
        );
        busboy.on('field', (name, val) => (fields[name] = val));
        busboy.on('error', reject);
        busboy.on('close', () => {
          request.body = fields;
          request.file = files[0];
          request.files = files;
          resolve(next.handle());
        });

        if (process.env.FIREBASE_CONFIG) {
          busboy.end(request.rawBody);
        } else {
          request.pipe(busboy);
        }
      });
    }
  }
  const Interceptor = mixin(MixinInterceptor);
  return Interceptor;
}

As for the check for FIREBASE_CONFIG, that's actually the reason why Google's environment is incompatible with multer. Multer uses request.pipe(busboy). Somehow, I suppose that because of Google's manipulation of the request headers, the above doesn't work. In the Google Cloud Docs, they specify busboy.end(request.rawBody).

I maintained both under the config check because Google's solution seems not work when NestJS is not running in Google's environment 🥴

// files.controller.ts
import {
  Body,
  Controller,
  FileTypeValidator,
  MaxFileSizeValidator,
  ParseFilePipe,
  Post,
  UploadedFile,
  UseInterceptors,
} from '@nestjs/common';
import { FileInfo } from './file-info';
import { FileInterceptor } from './file.interceptor';
import { UploadFileDto } from './upload-file.dto';

@Controller('files')
export class FilesController {
  @Post()
  @UseInterceptors(FileInterceptor())
  async upload(
    @Body() body: UploadFileDto,
    @UploadedFile(
      new ParseFilePipe({
        validators: [
          new MaxFileSizeValidator({
            maxSize: 1000000,
            message: 'File should be less than 1MB please',
          }),
          new FileTypeValidator({ fileType: 'image/jpeg' }),
        ],
      }),
    )
    file: FileInfo,
  ) {
    console.log({...body, filename: file.originalName })
  }
}

Like others did, let me leave the following solution for whom it might benefit.

The solutions of others are very okay. But I wanted something very close to what NestJS provides. At the same time, I wanted to be so close to the solution that Google Cloud or Firebase provided. I also didn't want to be storing the files on disk. I just wanted them to be available in the request body.

You will first need some FileInfo interface that is kind of equivalent to Express.Multer.File (you will understand if you are coming from NestJS's Docs)
// file-info.ts
export interface FileInfo {
  fieldName: string;
  originalName: string;
  encoding: string;
  mimeType: string;
  size: number;
  buffer: Buffer;
}
The following FileInterceptor decorator closely resembles what the docs above specify. It is modeled against the source code of the Interceptor exported from the @nestjs/common package.

Note that this Interceptor doesn't take any fieldName parameter like the docs specify. This interceptor processes all files and adds the first file on the request body for the UploadedFile decorator (if you will use it). It also adds all files for the UploadedFiles decorator. The difference is in the s.

One last thing. The following stores the file directly in memory (as a buffer). It doesn't store to disk like other examples. This was what I needed. Tweak to your convenience.
// file.interceptor.ts
import {
  CallHandler,
  ExecutionContext,
  NestInterceptor,
  Type,
  mixin,
} from '@nestjs/common';
import Busboy from 'busboy';
import { Observable } from 'rxjs';

export function FileInterceptor(): Type<NestInterceptor> {
  class MixinInterceptor implements NestInterceptor {
    async intercept(
      context: ExecutionContext,
      next: CallHandler,
    ): Promise<Observable<any>> {
      const request = context.switchToHttp().getRequest();

      if (!request.is('multipart/*')) return next.handle();

      const busboy = Busboy({ headers: request.headers });
      const fields = {};
      const files = [];

      return new Promise((resolve, reject) => {
        busboy.on(
          'file',
          (fieldname, file, { filename: originalName, encoding, mimeType }) => {
            const fileInfo = { fieldname, originalName, encoding, mimeType };
            const buffer = [];
            file
              .on('error', reject)
              .on('data', (data) => buffer.push(data))
              .on('close', () => {
                fileInfo['buffer'] = Buffer.concat(buffer);
                fileInfo['size'] = fileInfo['buffer'].length;
                files.push(fileInfo);
              });
          },
        );
        busboy.on('field', (name, val) => (fields[name] = val));
        busboy.on('error', reject);
        busboy.on('close', () => {
          request.body = fields;
          request.file = files[0];
          request.files = files;
          resolve(next.handle());
        });

        if (process.env.FIREBASE_CONFIG) {
          busboy.end(request.rawBody);
        } else {
          request.pipe(busboy);
        }
      });
    }
  }
  const Interceptor = mixin(MixinInterceptor);
  return Interceptor;
}
As for the check for FIREBASE_CONFIG, that's actually the reason why Google's environment is incompatible with multer. Multer uses request.pipe(busboy). Somehow, I suppose that because of Google's manipulation of the request headers, the above doesn't work. In the Google Cloud Docs, they specify busboy.end(request.rawBody).

I maintained both under the config check because Google's solution seems not work when NestJS is not running in Google's environment 🥴
// files.controller.ts
import {
  Body,
  Controller,
  FileTypeValidator,
  MaxFileSizeValidator,
  ParseFilePipe,
  Post,
  UploadedFile,
  UseInterceptors,
} from '@nestjs/common';
import { FileInfo } from './file-info';
import { FileInterceptor } from './file.interceptor';
import { UploadFileDto } from './upload-file.dto';

@Controller('files')
export class FilesController {
  @Post()
  @UseInterceptors(FileInterceptor())
  async upload(
    @Body() body: UploadFileDto,
    @UploadedFile(
      new ParseFilePipe({
        validators: [
          new MaxFileSizeValidator({
            maxSize: 1000000,
            message: 'File should be less than 1MB please',
          }),
          new FileTypeValidator({ fileType: 'image/jpeg' }),
        ],
      }),
    )
    file: FileInfo,
  ) {
    console.log({...body, filename: file.originalName })
  }
}

I was stuck for almost a week because of this issue and I can confirm that this is still a valid and elegant solution. There is only a small typo, on file-info.ts you define fieldName but on file.interceptor.ts you are using fieldname, fixed that modifying to { fieldName: fieldname, originalName, encoding, mimeType }.

nestjs / nest