search

anacronw / multer-s3

multer storage engine for amazon s3
MIT License
662 stars 193 forks source link

Header x-amz-meta-destination contains invalid value #178

Open Daniel-M opened 2 years ago

Daniel-M commented 2 years ago

Hello, I've been trying to use this module to work against Digital Ocean Spaces, provided they follow the AWS S3 specification, but I've been facing an error using either the V2 or V3 versions (with the corresponding version of the AWS SDK for NodeJS)

My implementation goes like this,

Using version 3.0.1 of multer-s3 with @aws-sdk/client-s3 version 3.105.0

const multer = require('multer')
const multerS3 = require('multer-s3')
const { S3Client }= require('@aws-sdk/client-s3')

// Create a S3Client with the proper region
const s3 = new S3Client({
  endpoint:  'https://sfo3.digitaloceanspaces.com',
  region: 'sfo3', // The region as per Digital Ocean docs
  // Authorization credentials as per Digital Ocean Docs
  // https://docs.digitalocean.com/products/spaces/resources/s3-sdk-examples/#configure-a-client
  credentials: { 
    accessKeyId: process.env.DO_SPACES_KEY,
    secretAccessKey: process.env.DO_SPACES_SECRET,
  }
})

Using version 2.10.0 of multer-s3 with aws-sdk version 2.1150.0

const { S3 }= require('aws-sdk')

const multer = require('multer')
const multerS3 = require('multer-s3')

// Create S3 Instance
const s3 = new S3({
  endpoint: 'https://sfo3.digitaloceanspaces.com',
  region: 'sfo3',
  credentials: {
    accessKeyId: process.env.DO_SPACES_KEY,
    secretAccessKey: process.env.DO_SPACES_SECRET,
  }
})

Now we can instantiate the multer storage engine

const storage = multerS3({
  s3,
  bucket: process.env.DO_SPACES_BUCKET_NAME, // The bucket name
  acl: process.env.DO_SPACES_BUCKET_ACL_POLICY, // 'public-read'
  contentType: multerS3.AUTO_CONTENT_TYPE,
  metadata: function (req, file, cb) {
    // Non relevant to the issue
    cb(null, metadata);
  },
  key: function (req, file, cb) {
    // Non relevant to the issue
    cb(null, fileDest)
  }
})

// S3 uploader
const s3Uploader = multer({
  storage,
})

Then define a request handler,

const uploadHandler = async function (req, res, next) {
  // The request handler that notifies the client that the file has been uploaded
  res.status(200).send('uploaded!')
}

And finally use both the s3Uploader and handler as a request chain

// Configure the express app to use the endpoint,
uploadApp.post('/endpoint/:id/image', s3Uploader.single('file'), uploadHandler)

When uploading a multipart file using Postman I got the following error on the server,

Using version 2.10.0 of multer-s3 with aws-sdk version 2.1150.0

InvalidHeader: Header x-amz-meta-destination contains invalid value
    at V4.<anonymous> (/app/node_modules/aws-sdk/lib/signers/v4.js:144:32)
    at V4.arrayEach (/app/node_modules/aws-sdk/lib/util.js:521:32)
    at V4.canonicalHeaders (/app/node_modules/aws-sdk/lib/signers/v4.js:139:24)
    at V4.canonicalString (/app/node_modules/aws-sdk/lib/signers/v4.js:124:21)
    at V4.stringToSign (/app/node_modules/aws-sdk/lib/signers/v4.js:113:41)
    at V4.signature (/app/node_modules/aws-sdk/lib/signers/v4.js:105:50)
    at V4.authorization (/app/node_modules/aws-sdk/lib/signers/v4.js:93:36)
    at V4.addAuthorization (/app/node_modules/aws-sdk/lib/signers/v4.js:35:12)
    at /app/node_modules/aws-sdk/lib/event_listeners.js:282:18
    at finish (/app/node_modules/aws-sdk/lib/config.js:396:7)
    at getStaticCredentials (/app/node_modules/aws-sdk/lib/config.js:423:7)
    at Config.getCredentials (/app/node_modules/aws-sdk/lib/config.js:430:9)
    at Request.SIGN (/app/node_modules/aws-sdk/lib/event_listeners.js:258:22)
    at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:102:18)
    at callNextListener (/app/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
    at Request.discoverEndpoint (/app/node_modules/aws-sdk/lib/discover_endpoint.js:326:67)

Using version 3.0.1 of multer-s3 with @aws-sdk/client-s3 version 3.105.0

TypeError: Cannot read property 'trim' of undefined
    at getCanonicalHeaders (/app/node_modules/@aws-sdk/signature-v4/dist-cjs/getCanonicalHeaders.js:17:62)
    at SignatureV4.signRequest (/app/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:97:80)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async /app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:18
    at async StandardRetryStrategy.retry (/app/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:56:20
    at async /app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async Promise.all (index 0)
    at async Upload.__uploadUsingPut (/app/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:46:39)
    at async Upload.__doConcurrentUpload (/app/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:91:28)
    at async Promise.all (index 0)
    at async Upload.__doMultipartUpload (/app/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:141:9)
    at async Upload.done (/app/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:37:16)

Upon inspection of node_modules/@aws-sdk/signature-v4/dist-cjs/getCanonicalHeaders.js and a little modification I found that

console.log(headers) // Added this to debug the issue
// Line 17
canonical[canonicalHeaderName] = headers[headerName].trim().replace(/\s+/g, " ");

Shows the following set of headers

{
  'content-type': 'image/jpeg',
  'x-amz-acl': 'public-read',
  'x-amz-storage-class': 'STANDARD',
  'x-amz-meta-originalname': 'photo_2022-05-15_11-37-20.jpg',
  'x-amz-meta-encoding': '7bit',
  'x-amz-meta-mimetype': 'image/jpeg',
  'x-amz-meta-destination': undefined,
  'x-amz-meta-filename': undefined,
  'x-amz-meta-size': undefined,
  'content-length': '119360',
  Expect: '100-continue',
  host: 'xxxxxx.sfo3.digitaloceanspaces.com',
  'x-amz-user-agent': 'aws-sdk-js/3.105.0',
  'user-agent': 'aws-sdk-js/3.105.0 os/linux/5.17.0-1-amd64 lang/js md/nodejs/14.17.5 api/s3/3.105.0',
  'amz-sdk-invocation-id': '4ff84ad5-62e8-450e-94d5-bf78b8d7abc0',
  'amz-sdk-request': 'attempt=1; max=3',
  'x-amz-date': '20220608T154617Z',
  'x-amz-content-sha256': '265a99c3f998bab86578eeea7f9f37ea3fe093834150b0f2f6decd817964cbba'
}

Having the set

'x-amz-meta-destination': undefined,
'x-amz-meta-filename': undefined,
'x-amz-meta-size': undefined,

Which confirms the error raised on v2 case. What am I doing wrong?, is this an issue?