andrewrk / node-s3-client

high level amazon s3 client for node.js
MIT License
1k stars 303 forks source link

inconsistent results for client.listObjects(params), depending on EncodingType option #182

Open bradwbradw opened 7 years ago

bradwbradw commented 7 years ago

If I call without setting the s3Params.EncodingType option, i get 1277 objects.

If I call with s3Params.EncodingType set to'url', i get 1124 objects.

I don't understand how there would be different object lists depending on how I set the encodingType?

My code is just a very basic promise wrapper around listObjects that pushes onto an array, then listens for 'end' and returns the array

    when.promise((resolve, reject) => {

      let list = [];
      let getURlEncodedList = S3Client.listObjects({
        s3Params: {
          Bucket: config.S3.bucket,
          EncodingType: 'url' // <---- if omitting this line I get more objects in the list array!
        }
      });

      getURlEncodedList.addListener('end', function () {
        console.log(`downloaded urlEncoded list: ${ _.size(list)} songs found.`);
        resolve(list);
      });

      getURlEncodedList.addListener('data', function (items) {
        _.each(items.Contents, item => {
            list.push(item);
        });
      });
    })

if i call the above alongside a version without EncodingType, i get this output:

downloaded urlEncoded list: 1124 songs found.
downloaded non-urlEncoded list: 1277 songs found.

I hypothesized that it might be because I was using when.all and calling each version in parallel, causing a conflict in the S3 module, but after rewriting the promises to be sequential, I get the same results.