Mocking S3.listObjectsV2().eachPage()

[iamNoah1]

Hei guys,

I have been trying to mock S3.listObjectsV2().eachPage() without success so far. Do you have any examples for that, or is it even possible?

Thank you Cheers

Noah

[zippadd]

No luck here (yet) as well.

[jruts]

Hi @MeBNoah and @zippadd , could you paste your code snippets?

Normally this should work:

AWS.mock('S3.listObjectsV2', 'eachPage', function(params, callback) {
  callback(null, {Item: {Key: 'Value'}});
});

Please let me know if this helps.

[jruts]

Also when I look at the https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property I cannot find this .eachPage function.

The description of listObjectsV2:

Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. Note: ListObjectsV2 is the revised List Objects API and we recommend you use this revised API for new application development.

So is this eachPage a function that exists on the response object of S3.listObjectsV2?

Assuming this function does not exists you will need to mock listObjectsV2 and not eachPage.

AWS.mock('S3', 'listObjectsV2', function(params, callback) {
  callback(null, //a fake response from listObjectsV2);
});

Here are some references to the usage of S3.listObjectsV2:

• https://stackoverflow.com/a/40036025
• https://stackoverflow.com/a/42396925

Kind regards

[zippadd]

Hello! Apologies for the long time getting back to this.

Not sure if @MeBNoah is having the same error, but this is what I'm getting: [TypeError: s3.listObjectsV2(...).eachPage is not a function]

The .eachPage function is part of AWS.Request, similar to .promise. When a callback to one of the services is omitted, an AWS.Request is generated, but isn't sent, allowing the use of these other methods or manually using .send.

I figured a snippet may be a bit much to post and isn't as readily reproducible, so I created a test repo with a basic handler/function and the test file and a Travis CI setup. I show both a passing test using .promise and a failing one using .eachPage.

I suppose one could mock out the .eachPage function, but I'm looking for that to function normally so that the handling of the eachPage callback can be properly tested, similar to not needing to mock .promise.

I tried your initial snippet of 'S3.listObjectsV2', 'eachPage', but that led to an error as well, which makes sense as .eachPage is a part of AWS.Request.

Hope this helps! Appreciate you taking the time to look at this.

[jruts]

Hi @zippadd,

I understand what you mean now. I have read the documentation of the AWS.Request page and understand the eachPage function now.

The only way to accommodate this right now is to mock out s3.listObjectsV2 and return back a mock of the AWS.Request object, as you already mentioned.

This module does not yet take into account some of the custom response objects from the AWS SDK and there are no plans for it yet.

Do feel free to fork and make a PR if you can add an elegant solution for this.

Sorry if this is not the answer you hoped for, but I do hope aws-sdk-mock can still help you to move forward @zippadd and @MeBNoah .

Kind regards.

[iamNoah1]

Hei guys,

@zippadd yes I got the same error. @jruts thanks for your efforts.

right now it is not critical for me, but I will try to find a solution or to make a PR.

Cheers Noah

[alexanderturinske]

Commenting here because this seems to be the only place I can find where ppl are discussing mocking eachPage.

I am having trouble simply getting eachPage to get called more than once; when I mock it out, I mock it with const eachPageStub = sinon.stub().onFirstCall(console.log('first')).yields(null, '1').onSecondCall(console.log('second')).yields(null, '2') 'first' AND 'second' get output, but I put a console.count() in my function and it only gets called once. How to I mock out the eachPage function being called more than once?

EDIT: Someone much smarter than I helped me out.

const eachPageStub = sinon.spy(callback => {
    callback(null, firstDataSet);
    callback(null, SecondDataSet);
});

The problem was that I was thinking that eachPage is what gets called multiple times, but really it is the callback that gets called multiple times.

[matthias-pichler]

@alexanderturinske could you explained how you ended up mocking eachPage with aws-sdk-mock?

[temyers]

Hi is there any update on this?

Whilst waiting for #195, is there a workaround using aws-sdk-mock?

Given the following function:

getPipelines(state) {
        return new Promise(function (resolve, reject) {
            state.cloudwatch.listMetrics({ "Namespace": "Pipeline" }).eachPage(function (err, data) {
                  console.log(data);
            });
        });
};

Can someone provide an example?

[alexanderturinske]

@Matthias-Pichler , the truth is I mocked it without using aws-sdk-mock because there was no solution with aws-sdk-mock at the time. Instead I removed aws-sdk-mock and used sinon to achieve the goal.

    const Helper.listObjectsV2 = ({ Keys, Prefix = '', eachPageStub = '' }) => ({
        ...
        eachPage: eachPageStub,
    });

    const eachPageStub = (pages, Keys, otherKeys) => sinon.spy(callback => {
        if (pages) {
            callback(null, Helper.listObjectsV2({ Keys }));

            if (otherKeys) {
                callback(null, Helper.listObjectsV2({ Keys: otherKeys }));
            }
        }

        callback(null, null);
    });

    const listObjectsV2Mock = (pages, Keys, otherKeys) => ({ Bucket, Prefix }) => {
        expect(Bucket).toBeDefined();
        return Helper.listObjectsV2({
            Keys,
            Prefix,
            eachPageStub: eachPageStub(pages, Keys, otherKeys),
        });
    };

    const s3Stub = sinon.stub(AWS, 'S3');

    const setupMock = (pages = 1, data = [], otherData = null) => {
        s3Stub.returns({ listObjectsV2: listObjectsV2Mock(pages, data, otherData) });
    };

    test('lists versions for multiple pages that are not returned in order', async () => {
        setupMock(1, [
            '0.0.1/thing.tar.gz',
            '0.0.1/thing.tar.gz',
            '3.0.0/thing.tar.gz',
            '3.0.0/thing.tar.gz',
        ], [
            '0.0.4/thing.tar.gz',
            '0.0.4/thing.tar.gz',
            '2.0.0/thing.tar.gz',
            '2.0.0/thing.tar.gz',
        ]);

        const store = new S3Backend('');
        expect(await store.versions()).toMatchObject(['0.0.1', '3.0.0', '0.0.4', '2.0.0']);
    });

[jgriepentrog]

@jruts - it looks like #195 was submitted to solve this. Can you advise on next steps for getting the changes reviewed and potentially merged?