findify / s3mock

Embedded S3 server for easy mocking
MIT License
386 stars 107 forks source link

ListObjectsV2Response doesn't include nextContinuationToken when isTruncated is true #144

Open er1c opened 5 years ago

er1c commented 5 years ago

Looks like this isn't being populated, and a quick code search doesn't show any references to nextContinuationToken

image

joshrosen-stripe commented 5 years ago

I just ran into a similar problem and investigated a bit further. I think the bug here is related to s3mock's (lack of) handling of continuation tokens passed in client requests by the client, not the generation of the tokens themselves:

Quoting from the code:

 @Override
        protected void doEndElement(String uri, String name, String qName) {
            if (atTopLevel()) {
                if (name.equals("ListBucketResult")) {
                    /*
                     * S3 only includes the NextContinuationToken XML element if the
                     * request specified a delimiter, but for consistency we'd
                     * like to always give easy access to the next token if
                     * we're returning a list of results that's truncated.
                     */
                    if (result.isTruncated()
                            && result.getNextContinuationToken() == null) {

                        String nextContinuationToken = null;
                        if (!result.getObjectSummaries().isEmpty()) {
                            nextContinuationToken = result.getObjectSummaries()
                                    .get(result.getObjectSummaries().size() - 1)
                                    .getKey();

                        } else {
                            log.error("S3 response indicates truncated results, "
                                    + "but contains no object summaries.");
                        }

                        result.setNextContinuationToken(nextContinuationToken);
                    }
                }
            }

In my case, I was performing a listing with setMaxKeys() but no delimiter, causing me to fall into this branch. I received no token back in the response but the client code above generated a token. (which i passed back to the server in a second request). However, the mock server ignored this token, causing my listing-to-iterator code to go into an infinite loop.