benmcclelland
commented
1 month ago First pass of these fixes merged. We are tracking one more issue with the list objects v2 continuation marker in #770 that is related to this issue.
Closed tonyipm closed 1 month ago
benmcclelland
commented
1 month ago First pass of these fixes merged. We are tracking one more issue with the list objects v2 continuation marker in #770 that is related to this issue.
Describe the bug aws s3 ls has an option 'page-size' which causes the cli to request object listings by splitting the transfer over multiple http GET requests. The aws docs state that when listing items using --page-size all items will still be listed, the change is only in using multiple smaller network requests instead of one large one. In Versity if I specify --page-size from the standard aws cli then I don't get all the items back, seemingly just the first 'page'.
To Reproduce Upload 4 items to an empty bucket: vgaws s3api put-object --bucket tonyitest --key d1/f1 --body some_file (repeat above for d2/f2-d4/f4) List without --page-size: DEV03 [/mmfs1/data/TISRC] :>vgaws s3 ls tonyitest PRE d1/ PRE d2/ PRE d3/ PRE d4/
List with --page-size=3: DEV03 [/mmfs1/data/TISRC] :>vgaws s3 ls tonyitest --page-size=3 PRE d1/ PRE d2/ PRE d3/
Expected behavior All 4 items should be listed, as happens with aws s3
Server Version Version : Build : release-rpm BuildTime: 2024-08-27_09:06:33AM
Additional context I traced the behaviour with wireshark and I can see that when using --page-size, aws s3 returns the first page of items and includes a value for NextContinuationToken in returned xml data. It also returns a header 'Transfer-Encoding: chunked'. Aws cli then sends another GET request quoting the continuation token, whereupon it receives the next (final in my case) page.
With Versity I just see the first page of results and no continuation token which presumably prevents the cli from requesting further data. There is also no 'Transfer-Encoding: chunked' header.