listKeys issue - Tag not found: MaxKeys

[nicprov]

I've been having an issue with the listKeys method under certain conditions. From what I can tell, I get an error if the result set is truncated. This happens if the total number of items in the bucket exceeds 1000 (since the default maxKeys value is 1000) or if I specify a maxKeys value below the total number of items in the bucket.

Here is the (truncated) error that I receive: HttpError (BadBody "Parse Error: Problem with the value at json.ListBucketResult:\n\n [\n {\n \"Name\": \"test\"\n },\n {\n \"Prefix\": null\n },\n {\n \"MaxKeys\": 10\n },\n {\n \"IsTruncated\": \"true\"\n },\n {\n \"Contents\": [\n {\n (...) ,\n {\n \"ETag\": \"\\\"acb4a00a0a6d35cccb2500aba9ffe3bc\\\"\"\n },\n {\n \"Size\": 62417\n },\n {\n \"StorageClass\": \"STANDARD\"\n },\n {\n \"Owner\": [\n {\n \"ID\": 12783012\n },\n {\n \"DisplayName\": 12783012\n }\n ]\n },\n {\n \"Type\": \"Normal\"\n }\n ]\n },\n {\n \"Marker\": null\n },\n {\n \"NextMarker\": \"3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0\"\n }\n ]\n\nTag not found: MaxKeys")

I've looked at the raw output sent from the server, and the MaxKeys tag is definitely present. For reference, here's the raw (truncated) output: <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>test</Name><Prefix></Prefix><MaxKeys>10</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>2b2ofstr1to1heq9hdgqfcdtbu30poe5tuum5amltfb1en0nl2trsafv2qjk3eh3scsesk3vmfltvhgu58kfo31j7hl579i076vpj90</Key><LastModified>2023-01-07T22:17:03.150Z</LastModified><ETag>&quot;44188a34d1fde4d0210d0ddb0270803a&quot;</ETag><Size>352880</Size><StorageClass>STANDARD</StorageClass><Owner><ID>12783012</ID><DisplayName>12783012</DisplayName></Owner><Type>Normal</Type></Contents>(...)<Key>3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0</Key><LastModified>2023-01-08T17:40:54.509Z</LastModified><ETag>&quot;acb4a00a0a6d35cccb2500aba9ffe3bc&quot;</ETag><Size>62417</Size><StorageClass>STANDARD</StorageClass><Owner><ID>12783012</ID><DisplayName>12783012</DisplayName></Owner><Type>Normal</Type></Contents><Marker></Marker><NextMarker>3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0</NextMarker></ListBucketResult>

This therefore seems to be an XML parsing bug somehow.

[nicprov]

After digging a littler deeper into the issue, I found that commenting out "NextMarker" fixes the issue which is odd.

doListBucketTagSpecs = [ ( "Name", Required ) , ( "Prefix", Optional ) --, ( "NextMarker", Optional ) , ( "MaxKeys", Required ) , ( "IsTruncated", Required ) , ( "Contents", Multiple ) , ( "Marker", Optional ) ]

That being said, that tag is necessary for fetching subsequent keys (ie. the next 1000 keys) so it's not really a solution.

[billstclair]

It looks like S3 is returning Marker and NextMarker after Contents. Try this:

listBucketTagSpecs : List TagSpec
listBucketTagSpecs =
    [ ( "Name", Required )
    , ( "Prefix", Optional )
    , ( "MaxKeys", Required )
    , ( "IsTruncated", Required )
    , ( "Contents", Multiple )
    , ( "Marker", Optional )
    , ( "NextMarker", Optional )
    ]

[nicprov]

@billstclair this fixed the issue, thanks a lot! I didn't realise the ordering mattered here, I thought it was just fetching the tag based on the name.

I opened a pull request with this fix, though I applied it to doListBucketTagSpecs instead of listBucketTagSpecs since this issue is with Digital Ocean Spaces.