Open nicprov opened 1 year ago
After digging a littler deeper into the issue, I found that commenting out "NextMarker" fixes the issue which is odd.
doListBucketTagSpecs = [ ( "Name", Required ) , ( "Prefix", Optional ) --, ( "NextMarker", Optional ) , ( "MaxKeys", Required ) , ( "IsTruncated", Required ) , ( "Contents", Multiple ) , ( "Marker", Optional ) ]
That being said, that tag is necessary for fetching subsequent keys (ie. the next 1000 keys) so it's not really a solution.
It looks like S3 is returning Marker
and NextMarker
after Contents
. Try this:
listBucketTagSpecs : List TagSpec
listBucketTagSpecs =
[ ( "Name", Required )
, ( "Prefix", Optional )
, ( "MaxKeys", Required )
, ( "IsTruncated", Required )
, ( "Contents", Multiple )
, ( "Marker", Optional )
, ( "NextMarker", Optional )
]
@billstclair this fixed the issue, thanks a lot! I didn't realise the ordering mattered here, I thought it was just fetching the tag based on the name.
I opened a pull request with this fix, though I applied it to doListBucketTagSpecs instead of listBucketTagSpecs since this issue is with Digital Ocean Spaces.
I've been having an issue with the listKeys method under certain conditions. From what I can tell, I get an error if the result set is truncated. This happens if the total number of items in the bucket exceeds 1000 (since the default maxKeys value is 1000) or if I specify a maxKeys value below the total number of items in the bucket.
Here is the (truncated) error that I receive:
HttpError (BadBody "Parse Error: Problem with the value at json.ListBucketResult:\n\n [\n {\n \"Name\": \"test\"\n },\n {\n \"Prefix\": null\n },\n {\n \"MaxKeys\": 10\n },\n {\n \"IsTruncated\": \"true\"\n },\n {\n \"Contents\": [\n {\n (...) ,\n {\n \"ETag\": \"\\\"acb4a00a0a6d35cccb2500aba9ffe3bc\\\"\"\n },\n {\n \"Size\": 62417\n },\n {\n \"StorageClass\": \"STANDARD\"\n },\n {\n \"Owner\": [\n {\n \"ID\": 12783012\n },\n {\n \"DisplayName\": 12783012\n }\n ]\n },\n {\n \"Type\": \"Normal\"\n }\n ]\n },\n {\n \"Marker\": null\n },\n {\n \"NextMarker\": \"3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0\"\n }\n ]\n\nTag not found: MaxKeys")
I've looked at the raw output sent from the server, and the MaxKeys tag is definitely present. For reference, here's the raw (truncated) output:
<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>test</Name><Prefix></Prefix><MaxKeys>10</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>2b2ofstr1to1heq9hdgqfcdtbu30poe5tuum5amltfb1en0nl2trsafv2qjk3eh3scsesk3vmfltvhgu58kfo31j7hl579i076vpj90</Key><LastModified>2023-01-07T22:17:03.150Z</LastModified><ETag>"44188a34d1fde4d0210d0ddb0270803a"</ETag><Size>352880</Size><StorageClass>STANDARD</StorageClass><Owner><ID>12783012</ID><DisplayName>12783012</DisplayName></Owner><Type>Normal</Type></Contents>(...)<Key>3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0</Key><LastModified>2023-01-08T17:40:54.509Z</LastModified><ETag>"acb4a00a0a6d35cccb2500aba9ffe3bc"</ETag><Size>62417</Size><StorageClass>STANDARD</StorageClass><Owner><ID>12783012</ID><DisplayName>12783012</DisplayName></Owner><Type>Normal</Type></Contents><Marker></Marker><NextMarker>3g3u6jbkch3va1bjifi2d7fdn0/tifpvfgug0kcv2iv2pvuda7dms/sh12elh5nmn9fdlfim0uevcjbcpfp2jsfd9ufduhnnd64lb0clc0</NextMarker></ListBucketResult>
This therefore seems to be an XML parsing bug somehow.