Listing S3 buckets with responses that require continuation tokens currently throws the exception below. This is typically only seen with S3 buckets with 1000's of objects to list.
Unhandled exception: InvalidArgument: The continuation token provided is incorrect (Awscr::S3::Http::ServerError)
from lib/awscr-s3/src/awscr-s3/http.cr:109:9 in 'handle_response!'
from lib/awscr-s3/src/awscr-s3/http.cr:86:7 in 'get'
from lib/awscr-s3/src/awscr-s3/paginators/list_object_v2.cr:28:7 in 'next_response'
from lib/awscr-s3/src/awscr-s3/paginators/list_object_v2.cr:23:22 in 'next'
from /usr/local/Cellar/crystal/0.31.1/src/iterator.cr:442:7 in '__crystal_main'
from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:97:5 in 'main_user_code'
from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:86:7 in 'main'
from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:106:3 in 'main'
Thanks to the investigation in this AWS forum post, this seems to be because of special characters in the obfuscated continuation token.
Sometimes 1-3 requests pass through until it hits one with a plus in it and fails.
Running URI.encode with space_to_plus: true fixed this error.
Looking for feedback on this approach and whether it is safe to run space_to_plus: true on all the query params or if we should scope it to continuation-token.
Listing S3 buckets with responses that require continuation tokens currently throws the exception below. This is typically only seen with S3 buckets with 1000's of objects to list.
Thanks to the investigation in this AWS forum post, this seems to be because of special characters in the obfuscated continuation token.
Passing token:
1TZkJICXgR0jZrdAHEJMonlSkBedCNzVrXzaGnw8ZBXsL5r5vhMtL7rJeJCTRvl1WYAoW27oMqx/JMbzac2uMqg==
Failing token:1oDMtg+plckMHvLSZvcmKg6Gn5QYOEbCY9z/2gPv9gOXdHJRMSuQ/tibeJlExsXSBJwhmPaiDpMOQOF+c/49MmA==
Sometimes 1-3 requests pass through until it hits one with a plus in it and fails.
Running
URI.encode
withspace_to_plus: true
fixed this error.Looking for feedback on this approach and whether it is safe to run
space_to_plus: true
on all the query params or if we should scope it tocontinuation-token
.