taylorfinnell / awscr-s3

A Crystal shard for S3.
https://taylorfinnell.github.io/awscr-s3/
MIT License
81 stars 48 forks source link

Fix continuation token invalid error #61

Closed mattprivman closed 4 years ago

mattprivman commented 4 years ago

Listing S3 buckets with responses that require continuation tokens currently throws the exception below. This is typically only seen with S3 buckets with 1000's of objects to list.

Unhandled exception: InvalidArgument: The continuation token provided is incorrect (Awscr::S3::Http::ServerError)
  from lib/awscr-s3/src/awscr-s3/http.cr:109:9 in 'handle_response!'
  from lib/awscr-s3/src/awscr-s3/http.cr:86:7 in 'get'
  from lib/awscr-s3/src/awscr-s3/paginators/list_object_v2.cr:28:7 in 'next_response'
  from lib/awscr-s3/src/awscr-s3/paginators/list_object_v2.cr:23:22 in 'next'
  from /usr/local/Cellar/crystal/0.31.1/src/iterator.cr:442:7 in '__crystal_main'
  from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:97:5 in 'main_user_code'
  from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:86:7 in 'main'
  from /usr/local/Cellar/crystal/0.31.1/src/crystal/main.cr:106:3 in 'main'

Thanks to the investigation in this AWS forum post, this seems to be because of special characters in the obfuscated continuation token.

Passing token: 1TZkJICXgR0jZrdAHEJMonlSkBedCNzVrXzaGnw8ZBXsL5r5vhMtL7rJeJCTRvl1WYAoW27oMqx/JMbzac2uMqg== Failing token: 1oDMtg+plckMHvLSZvcmKg6Gn5QYOEbCY9z/2gPv9gOXdHJRMSuQ/tibeJlExsXSBJwhmPaiDpMOQOF+c/49MmA==

Sometimes 1-3 requests pass through until it hits one with a plus in it and fails.

Running URI.encode with space_to_plus: true fixed this error.

Looking for feedback on this approach and whether it is safe to run space_to_plus: true on all the query params or if we should scope it to continuation-token.

taylorfinnell commented 4 years ago

I'm not sure if it's ok to use everywhere. Similar uncertainty in #34. I wonder if the ruby implementation would shed some light