search

JuliaCloud / AWS.jl

Julia interface to AWS
MIT License
159 stars 62 forks source link

S3.list_objects_v2 does not handle permanent redirects to other regions #500

Open rustyconover opened 2 years ago

rustyconover commented 2 years ago

My default profile uses region us-east-1 but I'm using a bucket in us-east-2.

julia> S3.list_objects_v2("rusty-data-first-step")
OrderedCollections.LittleDict{Union{String, Symbol}, Any, Vector{Union{String, Symbol}}, Vector{Any}} with 6 entries:
  "Code"      => "PermanentRedirect"
  "Message"   => "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future re…
  "Endpoint"  => "rusty-data-first-step.s3.us-east-2.amazonaws.com"
  "Bucket"    => "rusty-data-first-step"
  "RequestId" => "PKAY0PBYZ32EEVRB"
  "HostId"    => "HWok4bGZDkLjS1RhAjnQpRvMjAwbYD/0pYJdkpfijhu0dPkzntTAcyKhDP7GGXZvd2pWdhp2l0o="

There doesn't seem to be any code in the module that remembers the redirects for buckets in different regions and automatically uses those endpoints.

Also its not entirely clear how to set the correct endpoint that should be used explicitly.

mattBrzezinski commented 2 years ago

I don't believe we have a way currently to automatically just re-make the request pointing to the correct location. The easiest way to get this to work knowing ahead of time of the proper region would be to make the request as,

S3.list_objects_v2("rusty-data-first-step"; aws_config=AWSConfig(; region="us-east-2"))

It should be an easy enough change to see if the response is a PermanantRedirect, and just changing the endpoint in the request to response["Endpoint"] and re-sending it.