Credentialless interface via custom AbstractAWSConfig

[yakir12]

I broke this off of #117.

A use-case I'd like to address is a user without any aws-credentials (or indeed any knowledge of AWS) listing and getting files from an AWS repo that allows such public access.

It seems like a custom instance of <:AbstractAWSConfig would solve this, but I don't understand how to make it work for an AWS repo. I'll even suggest that this would be a relatively common need and that AWSS3.jl could have its own implementation of such a type: a credential-less type that allows listing and getting from a public S3 AWS repository. But it would be great to see an example of this, and if there is an interest I'll gladly PR the implementation+docs+tests.

Currently, I have:

struct AnonymousGCS <:AbstractAWSConfig end
struct NoCredentials end
AWS.region(aws::AnonymousGCS) = "" # No region
AWS.credentials(aws::AnonymousGCS) = NoCredentials() # No credentials
AWS.check_credentials(c::NoCredentials) = c # Skip credentials check
AWS.sign!(aws::AnonymousGCS, ::AWS.Request) = nothing # Don't sign request
function AWS.generate_service_url(aws::AnonymousGCS, service::String, resource::String)
    service == "s3" || throw(ArgumentError("Can only handle s3 requests to GCS"))
    return string("https://s3.$bucket.amazonaws.com.", resource)
end
config = AWS.global_aws_config(AnonymousGCS())

but listing or getting doesn't work.

[mattBrzezinski]

I can play around with this, I believe this package just needs to be updated to take in an AWS.AbstractAWSConfig in the functions rather than AWS.AWSConfig and the above credential-less struct will work.

[mattBrzezinski]

This package already makes use of AWS.AbstractAWSConfig, it looks like there are a few issues with the example code that you posted, but I modified it and have it working below.

using AWS
using AWSS3
using JSON

@service S3

struct AnonymousGCS <: AbstractAWSConfig end
struct NoCreds end  # NoCredentials is an exported struct from AWS.AWSExceptions

AWS.region(aws::AnonymousGCS) = "us-east-1" # No region
AWS.credentials(aws::AnonymousGCS) = NoCreds() # No credentials
AWS.check_credentials(c::NoCreds) = c # Skip credentials check
AWS.sign!(aws::AnonymousGCS, ::AWS.Request) = nothing # Don't sign request

function AWS.generate_service_url(aws::AnonymousGCS, service::String, resource::String)
    service == "s3" || throw(ArgumentError("Can only handle s3 requests to GCS"))
    region = AWS.region(aws)
    return string("https://s3.$region.amazonaws.com", resource)
end

config = AWS.global_aws_config(AnonymousGCS())

### Example using AWSS3.jl
r  = s3_list_objects(config, "ryft-public-sample-data")

for i in r
  JSON.print(i, 2)
end

### Example using AWS.jl
r = S3.list_objects("ryft-public-sample-data")
JSON.print(r["Contents"][1:3], 2)

[yakir12]

This totally worked. That's amazing. Spoke too soon. So while listing works great, actually downloading a file fails:

julia> bucket = "ryft-public-sample-data"#"nicolas-cage-skyroom"
"ryft-public-sample-data"

julia> file = r["Contents"][1]["Key"]
"AWS-x86-AMI-queries.json"

julia> s3_get_file(config, bucket, file)
ERROR: AWSException: 400 -- AWSException
HTTP.ExceptionRequest.StatusError(400, "GET", "/AnonymousGCS%28%29/ryft-public-sample-data", HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
x-amz-request-id: M55P79KFAJ4HR87V
x-amz-id-2: SWLgYibLxf7J7XF9ORa/CjnDpr4qQgZd8lEh1rculolet5vxXLMczS/2eGFO4AHpYQjhYGZRqKk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Mon, 12 Apr 2021 08:26:48 GMT
Server: AmazonS3
Connection: close

[Message Body was streamed]""")

Stacktrace:
  [1] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :require_ssl_verification, :response_stream), Tuple{Nothing, Bool, Base.BufferStream}}})
    @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/cxgat/src/ExceptionRequest.jl:22
  [2] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:require_ssl_verification, :response_stream), Tuple{Bool, Base.BufferStream}}})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/cxgat/src/MessageRequest.jl:66
  [3] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:require_ssl_verification, :response_stream), Tuple{Bool, Base.BufferStream}}})
    @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/cxgat/src/BasicAuthRequest.jl:28
  [4] macro expansion
    @ ~/.julia/packages/AWS/ASX1y/src/AWS.jl:336 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/Retry/vS1bg/src/repeat_try.jl:192 [inlined]
  [6] _http_request(request::Request)
    @ AWS ~/.julia/packages/AWS/ASX1y/src/AWS.jl:329
  [7] macro expansion
    @ ~/.julia/packages/Mocking/U41JO/src/mock.jl:29 [inlined]
  [8] macro expansion
    @ ~/.julia/packages/AWS/ASX1y/src/AWS.jl:397 [inlined]
  [9] macro expansion
    @ ~/.julia/packages/Retry/vS1bg/src/repeat_try.jl:192 [inlined]
 [10] submit_request(aws::AnonymousGCS, request::Request; return_headers::Bool)
    @ AWS ~/.julia/packages/AWS/ASX1y/src/AWS.jl:395
 [11] (::RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Bool}; aws_config::AnonymousGCS)
    @ AWS ~/.julia/packages/AWS/ASX1y/src/AWS.jl:600
 [12] #get_object#90
    @ ~/.julia/packages/AWS/ASX1y/src/services/s3.jl:1729 [inlined]
 [13] s3_get_file(aws::AnonymousGCS, bucket::AnonymousGCS, path::String, filename::String; version::String, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWSS3 ~/.julia/packages/AWSS3/H9WJU/src/AWSS3.jl:100
 [14] s3_get_file(aws::AnonymousGCS, bucket::AnonymousGCS, path::String, filename::String)
    @ AWSS3 ~/.julia/packages/AWSS3/H9WJU/src/AWSS3.jl:100
 [15] s3_get_file(::AnonymousGCS, ::Vararg{Any, N} where N; b::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWSS3 ~/.julia/packages/AWSS3/H9WJU/src/AWSS3.jl:109
 [16] s3_get_file(::AnonymousGCS, ::String, ::String)
    @ AWSS3 ~/.julia/packages/AWSS3/H9WJU/src/AWSS3.jl:109
 [17] top-level scope
    @ REPL[127]:1

Once this becomes relevant, I'm not sure how to add this to the docs... They seem to be entirely consisted of docstrings of existing functions. Or do you want to wrap this in some convenience function (not sure how though)?