S3 URL download failure when forcing s3 protocol

[jrasell]

The README states that you can force a protocol by prefixing the URL, such as s3:: in the case of using the s3 protocol. In my situation the following (edited) artifact URL is valid and works for downloading over HTTPS via go-getter: https://some-bucket-name.s3-accelerate.amazonaws.com/packagename/1.0.0/packagename-linux-amd64.tar.gz

When forcing the protocol to s3, the download fails:

go run go-getter/main.go s3::https://some-bucket-name.s3-accelerate.amazonaws.com/packagename/1.0.0/packagename-linux-amd64.tar.gz download
2019/01/07 09:30:15 Error downloading: URL is not a valid S3 URL
exit status 1

I would expect that when forcing the protocol the download will still work, however, maybe I am missing something here.

[gurre]

Does it work if you remove the s3::?

[azr]

Ah it looks like that because the URL contains dots this check fails:

https://github.com/hashicorp/go-getter/blob/9018a73beec16a02695da71f7530dc84b8ec3de9/detect_s3.go#L32-L40

Edit: ah, sorry, no never-mind the url is first split on slashes. is not a valid S3 URL matches a lot of errors :)

[thefallentree]

right now specifying a s3-accelerate endpoint still doesn't work : it tries to download from s3.accerlate.awsnaws.com instead of the correct one "s3-accerlate.awsnaws.com"

Using https doesn't work either because of the request needs aws headers.

[judell]

I'm having the same issue.

See: https://github.com/hashicorp/go-getter/issues/148

And: https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5Egithub%5C.com%2Fhashicorp%2Fgo-getter%24+file%3A%5Edetect_s3%5C.go+URL+is+not+&patternType=standard&sm=1&groupBy=path

[judell]

Per https://github.com/turbot/steampipe-plugin-sdk/blob/main/plugin/get_source_files_test.go#L135-L191, I have found it necessary to use the syntax that embeds the region. Does that help in your cases, @jrasell, @thefallentree?