Non standard S3 endpoint download url issue

[JoeeGrigg]

Hi,

I could be wrong but I think there is a problem with generating a url for an uploaded file when using a custom s3 storage endpoint. I am running Minio locally on my machine and uploading works fine. Unfortunately, when I try to request a download url via UploadedFile#url I get a url that doesn't point to my local Minio server.

I am using it within a Lucky project. Below is my configuration and usage.

# config/shrine.cr

client = Awscr::S3::Client.new(
  "minio",
  "minioadmin",
  "minioadmin",
  endpoint: "http://localhost:9000"
)

Shrine.configure do |config|
  config.storages["store"] = Shrine::Storage::S3.new(bucket: "day", client: client)
end

Shrine::UploadedFile.new("7609952bf7b7973b03595c315d728cd9.jpg", "store").url

# => https://s3-minio.amazonaws.com/day/7609952bf7b7973b03595c315d728cd9.jpg?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20210323%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20210323T083243Z&X-Amz-SignedHeaders=host&X-Amz-Signature=5bafad2cb415d929198ba3b0f9ede9fd78943cef30b81e3711ec16b1baeb6ff3

As you can see from the above, I have my endpoint configured to http://localhost:9000 but when I request a url it uses https://s3-minio.amazonaws.com.

Am I doing something wrong or is this a bug? Any help would be appreciated!

[da1nerd]

I had a similar issue in a project of mine. The solution was to patch Shrine's S3 storage class. I was going to open a PR with a fix, but I sort of forgot... :roll_eyes:

Here's the patch I applied in my local code to get Shrine working with Digital Ocean Spaces. I still have a lot on my plate at the moment, so if someone else wants to turn this into a PR that would be great :smile:

# TRICKY: we patch this because it's not providing the correct download url
class Shrine
  module Storage
    class S3 < Storage::Base
      def url(id : String, **options) : String
        endpoint : String?
        if ep = client.@endpoint
          endpoint = ep.gsub("https://", "")
        end
        presigned_options = Awscr::S3::Presigned::Url::Options.new(
          aws_access_key: client.@aws_access_key,
          aws_secret_key: client.@aws_secret_key,
          region: client.@region,
          object: "/#{object_key(id)}",
          bucket: bucket,
          host_name: endpoint
        )

        url = Awscr::S3::Presigned::Url.new(presigned_options)
        url.for(:get)
      end
    end
  end
end

[JoeeGrigg]

Thanks that will help massively!

I will add it into my project and if I get chance turn it into a PR 😀

[JoeeGrigg]

I tried adding this into my project but unfortunately it still didn't work for me. The problem seemed to be partly that the awscr-s3 shard only supports https endpoints. While I can understand this, it makes it a lot harder to run a local Minio server in development. I will keep working on a solution and potential end up submitting a couple of pull requests...

[nikobojs]

What do you think of this diff: https://github.com/nikobojs/shrine.cr/commit/c5a36774ffd5833557314dbd54f192ad62f8e75a It will not solve the forced https but at least use the right hostname. The signature stuff works and the file downloads on the generated link as long as i remove the 's' from https. I can't run specs right now, so not sure if its good enough for a PR?

[wout]

The problem seemed to be partly that the awscr-s3 shard only supports https endpoints.

In that case, you're probably better off using Ngrok or something for your local minio server, so you run it over https.