drone-plugins / drone-s3-cache

Caches build artifacts to S3 compatible storage backends
http://plugins.drone.io/drone-plugins/drone-s3-cache
Apache License 2.0
29 stars 30 forks source link

Support S3 transfer acceleration #37

Closed mithrandi closed 6 years ago

mithrandi commented 6 years ago

S3 transfer acceleration requires this: https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html#transfer-acceleration-getting-started

When trying to use this:

time="2018-07-07T13:57:54Z" level=info msg="Restoring cache at /fusionapp/fusion/master/archive.tar"
time="2018-07-07T13:57:54Z" level=info msg="Retrieving file in fusionapp at fusion/master/archive.tar"
time="2018-07-07T13:57:57Z" level=warning msg="Cache could not be restored The specified bucket does not exist."

I believe this is because it expects the first path component to be the bucket name.

tboerger commented 6 years ago

This plugin uses the Minio client, so I'm not sure if this can be enabled at all

mithrandi commented 6 years ago

There's a member in Options that seems like it might work, but maybe it does nothing?

https://github.com/drone-plugins/drone-s3-cache/blob/f480bb823beb2b73ee8310d36f4db85756f2fdac/storage/s3/s3.go#L36

mithrandi commented 6 years ago

Looks like minio.BucketLookupDNS is what's needed on the Minio side.

tboerger commented 6 years ago

Pathstyle is an option required for Minio access because Minio is working with paths per bucket and not subdomains per bucket.

tboerger commented 6 years ago

For reference: https://docs.minio.io/docs/golang-client-api-reference#SetS3TransferAccelerate

mithrandi commented 6 years ago

That call is about turning on acceleration, so I think not directly related here.

tboerger commented 6 years ago

That call is about setting the accelerated endpoint, and this seems to be exactly the option required to integrate support for it.

mithrandi commented 6 years ago

Whoops, you're right, I completely misread the docs!

mithrandi commented 6 years ago

I changed the title to be more specific in light of the above.