search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
36 stars 8 forks source link

how to use with non-AWS s3 providers? #55

Open rabernat opened 3 years ago

rabernat commented 3 years ago

Many object storage services implement the S3 protocol without actually being AWS S3. For example, Wasabi and Open Storage Network (OSN). For these services, you define a custom endpoint URL for S3.

I would like to try cloud-files with OSN. For the AWS CLI, I would use this configuration

[plugins]
endpoint = awscli_plugin_endpoint

[profile osn]
aws_access_key_id=<key>
aws_secret_access_key=<secret>
s3 =
    endpoint_url = https://ncsa.osn.xsede.org
s3api =
    endpoint_url = https://ncsa.osn.xsede.org

However, I'm having trouble figuring out how to make this work. I read the wiki and looked at the code, but I couldn't figure it out. It is supported?

Thanks a lot for your help and for providing this useful package.

william-silversmith commented 3 years ago

Hey Ryan! Glad you're finding this useful.

We use a kind of non-standard way of doing this, but I think it makes a lot of sense.

cf = CloudFiles("s3://https://ncsa.osn.xsede.org/BUCKET", secrets=SECRETS)

This functionality isn't super battle tested, so please report any issues or any apparent design flaws.

rabernat commented 3 years ago

Awesome, will give it a try.

william-silversmith commented 3 years ago

The thing that immediately occurs to me is that there isn't a great way to set a per-host secrets file. I think it will probably pick up your default aws-secret.json though.