search

thelastpickle / cassandra-medusa

Apache Cassandra Backup and Restore Tool
Apache License 2.0
258 stars 140 forks source link

Medusa 0.13 fails on s3_compatible endpoints #516

Open Skunnyk opened 2 years ago

Skunnyk commented 2 years ago

Project board link

We are testing medusa 0.13.x, and we can't backup anymore to a s3_compatible endpoint. It works with 0.11.x. We are using OVHCloud s3 storage (https://www.ovhcloud.com/en/public-cloud/object-storage/).

host = s3.gra.perf.cloud.ovh.net
bucket_name = my-bucket
region = gra

It looks like the region is not set at all when invoking awscli for multi_part uploads.

[2022-08-11 14:04:30,888] DEBUG: Uploading datanode01.int/data/cassel/collections_by_doc_index-1db740f19f0311ecb74a000000000000/md-235-big-Data.db as multi part
[2022-08-11 14:04:30,888] DEBUG: aws --endpoint-url https://s3.gra.perf.cloud.ovh.net s3 cp /data/cassel/collections_by_doc_index-1db740f19f0311ecb74a000000000000/snapshots/medusa-theory-20220811-1404/md-235-big-Data.db s3://backup-theory/datanode01.int/data/cassel/collections_by_doc_index-1db740f19f0311ecb74a000000000000/md-235-big-Data.db

And in awscli /tmp/xxx.output:

Could not connect to the endpoint URL: "https://s3.gra.amazonaws.com/backup-xxxx

Since https://github.com/thelastpickle/cassandra-medusa/commit/3c01bef73bf266f84c6c7357b07c72a8a5386b86 , medusa uses more botocore, and set the region with self._env['AWS_REGION'], no more with --region cli switch.

By changing it to self._env['AWS_DEFAULT_REGION'] in medusa/storage/s3_compat_storage/awscli.py, it works fine. Looks like botocore (and python?) only uses AWS_DEFAULT_REGION (read from various places on the Internet and https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

Can this be fixed ? I don't know if changing this variable will break other s3_compatible storages.

┆Issue is synchronized with this Jira Story by Unito

uwburn commented 1 year ago

Experiencing exactly the same behavior.

At the moment i'm working around it by raising the multi_part_upload_threshold to avoid hitting multipart, but it's doable until you don't have files too large.

rzvoncek commented 6 months ago

Hi @uwburn! We've shipped a whole bunch of changes regarding the storage backends. Any chance you managed to try the latest (0.19.1) Medusa to see if it has helped you?

uwburn commented 6 months ago

Hi @rzvoncek, thanks for pointing this out.

Unfortunately atm i'm running on a very outdated version of k8ssandra-operator for Kubernetes and basically i need to upgrade that for using newer medusa versions.

If i find the time to work on an upgrade strategy i will try that.