search

thelastpickle / cassandra-medusa

Apache Cassandra Backup and Restore Tool
Apache License 2.0
258 stars 140 forks source link

Backup fails for Cohesity S3 compatible storage #477

Open smatvienko-tb opened 2 years ago

smatvienko-tb commented 2 years ago

Project board link

Hey!

I had an issue backing up the ThingsBoard's Cassandra data to the Cohesity S3 compatible storage.

After investigating the issue, I found that Cohesity S3 does not return the content-type header, confirmed by their specs later.

I made a patch to work without a content-type header, and Medusa for Apache Cassandra worked fine on production for about one week.

My PR on libcloud that fix the issue is here https://github.com/apache/libcloud/pull/1695

Hopefully, this will help someone to deploy easier!

image

┆Issue is synchronized with this Jira Story by Unito

rzvoncek commented 6 months ago

Hi! We managed to drop libcloud usage meanwhile, so this probably became a non-issue. However, we still need to check if medusa works with Cohesity.

rzvoncek commented 5 months ago

We can't easily get access to Cohesity. @smatvienko-tb , any chance you could give it a try with a recent medusa?

smatvienko-tb commented 5 months ago

We can't easily get access to Cohesity. @smatvienko-tb , any chance you could give it a try with a recent medusa?

OK. Here is some feedback. We have used 0.16.3 for a couple of months. We did a libcloud patch as a legacy patch (has no effect) But we faced another issue with upload to Cohesity S3 storage issue

"Invalid part number 8193 specified. Part number must be a number from 1 to 8192 inclusive"

The Cohesity S3 storage has a limited chunk count (others as well). So we were not able to send files of more than 64 GiB (8192*8MiB)

For a SizeTieredCompactionStrategy or after major compaction without splitting, that might be a problem.

FYI: AWS S3 limits to 10k chunks as well

The current workaround

We replaced the hardcoded value 8 * MB with 50 * MB as a workaround that suits our needs.

It would be nice to have a parameter to set the multipart_chunksize

BTW, 8M transfer time in 2024 is a fraction of a second, even at home. :smile:

Unfortunately, we are still running with a patched library.

Here a @volodymyr-babak's comments on this

# Changes:
# transfer.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# processpool.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# manager.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,
# __init__.py: multipart_chunksize=8 * MB -> multipart_chunksize=50 * MB,

Libraries and files patched :

COPY transfer.py /opt/bitnami/python/lib/python3.9/site-packages/boto3/s3/transfer.py
COPY processpool.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/processpool.py
COPY manager.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/manager.py
COPY __init__.py /opt/bitnami/python/lib/python3.9/site-packages/s3transfer/__init__.py

Hopefully, it will help somehow to improve the Medusa !