search

thelastpickle / cassandra-medusa

Apache Cassandra Backup and Restore Tool
Apache License 2.0
264 stars 142 forks source link

libcloud could not upload empty files to azure storage account #533

Closed WentingWu666666 closed 1 year ago

WentingWu666666 commented 2 years ago

Project board link

SSTable-Attached Secondary Index (SASI) files (SI_*.db) could be empty. Medusa uses azure storage driver from libcloud to upload small files and az cli command to upload large files. However, libcloud fails to upload empty files with 403 error. Thus, we could use az cli command to upload empty files.

┆Issue is synchronized with this Jira Task by Unito ┆friendlyId: K8SSAND-1823 ┆priority: Medium

rzvoncek commented 1 year ago

Hi, we recently overhauled the storage backends, and this probably got fixed. I'll have a look if it's still an issue.

rzvoncek commented 1 year ago

So it seems this has been resolved. I've added some tests to cover this, and after running them, I checked our IT buckets and saw the empty files everywhere:

S3: Screenshot 2023-09-21 at 16 49 58

GCP: Screenshot 2023-09-21 at 16 52 58

Azure: Screenshot 2023-09-21 at 16 54 32

We will not see the empty blob when listing blobs, because of #595.

However, we do download the empty files because they get recorded in the manifest.

WentingWu666666 commented 1 year ago

@rzvoncek Thanks for looking into this issue! Actually, it's fixed by #535, but I forget to close it. Please feel free to close it. Thanks!

rzvoncek commented 1 year ago

Yes, I'll have to just close this. I attempted to make an IT for this (#650), but making the TOC.txt files empty is causing other problems. There's no easy way to reproduce the empty files, so we'll have to go on w/o the tests.