Backup/Restore issue

[iocn-2-3-4]

Bug Report

What did you do?

Try to create backup:

---
apiVersion: db.orange.com/v2
kind: CassandraBackup
metadata:
  name: testname
  namespace: cassandra
  labels:
    app: cassandra
spec:
  cassandraCluster: cluster_name
  datacenter: dc1
  secret: s3-access-secret
  snapshotTag: testtag
  storageLocation: s3://my-bucket

What did you expect to see? backup is created

What did you see instead? Under which circumstances? Tested with aws S3, Minio (oracle protocol), even file:/// as storageLocation First backup creates fine. All next backups fails with errors like: "failureCause": [ { "message": "Unable to upload some files successfully: data/dev/service_registry_2-b235ffd0612d11eebd1a892e753c63d8/schema.cql,data/dev/job-b043a5b0612d11eebd1a892e753c63d8/schema.cql,data/dev/person_channel_presence_v2-b68842f0612d11eebd1a892e753c63d8/schema.cql,data/dev/external_server_instance-b81d88f0612d11eebd1a892e753c63d8/schema.cql,data/dev/role-a233a420612d11eebd1a892e753c63d8/1-4225181546/me-1-big-CompressionInfo.db,data/dev/role-a233a420612d11eebd1a892e753c63d8/1-4225181546/me-1-big-Data.db,data/dev/role-a233a420612d11eebd1a892e753c63d8/1-4225181546/me-1-big-Digest.crc32,data/dev/role-a233a420612d11eebd1a892e753c63d8/1-4225181546/me-1-big-Filter.db"

List of files that couldn't be uploaded may be different Sidecar logs (just part of output, because there are a lot of errors like this). Also it happens even if storageLocation id file: k logs -f ice-dc1-rack1-0 -c backrest-sidecar

16:12:04.896 ERROR com.instaclustr.esop.impl.retry.Retrier$DefaultRetrier - This operation will be retried: Error occured while trying to get refresh status on ice/dc1/6184e16c-f53a-4a2f-9dbd-947f86546187/data/dev/socket_session-abb4f5d0612d11eebd1a892e753c63d8/1-1984573430/me-1-big-Statistics.db: s metadata, storage class, website redirect location or encryption attributes.
com.instaclustr.esop.impl.retry.Retrier$RetriableException: Error occured while trying to get refresh status on ice/dc1/6184e16c-f53a-4a2f-9dbd-947f86546187/data/dev/socket_session-abb4f5d0612d11eebd1a892e753c63d8/1-1984573430/me-1-big-Statistics.db: s metadata, storage class, website redirect location or encryption attributes.
    at com.instaclustr.esop.s3.BaseS3Backuper$1.call(BaseS3Backuper.java:93)
    at com.instaclustr.esop.s3.BaseS3Backuper$1.call(BaseS3Backuper.java:61)
    at com.instaclustr.esop.impl.retry.Retrier$DefaultRetrier.submit(Retrier.java:40)
    at com.instaclustr.esop.s3.BaseS3Backuper.freshenRemoteObject(BaseS3Backuper.java:61)
    at com.instaclustr.esop.impl.backup.UploadTracker$UploadUnit.lambda$call$0(UploadTracker.java:117)
    at com.instaclustr.esop.impl.retry.Retrier$DefaultRetrier.submit(Retrier.java:40)
    at com.instaclustr.esop.impl.backup.UploadTracker$UploadUnit.call(UploadTracker.java:117)
    at com.instaclustr.esop.impl.backup.UploadTracker$UploadUnit.call(UploadTracker.java:81)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: s metadata, storage class, website redirect location or encryption attributes. (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest; Request ID: 178A56532CFB0CE4; S3 Extended Request ID: 6786d501-d2eb-43f2-a3e1-3694840618b7; Proxy: null)

Environment

• casskop version: v2.0.3-release

• icarus version: 1.1.3

• go version: go1.21.0 darwin/arm64

• Kubernetes version information: 1.27

• Kubernetes cluster kind: EKS, self-hosted

• Cassandra version: Cassandra 3.11.14

[cscetbon]

@iocn-2-3-4 did you try with the latest version of casskop ? I see that our e2e tests worked https://github.com/cscetbon/casskop/actions/runs/6196237352/job/16822544991

[iocn-2-3-4]

Hello Thanks for advice, we'll try

[cscetbon]

@iocn-2-3-4 any feedback ?