search

k8ssandra / k8ssandra-operator

The Kubernetes operator for K8ssandra
https://k8ssandra.io/
Apache License 2.0
174 stars 79 forks source link

Restore remote datacenter into new datacenter with Medusa #1040

Open networkingana opened 1 year ago

networkingana commented 1 year ago

What did you do?

I created a backup using MedusaBackupJob of my datacenter with production data, after the backup finished it was stored on MinIO, then I used MedusaTask to sync the backup to the new separated cluster, and then run MedusaRestoreJob, the restore seems to finish successfully looking from the medusa-restore logs, but after that, nodes one start, the issue is the following:

INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,885 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-5-big (4.479KiB)
INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,887 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-6-big (1.994KiB)
INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,888 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-7-big (6.360KiB)
INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,889 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/nb-8-big (6.363KiB)
INFO  [main] 2023-08-21 14:17:49,917 ColumnFamilyStore.java:385 - Initializing system.repairs
INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,919 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/repairs-a3d277d1cfaf36f5a2a738d5eea9ad6a/nb-17-big (12.892KiB)
INFO  [SSTableBatchOpen:1] 2023-08-21 14:17:49,920 SSTableReaderBuilder.java:351 - Opening /var/lib/cassandra/data/system/repairs-a3d277d1cfaf36f5a2a738d5eea9ad6a/nb-18-big (1.875KiB)
INFO  [main] 2023-08-21 14:17:49,958 QueryProcessor.java:106 - Initialized prepared statement caches with 10 MB
ERROR [main] 2023-08-21 14:17:50,115 CassandraDaemon.java:909 - Cannot start node if snitch's rack (rack-2) differs from previous rack (rack-5). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_rack=true.

I tried to add the flag to ignore rack but then thenodetool status command showed two racks with same name, which was wrong

Environment

Source DC: k8ssandra-operator version 1.2.1 Destination DC: k8ssandra-operator version 1.7.1

  Image:         docker.io/k8ssandra/k8ssandra-operator:v1.7.1
   Image ID:      docker-pullable://k8ssandra/k8ssandra-operator@sha256:8921bbb07dabecd5149fe4ea748b0330702bfe3cf4fb910ed52ae49f55845f19
    Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:32:41Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: k8ssandra
  namespace: k8ssandra-operator
spec:
  auth: true
  cassandra:
    datacenters:
    - config:
        cassandraYaml:
          cas_contention_timeout_in_ms: 2000
          commitlog_sync: batch
          commitlog_sync_batch_window_in_ms: 50
          concurrent_writes: 32
          counter_write_request_timeout_in_ms: 7000
          endpoint_snitch: GossipingPropertyFileSnitch
          read_request_timeout_in_ms: 10000
          write_request_timeout_in_ms: 2500
        jvmOptions:
          additionalOptions:
          gc: G1GC
          heapSize: 2G
      metadata:
        name: dc2
      perNodeConfigInitContainerImage: mikefarah/yq:4
      racks:
      - name: rack-1
        nodeAffinityLabels:
          kubernetes.io/hostname: wknd01srv
      - name: rack-2
        nodeAffinityLabels:
          kubernetes.io/hostname: wknd02srv
      - name: rack-3
        nodeAffinityLabels:
          kubernetes.io/hostname: wknd04srv
      - name: rack-4
        nodeAffinityLabels:
          kubernetes.io/hostname: wknd03srv
      - name: rack-5
        nodeAffinityLabels:
          kubernetes.io/hostname: wknd05srv
      size: 5
      stargate:
        allowStargateOnDataNodes: true
        containerImage:
          registry: docker.io
          repository: stargateio
          tag: v1.0.67
        heapSize: 2G
        secretsProvider: internal
        serviceAccount: default
        size: 1
      stopped: false
    perNodeConfigInitContainerImage: mikefarah/yq:4
    serverType: cassandra
    serverVersion: 4.0.1
    storageConfig:
      cassandraDataVolumeClaimSpec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 450Gi
        storageClassName: local-path
    superuserSecretRef:
      name: k8ssandra-superuser
  medusa:
    storageProperties:
      bucketName: k8ssandra-medusa
      concurrentTransfers: 1
      host: minio.minio.svc.cluster.local
      maxBackupAge: 0
      maxBackupCount: 0
      multiPartUploadThreshold: 104857600
      port: 9000
      prefix: prod
      secure: false
      storageProvider: s3_compatible
      storageSecretRef:
        name: medusa-bucket-key
      transferMaxBandwidth: 50MB/s
  secretsProvider: internal

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: K8OP-78

adejanovski commented 1 year ago

Hi @networkingana,

rack aware restores were introduced in k8ssandra-operator v1.8.0. I'm not entirely sure v1.2.0 provided placement information in the backups, which could make them unsuitable for restoring with v1.8.0 🤔 You'd have to check the topology.json files from the backups to verify this. If you have the dc/rack information in that file, it means you should be able to safely restore backups to v1.8.0. You'll need to have the exact same dc and rack names in the target cluster.

EmanuelSanchez commented 2 months ago

@adejanovski, I have another issue but not sure if it's related. When trying to restore the backup into a new cluster, but with a different datacenter, I get this error: image Both clusters run on the same namespace, so I'm using different names of datacenters.

I'm using this documentation to create the restore: https://docs.k8ssandra.io/tasks/backup-restore/#restoring-a-backup

adejanovski commented 2 months ago

Hi @EmanuelSanchez, this is not supported. The datacenter names are part of the replication settings for each keyspace, and Medusa restores the schema by downloading the system_schema sstables. Hence we cannot modify the dc names. Note that you can override the datacenter name in order to create another "dc1" datacenter but with a different CassandraDatacenter name:

   ....
    datacenters:
      - config:
          jvmOptions:
            gc: G1GC
        datacenterName: dc1
        metadata:
          name: override-dc1
        ...

The CQL name of the dc will be dc1 but the CassandraDatacenter kubernetes object will be named override-dc1.

EmanuelSanchez commented 2 months ago

That's great. It worked. Thank you so much