search

scylladb / scylla-manager

The Scylla Manager
https://manager.docs.scylladb.com/stable/
Other
53 stars 34 forks source link

Give possibility for restoring DC using mapping sourceDC -> destinationDC #3829

Open karol-kokoszka opened 7 months ago

karol-kokoszka commented 7 months ago

https://github.com/scylladb/scylla-manager/issues/3871

Right now, there is no option in the Scylla Manager restore task to restore just a single data center (DC) from the backup location. This could lead to problematic situations, particularly when:

Encryption at Rest (EaR) is enabled, Two DCs use different encryption keys, Encryption keys are stored in different cloud regions, and There is only one backup location available. To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

The location flag may not be very intuitive, as the [dc] part defines the destination DC, not the source DC data. We need to discuss during the manager planning to determine if a new flag specifying the source is necessary. If we can restore just a single DC, then we can restore DC by DC, avoiding the need to create multi-regional keys.

(cc: @tzach)

tzach commented 6 months ago

To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

Agree, but how this is a Scylla Manager issue to fix?

karol-kokoszka commented 6 months ago

We could potentially address the problem by allowing to restore just a single DC from the location bucket. It's something what we doesn't support at the moment (possibly by a mistake).

rayakurl commented 6 months ago

@tzach - we need a resolution. for now almost all sct tests are failing since they are multi DC. We will add a couple of pipelines for a single DC + encryption but are are disabling the multi DC jobs as they constantly failing. @mikliapko as discussed, please create a task for the new pipelines. and disable the multi DC ones for now. Thanks

karol-kokoszka commented 6 months ago

grooming notes

The initial idea is to add new flag to the restore CLI, so that it's possible to define the origin DC from the backup location. Then, data from this DC is going to be restored to specified destination.

@mikliapko SCT will have to be updated to test the scenario with restoring single DC.

Michal-Leszczynski commented 1 month ago

The initial idea is to add new flag to the restore CLI, so that it's possible to define the origin DC from the backup location. Then, data from this DC is going to be restored to specified destination.

After giving it some more thought, I wouldn't recommend adding it in such way. The need for this feature raised from https://github.com/scylladb/scylla-manager/issues/3871, where it could be used to restore DC by DC. This is problematic, as restore task does not only download and load&stream the data, but it also:

So running many restore tasks, one by one, DC by DC, would result in lots of redundant work. Also, I could theoretically (not sure about that) lead to data resurrection, as tombstone_gc would be enabled in between DC restorations. Not to mention, that it would be user responsibility to remember about all DCs from the backup to be restored.

A better idea could be to extend restore with a flag like --dc-mapping (string -> list of strings). This would allow user to specify which dc from the backup should be restored by which dcs in restored cluster. It has a few benefits;

mikliapko commented 1 month ago

A better idea could be to extend restore with a flag like --dc-mapping (string -> list of strings).

@Michal-Leszczynski When it is ready, could you please then provide an example of input for this flag. I will switch some of our SCT tests back to run on multiDC cluster.