Is there a way to restore the data by populating the scylla PV with big-* files?

[supersaw-it]

Due to the recent operator issue https://github.com/scylladb/scylla-operator/issues/1690 , I decided to completely reinstall my local test cluster.

Before doing that, I moved the data from /var/lib/scylla to my machine, hoping I could restore it once I have a running scylla cluster up & running.

However upon moving the following directories _commitlog data hints viewhints that exist in the aforementioned scylla directory back to the pod with kubectl cp, and queyring the DB, it seems like the database does not recognize the data.

First it complained about the fact that the keyspace did not exist, I created it. Then, after a successful query, I got 0 rows. I created a table with the same name too, and moved the corresponding <...>big-*, e.g. me-7173-big-Data.db, files to the directory of newly created table. Same result with 0 rows.

Nodetool output:

kubectl exec -it scylla-eu-central-1-eu-central-1a-0 -n scylla -- nodetool tablestats joint_depth_trades.joint_summary
Defaulted container "scylla" out of: scylla, scylla-manager-agent, sidecar-injection (init)
Total number of tables: 66
----------------

Keyspace : joint_depth_trades
        Read Count: 52
        Read Latency: 0.0 ms
        Write Count: 0
        Write Latency: NaN ms
        Pending Flushes: 0
                Table: joint_summary
                SSTable count: 0
                Space used (live): 0
                Space used (total): 0
                Space used by snapshots (total): 0
                Off heap memory used (total): 0
                SSTable Compression Ratio: 0.0
                Number of partitions (estimate): 0
                Memtable cell count: 0
                Memtable data size: 0
                Memtable off heap memory used: 0
                Memtable switch count: 0
                Local read count: 52
                Local read latency: 0.005 ms
                Local write count: 0
                Local write latency: NaN ms
                Pending flushes: 0
                Percent repaired: 0.0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 0
                Bloom filter off heap memory used: 0
                Index summary off heap memory used: 0
                Compression metadata off heap memory used: 0
                Compacted partition minimum bytes: 0
                Compacted partition maximum bytes: 0
                Compacted partition mean bytes: 0
                Average live cells per slice (last five minutes): 0.0
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): 0.0
                Maximum tombstones per slice (last five minutes): 0
                Dropped Mutations: 0

Is there a way to make scylla accept the data I have backed up?

Thanks in advance!

[scylla-operator-bot[bot]]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[tnozicka]

I think you'd have to at very least copy the data to empty volume temporarily mounted somewhere else, so it's identical. I can see how copying the files into a running pod can be racy.

The only path we support is backup+restore https://operator.docs.scylladb.com/stable/nodeoperations/restore.html