search

nuvo / cain

Backup and restore tool for Cassandra on Kubernetes
Apache License 2.0
32 stars 21 forks source link

Restore backup issue #3

Closed alexbarta closed 5 years ago

alexbarta commented 5 years ago

Hi Maor,

I'm doing my tests with default cassandra docker image and latest minio. 

kubectl run cassandra --labels='app=test' --restart=Never  --image-pull-policy=IfNotPresent  --image cassandra --env='CASSANDRA_CLUSTER_NAME=test'

Probably I'm doing something wrong.

Backup goes fine:

~ $ cain backup -n default -l app=test -k keyspace1 --dst s3://cassandra
2018/12/25 15:49:33 Backup started!
2018/12/25 15:49:33 Getting clients
2018/12/25 15:49:33 Getting pods
2018/12/25 15:49:33 Backing up schema
2018/12/25 15:49:36 Taking snapshots
2018/12/25 15:49:36 cassandra Taking snapshot of keyspace keyspace1
2018/12/25 15:49:39 cassandra Requested creating snapshot(s) for [keyspace1] with snapshot name [20181225154936] and options {skipFlush=false}
2018/12/25 15:49:39 cassandra Snapshot directory: 20181225154936
2018/12/25 15:49:39 Calculating paths. This may take a while...
2018/12/25 15:49:39 Starting files copy
2018/12/25 15:49:39 [01/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Data.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Data.db
2018/12/25 15:49:43 [01/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Data.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Data.db
2018/12/25 15:49:43 [02/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-CRC.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-CRC.db
2018/12/25 15:49:43 [02/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-CRC.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-CRC.db
2018/12/25 15:49:43 [03/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Index.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Index.db
2018/12/25 15:49:43 [03/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Index.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Index.db
2018/12/25 15:49:43 [04/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-TOC.txt -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-TOC.txt
2018/12/25 15:49:44 [04/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-TOC.txt -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-TOC.txt
2018/12/25 15:49:44 [05/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Statistics.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Statistics.db
2018/12/25 15:49:44 [05/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Statistics.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Statistics.db
2018/12/25 15:49:44 [06/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Summary.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Summary.db
2018/12/25 15:49:44 [06/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Summary.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Summary.db
2018/12/25 15:49:44 [07/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Filter.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Filter.db
2018/12/25 15:49:44 [07/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Filter.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Filter.db
2018/12/25 15:49:44 [08/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Digest.crc32 -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Digest.crc32
2018/12/25 15:49:44 [08/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-1-big-Digest.crc32 -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-1-big-Digest.crc32
2018/12/25 15:49:44 [09/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Data.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Data.db
2018/12/25 15:49:46 [09/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Data.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Data.db
2018/12/25 15:49:46 [10/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-CRC.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-CRC.db
2018/12/25 15:49:46 [10/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-CRC.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-CRC.db
2018/12/25 15:49:46 [11/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Index.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Index.db
2018/12/25 15:49:46 [11/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Index.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Index.db
2018/12/25 15:49:46 [12/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-TOC.txt -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-TOC.txt
2018/12/25 15:49:46 [12/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-TOC.txt -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-TOC.txt
2018/12/25 15:49:46 [13/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Statistics.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Statistics.db
2018/12/25 15:49:46 [13/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Statistics.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Statistics.db
2018/12/25 15:49:46 [14/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Summary.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Summary.db
2018/12/25 15:49:46 [14/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Summary.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Summary.db
2018/12/25 15:49:46 [15/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Filter.db -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Filter.db
2018/12/25 15:49:47 [15/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Filter.db -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Filter.db
2018/12/25 15:49:47 [16/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Digest.crc32 -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Digest.crc32
2018/12/25 15:49:47 [16/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/mc-2-big-Digest.crc32 -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/mc-2-big-Digest.crc32
2018/12/25 15:49:47 [17/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/manifest.json -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/manifest.json
2018/12/25 15:49:47 [17/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/manifest.json -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/manifest.json
2018/12/25 15:49:47 [18/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/schema.cql -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/schema.cql
2018/12/25 15:49:47 [18/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-82d54860085c11e9a9c0f16c51d72c25/snapshots/20181225154936/schema.cql -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/standard1/schema.cql
2018/12/25 15:49:47 [19/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-8388e0f0085c11e9a9c0f16c51d72c25/snapshots/20181225154936/manifest.json -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/counter1/manifest.json
2018/12/25 15:49:47 [19/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-8388e0f0085c11e9a9c0f16c51d72c25/snapshots/20181225154936/manifest.json -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/counter1/manifest.json
2018/12/25 15:49:47 [20/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-8388e0f0085c11e9a9c0f16c51d72c25/snapshots/20181225154936/schema.cql -> s3://cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/counter1/schema.cql
2018/12/25 15:49:48 [20/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-8388e0f0085c11e9a9c0f16c51d72c25/snapshots/20181225154936/schema.cql -> cassandra/default/test/keyspace1/dc0f5d/20181225154936/cassandra/counter1/schema.cql

In fact I can see the data in the bucket

image

But restore returns an unknown error:

~ $ cain restore --src "s3://cassandra/default/test/keyspace1" -n default -k keyspace1 -l app=test
2018/12/25 15:53:22 Restore started!
2018/12/25 15:53:22 Getting clients
2018/12/25 15:53:22 Getting pods
2018/12/25 15:53:22 Getting current schema
2018/12/25 15:53:23 error in Stream: command terminated with exit code 2
alexbarta commented 5 years ago

Now working! It looks like that cain pretends cassandra to be a folder not a bucket.. I have also tried dropping the keyspace and deleting it from /var/lib/cassandra/data folder... cain didn't work. In such case do you have to create the schema manually before restoring?

~ $ cain backup -n default -l app=test -k keyspace1 --dst s3://db-backup/cassandra
2018/12/25 16:54:46 Backup started!
2018/12/25 16:54:46 Getting clients
2018/12/25 16:54:46 Getting pods
2018/12/25 16:54:46 Backing up schema
2018/12/25 16:54:49 Taking snapshots
2018/12/25 16:54:49 cassandra Taking snapshot of keyspace keyspace1
2018/12/25 16:54:51 cassandra Requested creating snapshot(s) for [keyspace1] with snapshot name [20181225165449] and options {skipFlush=false}
2018/12/25 16:54:51 cassandra Snapshot directory: 20181225165449
2018/12/25 16:54:51 Calculating paths. This may take a while...
2018/12/25 16:54:52 Starting files copy
2018/12/25 16:54:52 [01/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Data.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Data.db
2018/12/25 16:54:55 [01/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Data.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Data.db
2018/12/25 16:54:55 [02/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Summary.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Summary.db
2018/12/25 16:54:55 [02/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Summary.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Summary.db
2018/12/25 16:54:55 [03/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Statistics.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Statistics.db
2018/12/25 16:54:55 [03/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Statistics.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Statistics.db
2018/12/25 16:54:55 [04/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-CRC.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-CRC.db
2018/12/25 16:54:55 [04/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-CRC.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-CRC.db
2018/12/25 16:54:55 [05/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Digest.crc32 -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Digest.crc32
2018/12/25 16:54:56 [05/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Digest.crc32 -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Digest.crc32
2018/12/25 16:54:56 [06/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Index.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Index.db
2018/12/25 16:54:56 [06/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Index.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Index.db
2018/12/25 16:54:56 [07/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-TOC.txt -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-TOC.txt
2018/12/25 16:54:56 [07/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-TOC.txt -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-TOC.txt
2018/12/25 16:54:56 [08/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Filter.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Filter.db
2018/12/25 16:54:56 [08/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-1-big-Filter.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Filter.db
2018/12/25 16:54:56 [09/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Data.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Data.db
2018/12/25 16:54:57 [09/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Data.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Data.db
2018/12/25 16:54:57 [10/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Summary.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Summary.db
2018/12/25 16:54:57 [10/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Summary.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Summary.db
2018/12/25 16:54:57 [11/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Statistics.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Statistics.db
2018/12/25 16:54:58 [11/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Statistics.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Statistics.db
2018/12/25 16:54:58 [12/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-CRC.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-CRC.db
2018/12/25 16:54:58 [12/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-CRC.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-CRC.db
2018/12/25 16:54:58 [13/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Digest.crc32 -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Digest.crc32
2018/12/25 16:54:58 [13/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Digest.crc32 -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Digest.crc32
2018/12/25 16:54:58 [14/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Index.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Index.db
2018/12/25 16:54:58 [14/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Index.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Index.db
2018/12/25 16:54:58 [15/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-TOC.txt -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-TOC.txt
2018/12/25 16:54:59 [15/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-TOC.txt -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-TOC.txt
2018/12/25 16:54:59 [16/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Filter.db -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Filter.db
2018/12/25 16:54:59 [16/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/mc-2-big-Filter.db -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Filter.db
2018/12/25 16:54:59 [17/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/manifest.json -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/manifest.json
2018/12/25 16:54:59 [17/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/manifest.json -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/manifest.json
2018/12/25 16:54:59 [18/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/schema.cql -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/schema.cql
2018/12/25 16:54:59 [18/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/snapshots/20181225165449/schema.cql -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/schema.cql
2018/12/25 16:54:59 [19/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/snapshots/20181225165449/manifest.json -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/manifest.json
2018/12/25 16:54:59 [19/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/snapshots/20181225165449/manifest.json -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/manifest.json
2018/12/25 16:54:59 [20/20] copy: k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/snapshots/20181225165449/schema.cql -> s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/schema.cql
2018/12/25 16:54:59 [20/20] done: default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/snapshots/20181225165449/schema.cql -> db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/schema.cql
2018/12/25 16:54:59 Clearing snapshots
2018/12/25 16:54:59 cassandra Clearing snapshot of keyspace keyspace1
2018/12/25 16:55:01 cassandra Requested clearing snapshot(s) for [keyspace1] with snapshot name [20181225165449]
2018/12/25 16:55:01 All done!
~ $ cain restore --src 's3://db-backup/cassandra/default/test' -n default -k keyspace1 -l app=test -t
20181225165449
2018/12/25 16:56:20 Restore started!
2018/12/25 16:56:20 Getting clients
2018/12/25 16:56:20 Getting pods
2018/12/25 16:56:20 Getting current schema
2018/12/25 16:56:21 Found schema: dc0f5d
2018/12/25 16:56:21 Calculating paths. This may take a while...
2018/12/25 16:56:21 Validating pods match restore
2018/12/25 16:56:21 Getting materialized views to exclude
2018/12/25 16:56:22 Truncating tables
2018/12/25 16:56:22 cassandra Truncating table counter1 in keyspace keyspace1
2018/12/25 16:56:23 cassandra Truncating table standard1 in keyspace keyspace1
2018/12/25 16:56:24 Starting files copy
2018/12/25 16:56:24 [01/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/manifest.json -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/manifest.json
2018/12/25 16:56:24 [01/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/manifest.json -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/manifest.json
2018/12/25 16:56:24 [02/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/schema.cql -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/schema.cql
2018/12/25 16:56:25 [02/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/counter1/schema.cql -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/counter1-a0689b80086511e986b57f956749f954/schema.cql
2018/12/25 16:56:25 [03/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/manifest.json -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/manifest.json
2018/12/25 16:56:25 [03/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/manifest.json -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/manifest.json
2018/12/25 16:56:25 [04/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-CRC.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-CRC.db
2018/12/25 16:56:26 [04/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-CRC.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-CRC.db
2018/12/25 16:56:26 [05/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Data.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Data.db
2018/12/25 16:56:28 [05/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Data.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Data.db
2018/12/25 16:56:28 [06/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Digest.crc32 -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Digest.crc32
2018/12/25 16:56:28 [06/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Digest.crc32 -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Digest.crc32
2018/12/25 16:56:28 [07/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Filter.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Filter.db
2018/12/25 16:56:29 [07/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Filter.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Filter.db
2018/12/25 16:56:29 [08/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Index.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Index.db
2018/12/25 16:56:29 [08/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Index.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Index.db
2018/12/25 16:56:29 [09/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Statistics.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Statistics.db
2018/12/25 16:56:30 [09/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Statistics.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Statistics.db
2018/12/25 16:56:30 [10/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Summary.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Summary.db
2018/12/25 16:56:30 [10/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-Summary.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-Summary.db
2018/12/25 16:56:30 [11/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-TOC.txt -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-TOC.txt
2018/12/25 16:56:31 [11/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-1-big-TOC.txt -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-1-big-TOC.txt
2018/12/25 16:56:31 [12/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-CRC.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-CRC.db
2018/12/25 16:56:31 [12/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-CRC.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-CRC.db
2018/12/25 16:56:31 [13/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Data.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Data.db
2018/12/25 16:56:32 [13/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Data.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Data.db
2018/12/25 16:56:32 [14/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Digest.crc32 -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Digest.crc32
2018/12/25 16:56:33 [14/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Digest.crc32 -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Digest.crc32
2018/12/25 16:56:33 [15/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Filter.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Filter.db
2018/12/25 16:56:33 [15/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Filter.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Filter.db
2018/12/25 16:56:33 [16/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Index.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Index.db
2018/12/25 16:56:34 [16/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Index.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Index.db
2018/12/25 16:56:34 [17/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Statistics.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Statistics.db
2018/12/25 16:56:34 [17/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Statistics.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Statistics.db
2018/12/25 16:56:34 [18/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Summary.db -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Summary.db
2018/12/25 16:56:35 [18/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-Summary.db -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-Summary.db
2018/12/25 16:56:35 [19/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-TOC.txt -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-TOC.txt
2018/12/25 16:56:35 [19/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/mc-2-big-TOC.txt -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/mc-2-big-TOC.txt
2018/12/25 16:56:35 [20/20] copy: s3://db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/schema.cql -> k8s://default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/schema.cql
2018/12/25 16:56:36 [20/20] done: db-backup/cassandra/default/test/keyspace1/dc0f5d/20181225165449/cassandra/standard1/schema.cql -> default/cassandra/cassandra/var/lib/cassandra/data/keyspace1/standard1-9fbc7d00086511e986b57f956749f954/schema.cql
2018/12/25 16:56:36 Refreshing tables
2018/12/25 16:56:36 cassandra Refreshing table counter1 in keyspace keyspace1
2018/12/25 16:56:37 cassandra Refreshing table standard1 in keyspace keyspace1
2018/12/25 16:56:39 All done!
maorfr commented 5 years ago

Hey,

So the first attempt didn't work for you because cain couldn't get the schema. I am able to reproduce this locally by dropping the keyspace and running restore. I have added a validation for this part so that a clear error message will be displayed.

Assuming that there was a schema, the restore will still have failed, for two reasons:

  1. You specified the --src including the keyspace - added a warning for this case
  2. You didn't specify a tag - added validation for this case

Regarding backup and restore to an s3 path such as s3://cassandra - i have checked that and it works as expected.

If you have any other issues - don't hesitate to raise them! Very glad you are involved ;)

maorfr commented 5 years ago

And btw, I have just added version 0.4.1 which allows to restore the schema as well! ;)

alexbarta commented 5 years ago

Great, thanks for adding the feature!! BTW I have a couple of questions, if you don't mind:

maorfr commented 5 years ago

Hey,

If you have more questions, feel free to ask :)

alexbarta commented 5 years ago

Clear! Thanks a lot