search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
336 stars 171 forks source link

RFE: re-bootstrap cluster from old PVCs #535

Closed srfaytkn closed 2 years ago

srfaytkn commented 3 years ago

Describe the bug I use ebs volumes on aws. When i uninstall the scylla cluster with helm and reinstall it with same ebs volumes, it tries to connect to the old node ip addresses.

To Reproduce Steps to reproduce the behavior:

  1. Create Ebs storage
  2. Install scylla cluster with helm
  3. Insert data
  4. Uninstall scylla cluster with helm
  5. Reinstall scylla cluster with helm

Expected behavior Getting a new cluster up without losing data

Old Service Ip List 10.100.70.138, 10.100.197.104, 10.100.118.129

New Service Ip List 10.100.113.44, 10.100.39.182, 10.100.31.7

Logs

INFO  2021-04-02 17:18:38,592 [shard 0] init - starting gossip
INFO  2021-04-02 17:18:38,592 [shard 0] init - seeds={10.100.31.7, 10.100.39.182, 10.100.113.44}, listen_address=192.168.3.230, broadcast_address=10.100.113.44
INFO  2021-04-02 17:18:38,592 [shard 0] init - starting storage proxy
INFO  2021-04-02 17:18:38,596 [shard 0] init - creating hints manager
INFO  2021-04-02 17:18:38,596 [shard 1] init - creating hints manager
INFO  2021-04-02 17:18:38,597 [shard 2] init - creating hints manager
INFO  2021-04-02 17:18:38,597 [shard 0] init - starting migration manager
INFO  2021-04-02 17:18:38,597 [shard 0] init - starting query processor
INFO  2021-04-02 17:18:38,598 [shard 0] init - initializing batchlog manager
INFO  2021-04-02 17:18:38,600 [shard 0] format_selector - Selected md sstables format
INFO  2021-04-02 17:18:38,600 [shard 0] legacy_schema_migrator - Moving 0 keyspaces from legacy schema tables to the new schema keyspace (system_schema)
INFO  2021-04-02 17:18:38,600 [shard 0] legacy_schema_migrator - Dropping legacy schema tables
{"L":"DEBUG","T":"2021-04-02T17:18:40.234Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/storage_service/host_id","duration":"0ms","status":404,"bytes":37}
{"L":"ERROR","T":"2021-04-02T17:18:40.234Z","N":"member_controller","M":"error while executing nodetool status in readiness check","error":"giving up after 9 attempts: agent [HTTP 404] Not found","_trace_id":"iB-jz-4VQxCwUlv5DkjDSA","errorStack":"github.com/scylladb/scylla-operator/pkg/scyllaclient.(*retryableOperation).submit\n\tgithub.com/scylladb/scylla-operator/pkg/scyllaclient/retry.go:59\ngithub.com/scylladb/scylla-operator/pkg/scyllaclient.retryableTransport.Submit\n\tgithub.com/scylladb/scylla-operator/pkg/scyllaclient/retry.go:53\ngithub.com/scylladb/scylla-operator/pkg/scyllaclient/internal/scylla/client/operations.(*Client).StorageServiceHostIDGet\n\tgithub.com/scylladb/scylla-operator/pkg/scyllaclient/internal/scylla/client/operations/operations_client.go:10956\ngithub.com/scylladb/scylla-operator/pkg/scyllaclient.(*Client).Status\n\tgithub.com/scylladb/scylla-operator/pkg/scyllaclient/client.go:103\ngithub.com/scylladb/scylla-operator/pkg/controllers/sidecar.readinessCheck.func1\n\tgithub.com/scylladb/scylla-operator/pkg/controllers/sidecar/checks.go:94\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2042\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2417\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2843\nnet/http.(*conn).serve\n\tnet/http/server.go:1925\nruntime.goexit\n\truntime/asm_amd64.s:1374\n"}
{"L":"ERROR","T":"2021-04-02T17:18:40.234Z","N":"member_controller","M":"Readiness check failed, node not ready","_trace_id":"iB-jz-4VQxCwUlv5DkjDSA"}
INFO  2021-04-02 17:18:40,264 [shard 0] legacy_schema_migrator - Completed migration of legacy schema tables
INFO  2021-04-02 17:18:40,264 [shard 0] init - loading system sstables
INFO  2021-04-02 17:18:40,272 [shard 0] init - loading non-system sstables
INFO  2021-04-02 17:18:40,367 [shard 0] database - Populating Keyspace system_auth
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_auth: Reading CF roles id=5bc52802-de25-35ed-aeab-188eecebb090 version=9961c0fa-a95c-3d62-bd86-9b4f4cea30b8
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_auth: Reading CF role_members id=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d version=86589ed5-2eea-31e8-bc40-2c83e0bdff48
INFO  2021-04-02 17:18:40,367 [shard 0] database - Populating Keyspace system_traces
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_traces: Reading CF sessions_time_idx id=0ebf001c-c1d1-3693-9a63-c3d96ac53318 version=b130d80b-9aa6-39c8-bb62-689bba1418fa
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_traces: Reading CF sessions id=c5e99f16-8677-3914-b17e-960613512345 version=9e32b4bf-d649-3c9a-bcee-841d649a391e
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_traces: Reading CF node_slow_log_time_idx id=f9706768-aa1e-3d87-9e5c-51a3927c2870 version=b1116b30-52e1-325d-a67e-fd2e03f0fd7a
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_traces: Reading CF node_slow_log id=bfcc4e62-5b63-3aa1-a1c3-6f5e47f3325c version=7a5166f5-cde3-3ff8-a373-dc660f406fad
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_traces: Reading CF events id=8826e8e9-e16a-3728-8753-3bc1fc713c25 version=d484c909-5572-3dcc-8a6f-30d6a0fb6f25
INFO  2021-04-02 17:18:40,367 [shard 0] database - Populating Keyspace system_distributed
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_distributed: Reading CF view_build_status id=5582b59f-8e4e-35e1-b913-3acada51eb04 version=e61ea49b-8dbc-3ff6-a69f-896f4d38b7bb
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_distributed: Reading CF cdc_streams_descriptions id=249f0ac2-8856-36d4-9802-f55223eb01c2 version=f727f60d-bfaa-32de-a0d6-898e1cbaf2c9
INFO  2021-04-02 17:18:40,367 [shard 0] database - Keyspace system_distributed: Reading CF cdc_generation_descriptions id=ae653406-7b3e-34aa-af70-d5b90f837c67 version=95e9e159-839f-39b3-990f-8e8a04817ff8
INFO  2021-04-02 17:18:40,437 [shard 0] init - starting view update generator
INFO  2021-04-02 17:18:40,437 [shard 0] init - discovering staging sstables
INFO  2021-04-02 17:18:40,437 [shard 0] init - setting up system keyspace
INFO  2021-04-02 17:18:40,453 [shard 0] init - starting commit log
INFO  2021-04-02 17:18:40,453 [shard 0] compaction - [Compact system.truncated 77f1f750-93d7-11eb-abf1-000000000000] Compacting [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-39-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-42-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-36-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 0] compaction - [Compact system.compaction_history 77f1f751-93d7-11eb-abf1-000000000000] Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-18-big-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-15-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 2] compaction - [Compact system.truncated 77f1f750-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-41-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-35-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 1] compaction - [Compact system.truncated 77f1f750-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-43-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-40-big-Data.db:level=0, /var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-37-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 2] compaction - [Compact system_schema.computed_columns 77f1f751-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-50-big-Data.db:level=0, /var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 1] compaction - [Compact system_schema.computed_columns 77f1f751-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-46-big-Data.db:level=0, /var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-49-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 1] compaction - [Compact system.local 77f1f752-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-43-big-Data.db:level=0, /var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-40-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 1] compaction - [Compact system_schema.columns 77f1f753-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-49-big-Data.db:level=0, /var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-46-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 1] compaction - [Compact system_schema.scylla_tables 77f1f754-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-49-big-Data.db:level=0, /var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-46-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 2] compaction - [Compact system_schema.columns 77f1f752-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-50-big-Data.db:level=0, /var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,453 [shard 2] compaction - [Compact system_schema.scylla_tables 77f1f753-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-50-big-Data.db:level=0, /var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,454 [shard 2] compaction - [Compact system.compaction_history 77f21e60-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-20-big-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,454 [shard 0] init - initializing migration manager RPC verbs
INFO  2021-04-02 17:18:40,454 [shard 0] init - initializing storage proxy RPC verbs
INFO  2021-04-02 17:18:40,455 [shard 0] init - starting streaming service
INFO  2021-04-02 17:18:40,456 [shard 0] init - starting hinted handoff manager
INFO  2021-04-02 17:18:40,457 [shard 0] init - starting messaging service
INFO  2021-04-02 17:18:40,458 [shard 0] repair - Setting max_repair_memory=435578470, max_repair_memory_per_range=33554432, max_repair_ranges_in_parallel=12
INFO  2021-04-02 17:18:40,458 [shard 0] init - starting storage service
INFO  2021-04-02 17:18:40,458 [shard 0] storage_service - Loading persisted ring state
INFO  2021-04-02 17:18:40,462 [shard 0] storage_service - initial_contact_nodes={10.100.118.129, 10.100.197.104}, loaded_endpoints={10.100.118.129, 10.100.197.104}, loaded_peer_features=2
INFO  2021-04-02 17:18:40,462 [shard 0] storage_service - peer=10.100.197.104, supported_features=CDC,COMPUTED_COLUMNS,CORRECT_COUNTER_ORDER,CORRECT_NON_COMPOUND_RANGE_TOMBSTONES,CORRECT_STATIC_COMPACT_IN_MC,COUNTERS,DIGEST_FOR_NULL_VALUES,DIGEST_INSENSITIVE_TO_EXPIRY,DIGEST_MULTIPARTITION_READ,HINTED_HANDOFF_SEPARATE_CONNECTION,INDEXES,LARGE_PARTITIONS,LA_SSTABLE_FORMAT,LWT,MATERIALIZED_VIEWS,MC_SSTABLE_FORMAT,MD_SSTABLE_FORMAT,NONFROZEN_UDTS,PER_TABLE_CACHING,PER_TABLE_PARTITIONERS,RANGE_TOMBSTONES,ROLES,ROW_LEVEL_REPAIR,SCHEMA_TABLES_V3,STREAM_WITH_RPC_STREAM,TRUNCATION_TABLE,UNBOUNDED_RANGE_TOMBSTONES,VIEW_VIRTUAL_COLUMNS,WRITE_FAILURE_REPLY,XXHASH
INFO  2021-04-02 17:18:40,462 [shard 0] storage_service - peer=10.100.118.129, supported_features=CDC,COMPUTED_COLUMNS,CORRECT_COUNTER_ORDER,CORRECT_NON_COMPOUND_RANGE_TOMBSTONES,CORRECT_STATIC_COMPACT_IN_MC,COUNTERS,DIGEST_FOR_NULL_VALUES,DIGEST_INSENSITIVE_TO_EXPIRY,DIGEST_MULTIPARTITION_READ,HINTED_HANDOFF_SEPARATE_CONNECTION,INDEXES,LARGE_PARTITIONS,LA_SSTABLE_FORMAT,LWT,MATERIALIZED_VIEWS,MC_SSTABLE_FORMAT,MD_SSTABLE_FORMAT,NONFROZEN_UDTS,PER_TABLE_CACHING,PER_TABLE_PARTITIONERS,RANGE_TOMBSTONES,ROLES,ROW_LEVEL_REPAIR,SCHEMA_TABLES_V3,STREAM_WITH_RPC_STREAM,TRUNCATION_TABLE,UNBOUNDED_RANGE_TOMBSTONES,VIEW_VIRTUAL_COLUMNS,WRITE_FAILURE_REPLY,XXHASH
INFO  2021-04-02 17:18:40,462 [shard 0] storage_service - Checking remote features with gossip, initial_contact_nodes={10.100.118.129, 10.100.197.104}
INFO  2021-04-02 17:18:40,462 [shard 0] messaging_service - Starting Messaging Service on port 7000
INFO  2021-04-02 17:18:40,463 [shard 0] gossip - Gossip shadow round started with nodes={10.100.118.129, 10.100.197.104}
INFO  2021-04-02 17:18:40,780 [shard 0] compaction - [Compact system.truncated 77f1f750-93d7-11eb-abf1-000000000000] Compacted 3 sstables to [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-45-big-Data.db:level=0, ]. 87kB to 76kB (~88% of original) in 221ms = 347kB/s. ~384 total partitions merged to 2.
INFO  2021-04-02 17:18:40,781 [shard 0] compaction - [Compact system.compaction_history 77f1f751-93d7-11eb-abf1-000000000000] Compacted 2 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-21-big-Data.db:level=0, ]. 165kB to 96kB (~58% of original) in 222ms = 433kB/s. ~256 total partitions merged to 46.
INFO  2021-04-02 17:18:40,870 [shard 1] compaction - [Compact system_schema.keyspaces 78319860-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-55-big-Data.db:level=0, /var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-52-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,874 [shard 2] compaction - [Compact system_schema.dropped_columns 783234a0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-50-big-Data.db:level=0, /var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,874 [shard 1] compaction - [Compact system_schema.indexes 783234a0-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-46-big-Data.db:level=0, /var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-43-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,874 [shard 1] compaction - [Compact system_schema.tables 783234a1-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-49-big-Data.db:level=0, /var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-46-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,877 [shard 2] compaction - [Compact system_schema.indexes 7832a9d0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-44-big-Data.db:level=0, /var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:40,878 [shard 2] compaction - [Compact system_schema.tables 7832d0e0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-50-big-Data.db:level=0, /var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-47-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,016 [shard 2] compaction - [Compact system.truncated 77f1f750-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-44-big-Data.db:level=0, ]. 126kB to 63kB (~50% of original) in 421ms = 150kB/s. ~256 total partitions merged to 3.
INFO  2021-04-02 17:18:41,068 [shard 2] compaction - [Compact system_schema.keyspaces 784fcec0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-56-big-Data.db:level=0, /var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-53-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,090 [shard 1] compaction - [Compact system.truncated 77f1f750-93d7-11eb-a51c-000000000002] Compacted 3 sstables to [/var/lib/scylla/data/system/truncated-38c19fd0fb863310a4b70d0cc66628aa/md-46-big-Data.db:level=0, ]. 167kB to 56kB (~33% of original) in 417ms = 134kB/s. ~384 total partitions merged to 3.
INFO  2021-04-02 17:18:41,098 [shard 1] compaction - [Compact system_schema.computed_columns 77f1f751-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-52-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 421ms = 12kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,099 [shard 1] compaction - [Compact system.local 77f1f752-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/md-46-big-Data.db:level=0, ]. 18kB to 12kB (~64% of original) in 421ms = 28kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,099 [shard 1] compaction - [Compact system_schema.scylla_tables 77f1f754-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-52-big-Data.db:level=0, ]. 62kB to 57kB (~90% of original) in 421ms = 135kB/s. ~256 total partitions merged to 2.
INFO  2021-04-02 17:18:41,099 [shard 1] compaction - [Compact system_schema.columns 77f1f753-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-52-big-Data.db:level=0, ]. 69kB to 61kB (~88% of original) in 422ms = 145kB/s. ~256 total partitions merged to 2.
INFO  2021-04-02 17:18:41,104 [shard 2] compaction - [Compact system_schema.computed_columns 77f1f751-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/md-53-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 424ms = 12kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,107 [shard 2] compaction - [Compact system_schema.columns 77f1f752-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/md-53-big-Data.db:level=0, ]. 58kB to 49kB (~84% of original) in 425ms = 117kB/s. ~256 total partitions merged to 3.
INFO  2021-04-02 17:18:41,107 [shard 2] compaction - [Compact system.compaction_history 77f21e60-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-23-big-Data.db:level=0, ]. 182kB to 94kB (~52% of original) in 424ms = 223kB/s. ~256 total partitions merged to 44.
INFO  2021-04-02 17:18:41,107 [shard 2] compaction - [Compact system_schema.scylla_tables 77f1f753-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/md-53-big-Data.db:level=0, ]. 51kB to 45kB (~88% of original) in 425ms = 106kB/s. ~256 total partitions merged to 3.
INFO  2021-04-02 17:18:41,126 [shard 1] compaction - [Compact system_schema.functions 7858a860-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-19-big-Data.db:level=0, /var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-16-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,131 [shard 2] compaction - [Compact system_schema.functions 785944a0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-17-big-Data.db:level=0, /var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-20-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,190 [shard 2] compaction - [Compact system_schema.dropped_columns 783234a0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-53-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 194ms = 28kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,192 [shard 1] compaction - [Compact system_schema.keyspaces 78319860-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-58-big-Data.db:level=0, ]. 61kB to 56kB (~91% of original) in 239ms = 236kB/s. ~256 total partitions merged to 2.
INFO  2021-04-02 17:18:41,193 [shard 1] compaction - [Compact system.compaction_history 7862e190-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-19-big-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-16-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,224 [shard 2] compaction - [Compact system_schema.indexes 7832a9d0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-50-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 252ms = 21kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,250 [shard 1] compaction - [Compact system_schema.indexes 783234a0-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/md-49-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 252ms = 22kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,257 [shard 2] compaction - [Compact system_schema.tables 7832d0e0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-53-big-Data.db:level=0, ]. 57kB to 48kB (~84% of original) in 337ms = 145kB/s. ~256 total partitions merged to 3.
INFO  2021-04-02 17:18:41,269 [shard 1] compaction - [Compact system_schema.tables 783234a1-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/md-52-big-Data.db:level=0, ]. 67kB to 59kB (~88% of original) in 319ms = 187kB/s. ~256 total partitions merged to 2.
INFO  2021-04-02 17:18:41,307 [shard 1] compaction - [Compact system_schema.types 787446b0-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-16-big-Data.db:level=0, /var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-19-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,309 [shard 2] compaction - [Compact system_schema.types 787494d0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-20-big-Data.db:level=0, /var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,345 [shard 2] compaction - [Compact system_schema.keyspaces 784fcec0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/md-59-big-Data.db:level=0, ]. 49kB to 44kB (~89% of original) in 214ms = 206kB/s. ~256 total partitions merged to 3.
INFO  2021-04-02 17:18:41,367 [shard 2] compaction - [Compact system_schema.functions 785944a0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-23-big-Data.db:level=0, ]. 11kB to 5kB (~51% of original) in 178ms = 32kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,367 [shard 1] compaction - [Compact system_schema.functions 7858a860-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/md-22-big-Data.db:level=0, ]. 11kB to 5kB (~51% of original) in 181ms = 32kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,377 [shard 1] compaction - [Compact system.compaction_history 7862e190-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-22-big-Data.db:level=0, ]. 152kB to 98kB (~64% of original) in 151ms = 649kB/s. ~256 total partitions merged to 50.
INFO  2021-04-02 17:18:41,402 [shard 2] compaction - [Compact system_schema.triggers 7882c5a0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-20-big-Data.db:level=0, /var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,417 [shard 1] compaction - [Compact system_schema.triggers 78850f90-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-16-big-Data.db:level=0, /var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-19-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,436 [shard 2] compaction - [Compact system_schema.types 787494d0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-23-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 92ms = 60kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,451 [shard 1] compaction - [Compact system_schema.types 787446b0-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/md-22-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 110ms = 51kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,492 [shard 2] compaction - [Compact system_schema.view_virtual_columns 78908140-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-20-big-Data.db:level=0, /var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,509 [shard 1] compaction - [Compact system_schema.dropped_columns 78931950-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-49-big-Data.db:level=0, /var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-46-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,525 [shard 2] compaction - [Compact system_schema.triggers 7882c5a0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-23-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 90ms = 61kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,541 [shard 1] compaction - [Compact system_schema.triggers 78850f90-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/md-22-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 91ms = 61kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,572 [shard 2] compaction - [Compact system_schema.aggregates 789cb640-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-20-big-Data.db:level=0, /var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,597 [shard 1] compaction - [Compact system_schema.view_virtual_columns 78a086d0-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-19-big-Data.db:level=0, /var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-16-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,610 [shard 2] compaction - [Compact system_schema.view_virtual_columns 78908140-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-23-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 80ms = 70kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,626 [shard 1] compaction - [Compact system_schema.dropped_columns 78931950-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/md-52-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 87ms = 63kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,658 [shard 2] compaction - [Compact system_schema.views 78a9d5a0-93d7-11eb-ae89-000000000001] Compacting [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-20-big-Data.db:level=0, /var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-17-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,682 [shard 1] compaction - [Compact system_schema.aggregates 78ad5810-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-19-big-Data.db:level=0, /var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-16-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,697 [shard 2] compaction - [Compact system_schema.aggregates 789cb640-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-23-big-Data.db:level=0, ]. 11kB to 5kB (~51% of original) in 86ms = 66kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,713 [shard 1] compaction - [Compact system_schema.view_virtual_columns 78a086d0-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a0418295cfaa/md-22-big-Data.db:level=0, ]. 10kB to 5kB (~51% of original) in 84ms = 67kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,759 [shard 2] compaction - [Compact system_schema.views 78a9d5a0-93d7-11eb-ae89-000000000001] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-23-big-Data.db:level=0, ]. 13kB to 6kB (~51% of original) in 84ms = 82kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,763 [shard 1] compaction - [Compact system_schema.views 78b9db30-93d7-11eb-a51c-000000000002] Compacting [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-19-big-Data.db:level=0, /var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-16-big-Data.db:level=0, ]
INFO  2021-04-02 17:18:41,780 [shard 1] compaction - [Compact system_schema.aggregates 78ad5810-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/md-22-big-Data.db:level=0, ]. 11kB to 5kB (~51% of original) in 81ms = 70kB/s. ~256 total partitions merged to 1.
INFO  2021-04-02 17:18:41,817 [shard 1] compaction - [Compact system_schema.views 78b9db30-93d7-11eb-a51c-000000000002] Compacted 2 sstables to [/var/lib/scylla/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/md-22-big-Data.db:level=0, ]. 13kB to 7kB (~51% of original) in 46ms = 152kB/s. ~256 total partitions merged to 1.
{"L":"DEBUG","T":"2021-04-02T17:18:45.206Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/storage_service/host_id","duration":"0ms","status":200,"bytes":151}
{"L":"DEBUG","T":"2021-04-02T17:18:45.207Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/snitch/datacenter?host=10.100.118.129","duration":"0ms","status":200,"bytes":11}
{"L":"DEBUG","T":"2021-04-02T17:18:45.208Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/snitch/datacenter?host=10.100.197.104","duration":"0ms","status":200,"bytes":11}
{"L":"DEBUG","T":"2021-04-02T17:18:45.208Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/gossiper/endpoint/live/","duration":"0ms","status":200,"bytes":17}
{"L":"DEBUG","T":"2021-04-02T17:18:45.209Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/storage_service/nodes/joining","duration":"0ms","status":200,"bytes":2}
{"L":"DEBUG","T":"2021-04-02T17:18:45.209Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/storage_service/nodes/leaving","duration":"0ms","status":200,"bytes":2}
{"L":"DEBUG","T":"2021-04-02T17:18:45.209Z","N":"member_controller.scylla_client","M":"HTTP","host":"192.168.3.230:10001","method":"GET","uri":"/storage_service/nodes/moving","duration":"0ms","status":200,"bytes":2}
{"L":"DEBUG","T":"2021-04-02T17:18:45.209Z","N":"member_controller","M":"Host readiness","host":"10.100.118.129","status":"D","state":"N","_trace_id":"URgf91RhRA2APtK_9oZJLA"}
{"L":"DEBUG","T":"2021-04-02T17:18:45.209Z","N":"member_controller","M":"Host readiness","host":"10.100.197.104","status":"D","state":"N","_trace_id":"URgf91RhRA2APtK_9oZJLA"}
{"L":"ERROR","T":"2021-04-02T17:18:45.209Z","N":"member_controller","M":"Readiness check failed, node not ready","_trace_id":"URgf91RhRA2APtK_9oZJLA"}
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down gossiping
INFO  2021-04-02 17:18:45,457 [shard 0] gossip - gossip is already stopped
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down gossiping was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage service notifications
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage service notifications was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair API
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair API was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down messaging service API
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down messaging service API was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage service messaging
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage service messaging was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair message handlers
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair message handlers was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair service
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down repair service was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down streaming service
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down streaming service was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage proxy RPC verbs
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down storage proxy RPC verbs was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down cdc
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down cdc was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down sstables format selector
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down sstables format selector was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down migration manager
INFO  2021-04-02 17:18:45,457 [shard 0] migration_manager - stopping migration service
INFO  2021-04-02 17:18:45,457 [shard 1] migration_manager - stopping migration service
INFO  2021-04-02 17:18:45,457 [shard 2] migration_manager - stopping migration service
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down migration manager was successful
INFO  2021-04-02 17:18:45,457 [shard 0] init - Shutting down database
INFO  2021-04-02 17:18:45,457 [shard 0] compaction_manager - Asked to stop
INFO  2021-04-02 17:18:45,457 [shard 0] compaction_manager - Stopping 0 ongoing compactions
INFO  2021-04-02 17:18:45,457 [shard 0] compaction_manager - Stopped
INFO  2021-04-02 17:18:45,457 [shard 1] compaction_manager - Asked to stop
INFO  2021-04-02 17:18:45,457 [shard 2] compaction_manager - Asked to stop
INFO  2021-04-02 17:18:45,457 [shard 1] compaction_manager - Stopping 0 ongoing compactions
INFO  2021-04-02 17:18:45,457 [shard 2] compaction_manager - Stopping 0 ongoing compactions
INFO  2021-04-02 17:18:45,457 [shard 1] compaction_manager - Stopped
INFO  2021-04-02 17:18:45,457 [shard 2] compaction_manager - Stopped

Environment:

tnozicka commented 3 years ago

Aren't you missing step 4.1 to delete the PVCs?

tnozicka commented 3 years ago

Oh, I see what you mean, but that's not supported as the services you've deleted hold the ClusterIPs so in theory this could bootstrap with new ClusterIPs but replacing seed nodes is not supported yet and so far the replace operation is only manual.

srfaytkn commented 3 years ago

I see, so how can I manually make this operation using the same pvc? Thanks.

tnozicka commented 3 years ago

Currently, you can't change the IPs of seed nodes so your best chance would be to recreate the Kubernetes Service objects with the same ClusterIPs as they used to have (same as scylla node). But I have'nt tried it so check with some toy cluster first.

melgenek commented 3 years ago

I followed the same scenario and made it start by purging the existing seed list. In order to do this, I went to every node and started it outside the ring: 1) vi /etc/scylla/scylla.yaml. Add line load_ring_state: false 2) supervisorctl restart scylla 3) delete all the peers that are left from the old deployment

cqlsh
SELECT peer, rpc_address FROM system.peers;
DELETE FROM system.peers WHERE peer='the old non-existing ip here';

4) Join the ring again: vi /etc/scylla/scylla.yaml. Delete line load_ring_state: false 5) supervisorctl restart scylla

There's also an alternative "brutal" solution:

rm -rf /var/lib/scylla/data/system/peer*
supervisorctl restart scylla
rzetelskik commented 3 years ago

I've tried to recreate this issue and it seems to be working fine now. Steps I've taken to reproduce it:

  1. Deployed a cluster with 2 nodes on GKE. The IPs of the created services were 10.120.4.47, 10.120.8.254.
  2. Ran the following cassandra-stress test: 
    cassandra-stress write no-warmup n=100000 cl=ONE -mode native cql3 connectionsPerHost=1 -col n=FIXED\(5\) size=FIXED\(64\)  -pop seq=1..10000000 -node "scylla-cluster-client.scylla.svc" -rate threads=50  -log file=/cassandra-stress.load.data -schema "replication(factor=1)" -errors ignore; cat /cassandra-stress.load.data
  3. Deleted the cluster.
  4. Deployed the cluster again. Both nodes got up and the newly assigned IPs were 10.120.0.50, 10.120.13.12.
  5. Ran a corresponding read cassandra-stress. No problems there, so no data was lost.
tnozicka commented 3 years ago

@rzetelskik would you mind sending a PR with an e2e test covering this case before we close it?