Error while setting up Reaper

[Ashelam]

Project board link

I am trying to set up Reaper for a 2 DC Cassandra cluster and I get the below error, - can some one please suggest

WARN [2023-01-03 23:01:34,352] [main] i.c.s.CassandraStorage - Starting db migration from 0 to 30… WARN [2023-01-03 23:01:34,575] [Dev Cluster-worker-3] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='insert into schema_migration(applied_successful, version, script_name, script, executed_at) values(?, ?, ?, ?, ?)' WARN [2023-01-03 23:01:34,583] [Dev Cluster-worker-0] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='INSERT INTO schema_migration_leader (keyspace_name, leader, took_lead_at, leader_hostname) VALUES (?, ?, dateOf(now()), ?) IF NOT EXISTS USING TTL 300' WARN [2023-01-03 23:01:34,586] [Dev Cluster-worker-1] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='DELETE FROM schema_migration_leader where keyspace_name = ? IF leader = ?' WARN [2023-01-03 23:01:35,485] [Dev Cluster-worker-2] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='insert into schema_migration(applied_successful, version, script_name, script, executed_at) values(?, ?, ?, ?, ?)' WARN [2023-01-03 23:01:35,489] [Dev Cluster-worker-3] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='INSERT INTO schema_migration_leader (keyspace_name, leader, took_lead_at, leader_hostname) VALUES (?, ?, dateOf(now()), ?) IF NOT EXISTS USING TTL 300' WARN [2023-01-03 23:01:35,492] [Dev Cluster-worker-0] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='DELETE FROM schema_migration_leader where keyspace_name = ? IF leader = ?' ERROR [2023-01-03 23:01:35,594] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly... org.cognitor.cassandra.migration.MigrationException: Error during migration of script 017_add_custom_jmx_port.cql while executing 'ALTER TABLE cluster ADD properties text;' at org.cognitor.cassandra.migration.Database.execute(Database.java:269)

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: REAP-78

[adejanovski]

Hi @Ashelam,

Did you replicate the reaper_db keyspace on the two DCs correctly? If the migrations failed after an ALTER TABLE statement but before the end of a migration for some reason, it can leave the db in a faulty state like this which won't recover. Then I'd recommend to drop the keyspace and recreate it, then restart Reaper to re-run the migrations.

[Ashelam]

Hi @adejanovski

Yes, I have replicated the reaper_db keyspace correctly

sysadmin@cqlsh> desc reaper_db;

CREATE KEYSPACE reaper_db WITH replication = {'class': 'NetworkTopologyStrategy', 'Cin-DC': '1', 'Nor-DC': '1'} AND durable_writes = true;

and still I am getting the same issue

[Ashelam]

Hello Team,

Any update here?

[Ashelam]

Hello @adejanovski

Just wanted to check if you have any update here.

[adejanovski]

Hi @Ashelam,

did you drop the keyspace, recreate it and run the migrations again? Did the migration fail once and then you restarted another without dropping the content of the keyspace? It's complaining about that statement ALTER TABLE cluster ADD properties text, which usually means that the properties column already exists, because that statement was processed already in the past, but not the whole migration. If you're struggling with getting all those migrations run in your cluster due to instabilities (schema changes need to propagate throughout the cluster, which can be expensive), then I'd recommend to start Reaper on a test cluster that is unused, dump the schema and run the statements manually.

Also, I see that you're placing a single replica per DC. This is not how Cassandra is supposed to be used and that's going to get you into troubles. I recommend to put 3 replicas in each DC to get better availability. Reaper doesn't store a lot of data anyway, so that's not gonna cost you much in disk space.

[Ashelam]

Let me check it out and get back to you ...