search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra
Apache License 2.0
54 stars 34 forks source link

Tunable consistency as a config parameter #80

Closed hopugop closed 1 year ago

hopugop commented 2 years ago

Currently consistency can be tuned by passing config Spark config parameters to the submit job. This could be a configurable option in source/target entries on the config file.

Examples of Spark submit parameters:

--conf spark.cassandra.input.consistency.level=LOCAL_QUORUM 
--conf spark.cassandra.output.consistency.level=LOCAL_QUORUM

The default consistency for input is LOCAL_ONE, while output is LOCAL_QUORUM. On a cluster that does not run repairs, this means data can be inconsistent among different runs.

netguy204 commented 2 years ago

yes! And the example config should be updated to mention these new parameters as well.

hopugop commented 2 years ago

cc @tarzanek

hopugop commented 1 year ago

Merged https://github.com/scylladb/scylla-migrator/pull/89