search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
55 stars 34 forks source link

Upgrade to scala 2.12 and spark 3.0.1 and fix SSL set-up issue #53

Closed WentingWu666666 closed 3 years ago

WentingWu666666 commented 3 years ago

This PR makes scylla-migrator compatible with spark-cassandra-connector-assembly 3.0.1. Since there is no TokenRangeAccumulator in the 3.0.1 spark connector, any related stuff is commented out.

And it also fixes SSL support issue. The default SSL protocol in scylla is TLS. However, some Cassandra clusters are using TLSv1.2. So SSL protocol should be set when using scylla.

tarzanek commented 3 years ago

TokenRangeAccumulator is an important feature to be able to resume from savepoints

tarzanek commented 3 years ago

DynamoDB migration is also a feature, so we cannot just comment it out :-)

tarzanek commented 3 years ago

for ssl support I just merged https://github.com/scylladb/scylla-migrator/pull/49 so this PR needs more work @WentingWu666666

tarzanek commented 3 years ago

while on the topic, even current version has some compatibility issues with cloud setups of clusters (private IPs) due to 4.x cassandra drivers in current spark connector supported the version before DynamoDB integration and connector upgrade seems to be the last stable for all clusters (but unfortunately for spark 2.4)