search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
58 stars 35 forks source link

Switch to our fork of emr-dynamodb-connector #183

Closed julienrf closed 1 month ago

julienrf commented 2 months ago

Fixes #167

julienrf commented 2 months ago

Related: https://github.com/scylladb/scylla-migrator/issues/117#issuecomment-2248252132

julienrf commented 2 months ago

I tested this PR locally and manually checked that #167 is still fixed by checking the absence of logs containing “BatchWriteItem”. We also have our own test suite DynamoDBInputFormatTest checking that #130 is still fixed.

julienrf commented 2 months ago

The test failure is caused by a difference in the way we compute segments, see https://github.com/scylladb/emr-dynamodb-connector/pull/8

julienrf commented 2 months ago

This PR is ready for review (and merge!).

In case we merge it, the next release version should be 1.0.0. If needed, we could keep a branch 0.9.x alive to publish patch release from there.