scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra
Apache License 2.0
54 stars 34 forks source link

Add initial testing infrastructure #107

Closed julienrf closed 4 months ago

julienrf commented 4 months ago

Move the main module into a submodule migrator and add end-to-end tests in another submodule named tests.

The tests require a complete stack to be set up with services like DynamoDB, Scylla, and Spark. This is achieved with Docker Compose.

We also introduce a GitHub Workflow that runs the tests on every pull request and every commit pushed to the main branch. The GitHub Workflow has been checked locally with act, but we need to merge the PR to really test it on GitHub.

We add a couple of tests that cover DynamoDB to Alternator migrations. Note that scenarios involving Kinesis (ie, when streamChanges = true) can not be tested locally because the spark-kinesis project performs calls to the real AWS servers.

Fixes #106

/cc @tarzanek and @guy9

tarzanek commented 4 months ago

merging to action-tests branch to fine tune actions, will merge to master after tests work there

tarzanek commented 4 months ago

thank you!

tarzanek commented 4 months ago

https://github.com/scylladb/scylla-migrator/actions/runs/8229745228 runs ...