search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
61 stars 36 forks source link

Fix AWS credentials forwarding #214

Open julienrf opened 2 months ago

julienrf commented 2 months ago

The GitHub action we use in our workflow tests-aws.yml exposes the AWS credentials by setting environment variables. We copy them to the Spark containers.

I manually triggered the tests that access AWS. The results can be seen here: https://github.com/scylladb/scylla-migrator/actions/runs/10797896910.

Unfortunately, the tests still fail: the changes applied to the source DynamoDB table after migrating the initial snapshot are never replicated to the target ScyllaDB instance.

tarzanek commented 2 months ago

interesting, so then kinesis doesn't work? or in manual tests it works and here in dockerized GH actions env it doesn't ?

julienrf commented 2 months ago

The test suite that uses Kinesis passed on my machine with my personal AWS credentials. I am not sure what is the difference in the GitHub actions except the credentials. But it is complicated to use those credentials outside the GitHub action (to try to reproduce locally)…