search

scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra/parquet files. Alt. from DynamoDB to Scylla Alternator.
https://migrator.docs.scylladb.com/stable/
Apache License 2.0
55 stars 34 forks source link

execution error: yarn.ApplicationMaster ... "DecodingFailure" #24

Closed cansato closed 4 years ago

cansato commented 4 years ago

Moreno (from Scylla) send to us a migrator .jar file some time ago and we use it to migrate 16Tb of data. It worked fine.

Now, I cloned the git repository and built jar from master.

When I ran this built jar in Yarn, I got this error: 20/09/03 18:06:33 ERROR yarn.ApplicationMaster: User class threw exception: DecodingFailure(Attempt to decode value on failed cursor, List(DownField(type), DownField(source))) DecodingFailure(Attempt to decode value on failed cursor, List(DownField(type), DownField(source)))

I tried with original spark version 2.4.4 an tried again building jar with 2.4.0 spark dependencies (our Saprk Cluster). In both I got this error.

Can you help me?

This is Yarn log for 2.4.0 spark version. application_1599154406118_0378.log

Here are terminal build messages: scylla-migrator-build-2.4.0.txt

Here execution files: (I renamed the jar file because Moreno's jar file was in the same folder) config_bcl_bg.bgdtcoo.yaml.txt spark_expurgo.sh.txt

tarzanek commented 4 years ago

repasting from other channels: missing the type field under source in the config. Check out the example config, the format changed slightly

tarzanek commented 4 years ago

https://github.com/scylladb/scylla-migrator/blob/master/config.yaml.example#L3