The assembly of the migrator project requires a huge amount of resources because it has to process thousands of files that are duplicated between the fat-jar of the spark-cassandra-connector and scylla-migrator’s dependencies.
On my machine, the OS often crashes when I try to build the assembly…
Instead, we should build the spark-cassandra-connector as a usual Maven/Ivy package, publish it locally, and depend on it in the scylla-migrator project.
julienrf
commented
2 weeks ago
The assembly of the migrator project requires a huge amount of resources because it has to process thousands of files that are duplicated between the fat-jar of the spark-cassandra-connector and scylla-migrator’s dependencies.
On my machine, the OS often crashes when I try to build the assembly…
Instead, we should build the spark-cassandra-connector as a usual Maven/Ivy package, publish it locally, and depend on it in the scylla-migrator project.
This PR needs https://github.com/scylladb/spark-cassandra-connector/pull/4 to be merged first.