Closed labra closed 2 years ago
HI, @labra this issue has been actively reported in Spark 3 and Cats 2.2.0 (https://github.com/typelevel/cats/issues/3628, https://issues.apache.org/jira/browse/SPARK-33077). Apparently should be solved... Anyway, I managed to submit to spark the application providing the flags indicated by https://github.com/typelevel/cats/issues/3628#issuecomment-859810657.
Unfortunately, once I try this workaround I ended on #2.
Usage of
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true
is more harmful than beneficial.
By using these flags, any spark class loses preference over any of the user's flags or any other flags that may be on the user's premises. Therefore the errors that appear afterward are very painful to solve. The solution we have found is to shadow the dependencies of cats that we use to be able to use them under another name. In our case we simply add the following lines in the build.sbt
and set Spark dependencies as provided.
assembly / assemblyShadeRules := {
val shadePackage = "org.globalforestwatch.shaded"
Seq(
ShadeRule.rename("cats.kernel.**" -> s"$shadePackage.cats.kernel.@1").inAll
)
}
This solution was found in this GitHub commit https://github.com/wri/gfw_forest_loss_geotrellis/commit/bef3d50cd107ed024b997d4203486f78965e0122.
Looks like this issue hsa been solved in the latest wdsub releases.
Although the project seems to work locally and the tests pass. When trying to launch it using
spark-submit
it raises an error that doesn't find some cats libraries.We were using
decline
to parse command line arguments and it failed because it didn't find the cats methods used by decline, so we changed it to use plain arguments. It seems to start working, but crashes when starting to work.At this moment this is the output of the crash:
My feeling is that the assembly merging policy we use to generate the assembly is failing either because it discards some required file to locate those classes or because it uses an old version of cats.
This is the build.sbt lines that declare the assembly merging policy:
and when assembling, the system says:
One intuition is that some required META-file is discarded...
The documentation on sbt-assembly describes several strategies about assembling that we may need to consider.