search

sbt / sbt-assembly

Deploy über-JARs. Restart processes. (port of codahale/assembly-sbt)
MIT License
1.95k stars 223 forks source link

java.lang.NoClassDefFoundError: Could not initialize class io.circe.Decoder$ after Shading #465

Open clintoncheang opened 2 years ago

clintoncheang commented 2 years ago

Hi all, I tried to shade shapeless when I did a spark-submit using sbt-assembly. In my Spark Assembly scala file, I added assembly / assemblyShadeRules := Seq(ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll). But when I submit it to Spark 3.1.2, I got this error above. I am wondering am I adding the Shading correctly?

dmykhailov commented 1 year ago

Same here. Have you found any solution so far?

joan38 commented 1 year ago

Same here when I try to upgrade to Spark 3.x. Works fine on Spark 2.4.4

mattharder91 commented 1 year ago

Possibly related to this issue:

https://github.com/typelevel/cats/issues/3628

my issue was that circe had a different version of cats than spark 3

shading should help:

assembly / assemblyShadeRules := Seq( ShadeRule.rename("shapeless.**" -> "new_shapeless.@1").inAll, ShadeRule.rename("cats.kernel.**" -> s"new_cats.kernel.@1").inAll ),