search

almond-sh / almond

A Scala kernel for Jupyter
https://almond.sh
BSD 3-Clause "New" or "Revised" License
1.6k stars 239 forks source link

Newest release breaks AmmoniteSparkSessionBuilder with ammonite.ops.PathConvertible #277

Closed mpacer closed 5 years ago

mpacer commented 5 years ago

I started using the latest release to get the --predef file to work (which seems to work great! 👍 🙌 ) .

Once I started doing this I can no longer get the AmmoniteSparkSessionBuilder to work because of some change in how the ammonite.ops classpath is being loaded.

Specifically I'm get the following error:

java.lang.NoClassDefFoundError: ammonite/ops/PathConvertible
  org.apache.spark.sql.AmmoniteSparkSession$.builder(AmmoniteSparkSession.scala:32)
  ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
  ammonite.$sess.cmd1$.<init>(cmd1.sc:7)
  ammonite.$sess.cmd1$.<clinit>(cmd1.sc:-1)
java.lang.ClassNotFoundException: ammonite.ops.PathConvertible
  java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  ammonite.runtime.SpecialClassLoader.findClass(ClassLoaders.scala:215)
  java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  org.apache.spark.sql.AmmoniteSparkSession$.builder(AmmoniteSparkSession.scala:32)
  ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
  ammonite.$sess.cmd1$.<init>(cmd1.sc:7)
  ammonite.$sess.cmd1$.<clinit>(cmd1.sc:-1)

I have isolated this to the version of almond (not of ammonite-spark) by trying the same code outside of a predef which works with earlier versions of almond but not on 0.1.12.

I'm happy to help with fixing this, but I'm not sure where to look. Any guidance on that would be greatly appreciated.

@mseal @alexarchambault @jameskoch2

alexarchambault commented 5 years ago

A short term fix would be to bump the version of Ammonite in ammonite-spark (in project/Deps.scala), to the same as in almond say. Newer versions of ammonite-spark would be compatible with newer versions of Ammonite and almond (and no more with older versions).

Long term, I guess it would be time to lobby for more stability for the Ammonite API (ammonite.interp.InterpAPI and ammonite.repl.ReplAPI in particular), so that this kind of things couldn't happen any more. Ideally, these classes should be moved to separate modules, and things like mima ensure binary compatibility is preserved.

mpacer commented 5 years ago

:) Thanks for the guidance!