Minimize classpath for notebook cells

(@alexarchambault we might have discussed this long ago. Writing it up to make sure I'm understanding current state of things, and to seed any "what to do about it" discussions if anyone wants to pickup the work.)

Below I'm listing libraries which must be included in the notebook cells' classloader. I think it's the transitive dependencies of Almond's scala-kernel-api.

I'm generating this list in a locally-installed Almond kernel (0.6.0, Scala 2.11), by evaluating:

// Cell:
(new AnyRef{}).getClass.getClassLoader.getParent.getParent.asInstanceOf[java.net.URLClassLoader].getURLs.filterNot(_.toString.endsWith("-sources.jar")).map(url => url.toString.split("/").last).sorted.foreach(println)

// Output:
ammonite-interp_2.11.12-1.6.7.jar
ammonite-ops_2.11-1.6.7.jar
ammonite-repl_2.11.12-1.6.7.jar
ammonite-runtime_2.11-1.6.7.jar
ammonite-terminal_2.11-1.6.7.jar
ammonite-util_2.11-1.6.7.jar
argonaut-shapeless_6.2_2.11-1.2.0-M11.jar
argonaut_2.11-6.2.3.jar
coursier-cache_2.11-1.1.0-M14-3.jar
coursier-core_2.11-1.1.0-M14-3.jar
coursier_2.11-1.1.0-M14-3.jar
fansi_2.11-0.2.6.jar
fastparse_2.11-2.1.0.jar
geny_2.11-0.1.6.jar
interpreter-api_2.11-0.6.0.jar
javaparser-core-3.2.5.jar
javassist-3.21.0-GA.jar
jline-reader-3.6.2.jar
jline-terminal-3.6.2.jar
jline-terminal-jna-3.6.2.jar
jna-4.2.2.jar
jupyter-api_2.11-0.6.0.jar
jvm-repr-0.4.0.jar
macro-compat_2.11-1.1.1.jar
os-lib_2.11-0.2.9.jar
pprint_2.11-0.5.4.jar
requests_2.11-0.1.8.jar
scala-collection-compat_2.11-0.3.0.jar
scala-compiler-2.11.12.jar
scala-kernel-api_2.11.12-0.6.0.jar
scala-library-2.11.12.jar
scala-parser-combinators_2.11-1.0.4.jar
scala-reflect-2.11.12.jar
scala-xml_2.11-1.2.0.jar
scalaparse_2.11-2.1.0.jar
scopt_2.11-3.7.1.jar
shapeless_2.11-2.3.3.jar
sourcecode_2.11-0.1.5.jar
ujson_2.11-0.7.4.jar
upack_2.11-0.7.4.jar
upickle-core_2.11-0.7.4.jar
upickle-implicits_2.11-0.7.4.jar
upickle_2.11-0.7.4.jar

I have personally had to monkey-patch upgrade the JNA version here to work-around an issue w/ other libraries I needed. I can imagine others hitting issues w/ shapeless.

@jameskoch It's smaller than that in the latest 0.10.x versions:

def cp(cl: ClassLoader = Thread.currentThread.getContextClassLoader): Stream[java.net.URL] =
  if (cl == null) Stream() else {
    val current = cl match {
      case u: java.net.URLClassLoader => u.getURLs.toStream
      case _ => Stream()
    }
    current ++ cp(cl.getParent)
  }

cp()
  .toVector
  .map(_.toURI.toASCIIString)
  .sorted
  .filter(_.startsWith("file:/Users/alexandre/Library/Caches/Coursier/v1/"))
  .map(_.stripPrefix("file:/Users/alexandre/Library/Caches/Coursier/v1/"))
  .filter(!_.endsWith("-sources.jar"))
  .foreach(println)

gives

https/jitpack.io/com/github/jupyter/jvm-repr/0.4.0/jvm-repr-0.4.0.jar
https/repo1.maven.org/maven2/com/github/scopt/scopt_2.13/3.7.1/scopt_2.13-3.7.1.jar
https/repo1.maven.org/maven2/com/lihaoyi/ammonite-interp-api_2.13.3/2.1.4-11-307f3d8/ammonite-interp-api_2.13.3-2.1.4-11-307f3d8.jar
https/repo1.maven.org/maven2/com/lihaoyi/ammonite-ops_2.13/2.1.4-11-307f3d8/ammonite-ops_2.13-2.1.4-11-307f3d8.jar
https/repo1.maven.org/maven2/com/lihaoyi/ammonite-repl-api_2.13.3/2.1.4-11-307f3d8/ammonite-repl-api_2.13.3-2.1.4-11-307f3d8.jar
https/repo1.maven.org/maven2/com/lihaoyi/ammonite-util_2.13/2.1.4-11-307f3d8/ammonite-util_2.13-2.1.4-11-307f3d8.jar
https/repo1.maven.org/maven2/com/lihaoyi/fansi_2.13/0.2.9/fansi_2.13-0.2.9.jar
https/repo1.maven.org/maven2/com/lihaoyi/geny_2.13/0.6.0/geny_2.13-0.6.0.jar
https/repo1.maven.org/maven2/com/lihaoyi/os-lib_2.13/0.7.0/os-lib_2.13-0.7.0.jar
https/repo1.maven.org/maven2/com/lihaoyi/pprint_2.13/0.5.9/pprint_2.13-0.5.9.jar
https/repo1.maven.org/maven2/com/lihaoyi/sourcecode_2.13/0.2.1/sourcecode_2.13-0.2.1.jar
https/repo1.maven.org/maven2/io/get-coursier/interface/0.0.24/interface-0.0.24.jar
https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.3.1/jna-5.3.1.jar
https/repo1.maven.org/maven2/org/jline/jline/3.15.0/jline-3.15.0.jar
https/repo1.maven.org/maven2/org/scala-lang/modules/scala-collection-compat_2.13/2.1.6/scala-collection-compat_2.13-2.1.6.jar
https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.3/scala-compiler-2.13.3.jar
https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.3/scala-library-2.13.3.jar
https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.3/scala-reflect-2.13.3.jar
https/repo1.maven.org/maven2/sh/almond/interpreter-api_2.13/0.10.3/interpreter-api_2.13-0.10.3.jar
https/repo1.maven.org/maven2/sh/almond/jupyter-api_2.13/0.10.3/jupyter-api_2.13-0.10.3.jar
https/repo1.maven.org/maven2/sh/almond/scala-kernel-api_2.13.3/0.10.3/scala-kernel-api_2.13.3-0.10.3.jar

All these JARs are pulled (directly or transitively) by sh.almond:scala-kernel-api_2.13.3:0.10.3 (cs resolve -r jitpack sh.almond:scala-kernel-api_2.13.3:0.10.3 lists those).

jna is pulled via scala-compiler, it can probably be excluded as it shouldn't be used from almond:

$ cs resolve -r jitpack sh.almond:scala-kernel-api_2.13.3:0.10.3 \
    --what-depends-on net.java.dev.jna:jna
  Result:
└─ net.java.dev.jna:jna:5.3.1
   └─ org.scala-lang:scala-compiler:2.13.3
      └─ com.lihaoyi:ammonite-interp-api_2.13.3:2.1.4-11-307f3d8
         └─ com.lihaoyi:ammonite-repl-api_2.13.3:2.1.4-11-307f3d8
            └─ sh.almond:scala-kernel-api_2.13.3:0.10.3

A few more dependencies could be hidden. That needs to be done from Ammonite. Ammonite has a --thin option, where it doesn't expose (most of) its internal dependencies (it then only exposes com.lihaoyi:::ammonite-interp-api and com.lihaoyi:::ammonite-repl-api and their dependencies, IIRC) Almond uses a similar trick as --thin to hide both its internal dependencies and those of Ammonite.

With a bit more work, Ammonite could probably hide

com.github.scopt::scopt
com.lihaoyi::ammonite-ops
com.lihaoyi::geny
com.lihaoyi::os-lib
org.jline:jline

The resulting dependency set would be quite small then I think (mostly the scala-compiler JARs, and JARs for user-facing APIs)

almond-sh / almond

Minimize classpath for notebook cells #614