Macro classpath entries are flagged as unused

[jjudd]

Hi everyone,

I am currently working on trying out the classpath-shrinker plugin with our codebase. I ran into an issue in which the plugin warns about certain used classpath entries as unused. The classpath entries warned about contain macros. If you remove the classpath entries that are flagged as unused, the code no longer compiles.

I created a repro example here: https://github.com/lucidsoftware/classpath-shrinker-example1

In this example, I have a simple source file

package foo

object Foo {
  def main(args: Array[String]): Unit = {
    import cats.implicits._
    println(true === false)
  }
}

that compiles under both Scala 2.11 and 2.12 and warns about unused classpath entries:

2.11:

james@james:~/opensource/classpath-shrinker-example1$ ./compile.sh /home/james/Desktop/scala-2.11.12/bin/scalac 2-11 macro
/home/james/Desktop/scala-2.11.12/bin/scalac -Xplugin:2-11-cp-shrinker/classpath-shrinker_2.11-0.1.1.jar -cp 2-11/cats-core_2.11-1.0.1.jar:2-11/cats-kernel_2.11-1.0.1.jar:2-11-macro/cats-macros_2.11-1.0.1.jar:2-11-macro/machinist_2.11-0.6.1.jar Foo.scala
cat: /release: No such file or directory
warning: Detected the following unused classpath entries:
/home/james/opensource/classpath-shrinker-example1/2-11-macro/cats-macros_2.11-1.0.1.jar
/home/james/opensource/classpath-shrinker-example1/2-11-macro/machinist_2.11-0.6.1.jar
one warning found

2.12:

james@james:~/opensource/classpath-shrinker-example1$ ./compile.sh /home/james/Desktop/scala-2.12.4/bin/scalac 2-12 macro
/home/james/Desktop/scala-2.12.4/bin/scalac -Xplugin:2-12-cp-shrinker/classpath-shrinker_2.12-0.1.1.jar -cp 2-12/cats-core_2.12-1.0.1.jar:2-12/cats-kernel_2.12-1.0.1.jar:2-12-macro/cats-macros_2.12-1.0.1.jar:2-12-macro/machinist_2.12-0.6.2.jar Foo.scala
warning: Detected the following unused classpath entries:
/home/james/opensource/classpath-shrinker-example1/2-12-macro/cats-macros_2.12-1.0.1.jar
/home/james/opensource/classpath-shrinker-example1/2-12-macro/machinist_2.12-0.6.2.jar
one warning found

If I remove those classpath entries, then both 2.11 and 2.12 fail to compile:

2.11:

james@james:~/opensource/classpath-shrinker-example1$ ./compile.sh /home/james/Desktop/scala-2.11.12/bin/scalac 2-11 nomacro
/home/james/Desktop/scala-2.11.12/bin/scalac -Xplugin:2-11-cp-shrinker/classpath-shrinker_2.11-0.1.1.jar -cp 2-11/cats-core_2.11-1.0.1.jar:2-11/cats-kernel_2.11-1.0.1.jar Foo.scala
cat: /release: No such file or directory
Foo.scala:6: error: macro implementation not found: $eq$eq$eq
(the most common reason for that is that you cannot use macro implementations in the same compilation run that defines them)
    println(true === false)
                 ^
one error found

2.12:

james@james:~/opensource/classpath-shrinker-example1$ ./compile.sh /home/james/Desktop/scala-2.12.4/bin/scalac 2-12 nomacro
/home/james/Desktop/scala-2.12.4/bin/scalac -Xplugin:2-12-cp-shrinker/classpath-shrinker_2.12-0.1.1.jar -cp 2-12/cats-core_2.12-1.0.1.jar:2-12/cats-kernel_2.12-1.0.1.jar Foo.scala
Foo.scala:6: error: macro implementation not found: $eq$eq$eq
(the most common reason for that is that you cannot use macro implementations in the same compilation run that defines them)
    println(true === false)
                 ^
one error found

My knowledge of the Scala compiler is not very good. I've been poking at the plugin and the Scala source without much luck. Anyone have any guidance or recommendations?

Thanks in advance for the help!

[retronym]

A complete solution to this would probably involve changing the classloader used to reflectively load and execute macro implementations to track which JARs have actually been touched. classpath-shrinker could then take the union of those JARs and the ones touched when populating the Symbol table to determine the set of used JARs. This solution would require a new release of scalac.

An intermediate solution might be to change ~jar-lister~ classpath-shrinker to accept as a configuration parameter a whitelist of JARs that it should not treat as unused.

[jjudd]

I like the first solution, but am new to the Scala compiler, so it will likely take me a while.

I also like the idea of the intermediate whitelist solution. I'll work on that for now, it should make it easy to workaround any other issues that crop up in the future as well.

I assume when you refer to jar-lister that you are referring to the internals of the classpath-shrinker plugin? If not, I'd appreciate being pointed in the right direction.

[retronym]

Yes, I meant classpath-shrinker.

[jvican]

I think I have an alternative solution, I’ll try it out today.

[jvican]

@retronym I think we could have classpath-shrinker install a MacroPlugin and then walk the symbols of the macro expansions (we would need to walk the tree instead of the root symbol, which means it could be less efficient, but that would work). Do you think this would be a viable approach?

[retronym]

The macro expansions would not reference these "unused" JARs. If they did, some Symbol from that JAR would be loaded, and that JAR would not be reported as unused by classpath-shrinker.

The problem here is utility JARs called by the macro implementation method itself. Most macros are self contained, only calling into scala-{reflect,library}. But projects are more modular, or depend on some macro utility libraries.

Ideally, the compiler would have a separate setting for -macro-impl-classpath, rather than overloading -classpath with classes you want to reference and macro implementations and their dependencies.

A custom macro classloader could log all JARs that are actually referenced. This isn't a big change, but it can't be made without a change to the compiler itself.

[ittaiz]

@johnynek sounds like the above suggestion about -macro-impl-classpath could really help us in the whole ijar-bazel issue right?