macro classloader

[mkeskells]

macros run in a separate classloader to the compiler, for obvious reasons

there seems to be unexploited opportunities to share classes between the classloaders

[jvican]

I measured this, and the time to classload is less than 1ms IIRC.

[mkeskells]

@jvican - the cost is not the cost of loading the classes. A duplicate class loader duplicates the C1 and C2 compile and generated duplicate code that runs on the CPU, so the impact is very deep, all the way through to the running code and the wasted purging of the CPU cache, additioanl GC time etc

To make his worse, a multi-module compile will duplicate his for each loaded module. In Zinc 1.0 we share some classloading, but it doesn't share the macro classloding

If you have a test for this can you share what you have measured, so it can be the starting point when this task gets restarted

[jvican]

I see what you mean. Do you have any profiles or previous use cases that prove that's a sound theoretical bottleneck? Intuitively it sounds right, but I've lost faith in my intuition to reason about performance.

I propose to use CDS https://docs.oracle.com/javase/8/docs/technotes/guides/vm/class-data-sharing.html to solve this issue, which is also going to be open-sourced soon as advertised by Oracle. If you only care about the fact that you duplicate classes across different classloaders and don't benefit from optimizations/profiles in the main classloader, I believe this would solve this issue.

[jvican]

I've had a closer look at this. It seems that the macro classloader is already sharing classes info from the compiler classloader, since it uses the compiler classloader as the parent:

  protected def findMacroClassLoader(): ClassLoader = {
    val classpath = global.classPath.asURLs
    macroLogVerbose("macro classloader: initializing from -cp: %s".format(classpath))
    ScalaClassLoader.fromURLs(classpath, self.getClass.getClassLoader)
  }

[mkeskells]

when I first raised this issue it was related to the classloader running for plugins. I think that he issue will only affect that environment, where you have classes loaded by the plugins and classes loaded by the macros. There are also some special macros tha run in the compiler with a comment around classloading.

In the plugin case the plugin class loader is (I think) a sibling of the macro class loader, so there is duplication there, but this is quite specialised, and limited to when the plugin and macros share some infra (which they do in my work env). Not sure this will register any performance improvement in the common case of any of the cases that are currently benchmarked

MY idea here was that is some limited circumstances the plugin could share a parent classloader with the plugin. This is very niche, and unproven, hence the investigation label