Ceylon embedded runner

[vietj]

We need a convenient way to easily embed Ceylon in a Java runtime.

The runner should delegate module loading to a resolver that resolves the module identifier and returns a response that can be:

• the module resource response (URL / InputStream / etc... of a jar/car )
• a provided response that basically means the classes will be provided by the parent class loader and the Ceylon runtime should trust it.
• a not found response that will make the runner fail

The loader should provide a life cycle interface to init/destroy the runtime so it can be garbaged in order to be reconstructed and reloaded later (a new version of course).

[quintesse]

Maybe this issue belongs in ceylon-runtime? Somewhat related to the CMR as well, but not language I think.

[FroMage]

Non-jboss modules runtime is here so far

[alesj]

Can you give more exact scenario / use-case? Or where is the problem atm?

[vietj]

Embedding Ceylon in Vert.x runtime is my use case. I'm pretty sure that it would fit also anyone wanting to do the same in their runtime.

[vietj]

Problem is that JBoss Modules cannot be embedded in an existing runtime making it difficult to embed Ceylon.

[vietj]

To give a comparison and set the standard, here is how you do it in Groovy (and in most other JVM languages):

ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
Class groovyClass = loader.parseClass(new File("src/test/groovy/script/HelloWorld.groovy"));

// let's call some method on an instance
GroovyObject groovyObject = (GroovyObject) groovyClass.newInstance();
Object[] args = {};
groovyObject.invokeMethod("run", args);

[FroMage]

I like how you can say “and in most other JVM languages” with a straight face when I'm quite sure you've never checked ;)

So I assume that for this classloader to work you need the full Groovy jar in your classpath, right? Which means the equivalent for Ceylon would be to have the Ceylon distro already in your classpath too, right? ATM we have the ceylon classpath tool that can give you a classpath for the distro, but it's many jars. Do you think we should add something like ceylon make-distro-jar which would generate a fat jar out of all the jars in the distro so you can put it in your classpath in one go like groovy-all.jar?

Now, the other things to note:

• Apparently the GroovyClassLoader does more than load classes: it also compiles code and loads it for runtime. We currently don't have any official API for compiling or running things, unless you count direct usage of the Tool class, but perhaps that's enough for a start.
• Due to modularity, it's possible that the files you compile will require jars to be pulled from a module repo, and at runtime require an extra classpath. The current functionality we have at runtime is either fully-managed (jboss modules via run tool), or fully classpath with no classloading at all (the recent Main, with no jboss modules and flat classpath with ceylon classpath). What you're asking for seems to be a mix with the Ceylon distro in the regular classpath, but the compiled module and its dependencies in a dynamic URLClassLoader which is not using jboss modules. I guess we can do that, but you have to realise that compiling and running may require additional modules to be downloaded from Herd then.

[vietj]

To be clear, I want an API for

1/ compile resources (files) to a module object 2/ control the life cycle of this module (specially destroy to release runtime resources) 3/ get objects (top level) out of this module or function to invoke them

For the record, here is the integration of language integration doc http://vertx.io/language_support.html and the several languages integration:

• https://github.com/vietj/vertx-golo/blob/master/src/main/java/org/vertx/golo/deploy/impl/golo/GoloVerticleFactory.java
• https://github.com/bdecoste/vert.x/blob/6b852ead1da9c02478d20e0e7b5ca66ed7eab6a2/vertx-lang/vertx-lang-java/src/main/java/org/vertx/java/deploy/impl/java/JavaVerticleFactory.java

[vietj]

And here is the current ceylon prototype : https://github.com/vietj/vertx-ceylon/tree/master/src/main/java/org/vertx/ceylon/platform/impl

the compiling part and the running part are miserable.

[vietj]

Keep in mind that such API is important for adoption of Ceylon as an embedded language.

[FroMage]

I'm not questioning that it's important to do, but you haven't answered my questions. Neither Golo nor Java nor Groovy support modules, so I'm pretty sure that when you will need to compile a Ceylon file it won't be a single file and it will have module imports. How do we proceed? What's the initial classpath?

[quintesse]

And how do we automatically download modules and add them to the classpath if we can't use JBoss Modules? Btw, is that a something in JBs design that doesn't lend itself to embedding?

[FroMage]

Downloading is the CMR's job, and for the classpath we can add them to our own ClassLoader. That's not an issue really. We just have to agree on what the system should do.

[quintesse]

It's not the downloading , it's the automatic downloading, and the CMR definitely can't do that without JBoss Modules right now.

[quintesse]

Unless you don't mind that the embedded version has less functionality and we always download all dependencies, even optional ones, before passing execution on to the main module.

[vietj]

It makes sense to me that by default it uses the default behavior of the ceylon tool (.ceylon/repo) and it be overridden with specific programmatic settings.

[vietj]

I'm not questioning that it's important to do I just meant that it is important not only for me, but for the ecosystem (vert.x is part of the ecosystem).

[FroMage]

Sure, but answer my questions ;)

[vietj]

At runtime this is a bit different, if we are assuming a flat classpath, the runner should play nice with the its associated classloader and if a class is existing in this classloader then it should not load a new copy of the class itself.

The use case is that today we have the legacy vert.x jars in the module repository (Herd) however at runtime we need to use those classes provided by a classloader (the one provided by Vert.x when integrating a language).

Otherwise this result in different classes objects and lead to a class cast exception. There is a (horrible) workaround for this in the current vertx-ceylon integration.

[FroMage]

OK let me rephrase my questions then:

• Do you need the Ceylon distrib in the classpath when calling this API?
• If yes, is ceylon classpath enough to give you the classpath or do you want a ceylon make-all-jar command to generate a single jar for the Ceylon distribution?
• Are the module dependencies of the compiled Ceylon module also expected to already be in the classpath, or should we load them using the CMR?
• How do you specify the modules to compile? Module name? List of files?
• How do you specify the module to run? Module name? Class object?

[quintesse]

ceylon make-all-jar? I think I'm going to open an issue to rename ceylon to stefs-kitchen-sink ;)

[FroMage]

Well, we're not going to ship an über-jar, since that would take too much space in the distrib and it can be created from what's in the distrib. The problem is that it's much easier to add a single jar in the classpath than to add them manually. ceylon classpath helps, but it still makes for a very long classpath.

[quintesse]

Just joking Stef, classpath has already been very useful and I'm sure making an über jar will be as well. Might even take an optional module name to pack it and its dependencies as well perhaps? (hmm that almost sounds like an assembly)

[vietj]

Uberjar does not have to be in distrib, however having it in Maven Central would make lot of sense.

[quintesse]

Really? The entire 14MB thing?

[FroMage]

We don't have only jars in the distrib, it may be smaller. Groovy-all is 7mb, though I suppose it comes with the SDK.

@vietj: still waiting for a list of answers to my questions, you can't constantly comment here and not reply ;)

[quintesse]

Excluding .src doesn't even make 1MB of difference, anything else is just peanuts. You could leave out maven support for 3MB.

[FroMage]

There's also documentation and spec and samples. But OK the jars weight a total of 11M (10.5M when all jarred together). BTW, I think we have duplicate classes in those jars. org/apache/http/ConnectionClosedException.class for example (in httocore and maven-support). Or org/apache/commons/logging/Log.class (in commons-logging and maven-support). There could be others I stopped pressing overwrite buttons after a while.

[vietj]

I'll do soon :-)

[vietj]

I mean like in an hour or so

[vietj]

1/ what do you call exactly "Ceylon distrib" ?

2/ I don't want command, I want a programmatic Java API for everything

3/ for module dependencies :

• for compile time CMR is enough (at least at the moment)
• for run time both
  • provided modules by the embedding runtime (like vertx API jar) pretty much like java/base
  • resolved by CMR for user modules (like a backend service for instance)

4/ let's not talk about classpath but rather "parent loader" which is a classloader the embedding runtime will use

5/ with the JavaFileObject API or something equivalent

6/ providing a module name should resolve a "module" object, this module object should allow:

• to discover Ceylon verticles (i.e shared Ceylon classes that satisfies the io.vertx.ceylon.platform.Verticle abstract class) as some class object. Then we need to instantiate this class with Vert.x objects and then call start / stop. (https://github.com/vietj/ceylon-vertx/blob/master/source/io/vertx/ceylon/platform/Verticle.ceylon)
• to discover shared top level functions that satisfies some criteria (i.e that takes a Vertx objects as argument)

[FroMage]

1/ what do you call exactly "Ceylon distrib" ?

Well, the Ceylon distribution. All the jars needed by the Ceylon compiler and runtime.

2/ I don't want command, I want a programmatic Java API for everything

Sure, but you still have to tell me if the Ceylon distrib is in the classpath already or if the loader should load it. I guess it must be in the classpath, because the loader/API needs to be visible for you to use it. Do you have the groovy-all jar in the classpath?

3/ for module dependencies :

for compile time CMR is enough (at least at the moment)

OK

for run time both provided modules by the embedding runtime (like vertx API jar) pretty much like java/base resolved by CMR for user modules (like a backend service for instance)

Ah, so you need to give us a list of modules to not load (like provided), and where their jar is.

4/ let's not talk about classpath but rather "parent loader" which is a classloader the embedding runtime will use

Well, if you want that, you will need to tell us where the jars are, otherwise the metamodel can't represent them.

5/ with the JavaFileObject API or something equivalent

So a list of files. But it's a bit weird that you give me a list of files and I give you back a list of modules (since the files can represent more than one module).

6/ providing a module name should resolve a "module" object, this module object should allow:

So I'll give you back a list of module metamodels then. You can detect what you want with it.

[vietj]

classpath stuff...

At the moment the Java classes are already loaded by the the classloader. Vert.x 2.1 provides plugin in mod.zip (see http://vertx.io/mods_manual.html). It should not depend upon external static configuration of the JVM (like classpath or system property).

Ah, so you need to give us a list of modules to not load (like provided), and where their jar

indeed using a "provided" list seems like a good approach. Note: we need to detect the case of a user declaring a dependency that is provided but with a different version. (for instance users depends on vertx 1.0 and runtime provides vertx 1.0.1).

Well, if you want that, you will need to tell us where the jars are, otherwise the metamodel can't represent them.

do you mean the jars containing Ceylon classes that are not resolved by the CMR ? like ceylon-language ?

So a list of files...

Vert.x will not know in advance the list of modules, it will introspect them to figure out Verticles and or top level functions that accepts Vertx.

[FroMage]

At the moment the Java classes are already loaded by the the classloader. Vert.x 2.1 provides plugin in mod.zip (see http://vertx.io/mods_manual.html). It should not depend upon external static configuration of the JVM (like classpath or system property).

Well, still we will need to know the list of modules available and their list of packages.

Note: we need to detect the case of a user declaring a dependency that is provided but with a different version

Yes, we need to detect module dependency version conflict issues in a flat(ish) classpath and support overrides like we do for Maven I guess, but also for Ceylon then I suppose. That already sounds like a bigger and bigger issue every day :(

do you mean the jars containing Ceylon classes that are not resolved by the CMR ? like ceylon-language ?

Yes.

Vert.x will not know in advance the list of modules, it will introspect them to figure out Verticles and or top level functions that accepts Vertx

OK, so a list of files.

[vietj]

I think I'm a bit lost in this context discussion can we make a sync over IRC this week to be sure we agree on everything and then put this recap here ?

[FroMage]

So I started working on this, but I noticed a problem with the idea of having modules "provided" by a ClassLoader: our JVM compiler cannot use a ClassLoader to load classes. It relies on javac for that, which can only load class files from the classpath which, granted, we can set to something else than the default classpath, but still it requires access to a File for each jar. Do you think you can provide us with a File for each of the provided module for compilation-time? At runtime we can use a provided ClassLoader, so that's fine, though I guess if you have access to the File for compile-time, you will also be able to give it to us for the run-time part, which will allow us to scan the package list and free you from manually giving it to us.

[vietj]

are you using those jar files to resolve JavaFileObject classes ?

[FroMage]

No, but if the code you compile wants to import some of the provided modules, javac needs to be able to load the corresponding class files, and it can't work with a ClassLoader: it needs the jar file.

[vietj]

So we mean jar files for the provided modules, right ?

[FroMage]

Yes.

[vietj]

do we need to provide the transitive closures of the provided modules, or the declared module are enough ?

[vietj]

So that would mean in our case the vert.x core jar / jackson jar / netty jar . I see this acceptable however we should check if we can provide such resources via JavaFileObject classes.

This code does compiles java source code using ClassLoader by providing JavaFileObject with some analysis. Do you think we can ultimately do the same at some point ?

[FroMage]

I don't think it will help you to provide the jar files as JavaFileObject files for the .class files, as in both cases you can't use just the ClassLoader to get them. And yes that's a transitive closure.

[vietj]

well we'll see that makes the creation of this list hard to be dynamic and should be known in advance but that sounds doable. Anyway let's give a try and we'll see what we can work out!

[FroMage]

The list of provided modules will be all those required by Ceylon: the distrib repo, and for vert.x, I guess the vert.x jar itself, its dependencies (netty), and the vert.x Ceylon module, right?

So, how is it organised in vert.x? The Ceylon distrib jars would be in a mod.zip? Where would the Vert.x jars be?

I think this is going to be problematic, as javac's classpath only works on local files, not jars in zips. I'll investigate further. In any case, I wonder if we should not involve the CMR for the resolving of those jars. Even if they do not sit in a conventional repository, we may be able to create a custom Repository that would look them up from where they sit, and then we can use the CMR for the module dependencies, since the Ceylon jars will have module dependencies inside them, and I assume the Vert.x/Netty jars too since they should have a pom.xml in the jar. This may involve @alesj.

[vietj]

the distrib repo, and for vert.x, I guess the vert.x jar itself, its dependencies (netty), and the vert.x Ceylon module, right?

As far as I can see, yes.

The Ceylon distrib jars would be in a mod.zip

• vert.x jar and dependencies are provided in the classpath resolved from vert.x lib by vertx.sh
• mod.zip is unpacked by Vert.x before execution so this is ok so we can get the corresponding jar file, however I am not certain yet about how we can obtain this list easily

let's try it the java.io.File jar first (quicker) and then we will see if we can do better, specially for other users that may have a dynamic list of jar files based on the classloader URLs.

[FroMage]

OK good.

[FroMage]

I wonder if I should make the API take a list of arguments as a List<String> like javac does, or if I should make it type-safe with an Options class that holds all possible options.

[FroMage]

That would save me lots of trouble with argument parsing, which isn't done by the runner nor compilers: it's done by the tool plugin infra, and I can't reuse it.

[vietj]

+1 for Options typed configuration