dynamic blocks on the JVM

[CeylonMigrationBot]

[@FroMage] I guess we should figure out what they are supposed to do. We have a number of alternatives, ranging from doing what Scala does and just forward any dynamic method call or attribute to a method of the Dynamic type such as invoke or readAttribute, to interop with dynamic languages such as JRuby or Groovy. We need to figure out what is possible and what we want.

[Migrated from ceylon/ceylon-compiler#1102]

[CeylonMigrationBot]

[@gavinking] dynamic is really meant for interop with scripting languages, not for just disabling typechecks.

[CeylonMigrationBot]

[@FroMage] Well, so you mean this is an alternative/option that is out of question? http://weblogs.java.net/blog/cayhorstmann/archive/2012/12/13/dynamic-types-scala-210

The practical examples at the end look interesting.

[CeylonMigrationBot]

[@gavinking] Right, that's not what this is intended to be used for. It's not a Dynamic type, it is support for things whose types we don't know / can't know at compile time.

The practical examples at the end look interesting.

Nyeah, to me this is a bit lame. The approach F# (and C#?) uses is much better, I think.

[CeylonMigrationBot]

[@FroMage] This would be interesting to discuss with @headius (Charles Nutter from JRuby) and @glaforge (Guillaume Laforge from Groovy).

[CeylonMigrationBot]

[@headius] I would also like to see this work with Java...it wouldn't be any more difficult than integrating with scripting languages in most cases.

I'd certainly be willing to help bootstrap this feature. I just added invokedynamic dispatching to Redline Smalltalk in about two hours...it's not hard. For calling out to dynlangs, it ought to be similar code. I did talk a bit with Stéphane at Devoxx FR...not enough to really get details about the layout of the system, but enough to make an introduction :-)

Perhaps someone can give me pointers to where in the compiler dynamic blocks are handled.

[CeylonMigrationBot]

[@gavinking]

I just added invokedynamic dispatching to Redline Smalltalk in about two hours...it's not hard.

What might make it harder in our case is that it's not our code generating the bytecode ... we delegate to javac...

[CeylonMigrationBot]

[@headius] That definitely puts a difficult wrinkle into it. There's no way to cause javac to emit an invokedynamic, but we might be able to come close by building in our own call site caching logic and dispatching to MethodHandle directly. On current Java versions that won't perform great (old MH logic doesn't optimize well for direct dispatches) but on Hotspot 24 and higher (Java 8 builds and 7u16 or whatever the next feature release will be) I believe it should optimize better...at least better than java.lang.reflect.Method.

Of course, we'll want to measure it...and if it isn't faster than Method we can just cache Method objects for dispatch. It won't be the fastest thing in the world, but I'm not sure folk will expect dynamic blocks to be as fast as static blocks on the JVM anyway.

[CeylonMigrationBot]

[@gavinking]

There's no way to cause javac to emit an invokedynamic

Not even by hacking javac?

[CeylonMigrationBot]

[@FroMage] Well, we don't produce javac code, we produce javac AST, and that has extra nodes not present in the Java language, such as Let. I'm pretty certain we can come up with an Indy node if we have to. I definitely plan to need it for other things anyways, so it's always been on the roadmap.

[CeylonMigrationBot]

[@gavinking] @headius P.S. I just reread this thread and noticed that maybe I don't come off as very enthusiastic about this work. FTR, I'm very enthusiastic about it. I would love to see interop between Ceylon and JRuby, for example.

[CeylonMigrationBot]

[@tombentley] > I'm pretty certain we can come up with an Indy node if we have to. I definitely plan to need it for other things anyways, so it's always been on the roadmap.

I did that about a year ago (proof of concept anyway) but never got around to pushing it.

[CeylonMigrationBot]

[@FroMage] Well, push it then :)

[CeylonMigrationBot]

[@headius] Re: using javac AST directly...

Ahh well that's a different story altogether; I thought you meant you were generating Java code and delegating to javac to compile it (I have a couple languages that work that way).

Yes, it's certainly possible we could make an Indy node. Hmm...

I am actually looking at doing something similar to this in JRuby, to replace slow-path lookup-every-time calls with indy calls in bytecode by doing a transformation either at boot time or against the existing jar. Emitting a special node into javac would be cool...and could open the path to a patched javac that has dynamic invocation as part of Java too :-)

@tombentley +1 for pushing what you have. Indy's the bomb.

Let me know when you push it...that will give me a good picture of how the current compiler works. Or, if it has bitrotted, just throw up a patch on gist or something.

@gavinking Oh sure...I didn't get the impression you weren't enthusiastic. I'd love to see this work too. It would make it possible for people to write JRuby extensions (or parts of JRuby itself) in Ceylon :-)

[CeylonMigrationBot]

[@tombentley] About 10 months ago I spent a day trying to add a JCIndy node to the AST. The various details are pretty sketchy now... It uses the same basic hack which the java devs used when they were putting working on invokedynamic (though I'm sure theirs was a lot less of a hack): There's a special magic class (for us this is com.redhat.ceylon.compiler.java.Indy) which isn't typechecked in the usual way, and which ultimately ends up in the class file as an invokedynamic. I remember failing to hang it directly off Apply (i.e. method invocation), but not the exact reasons. So the JCIndy node I added is a subclass of JCFieldAccess which is used like the method name in a static method invocation. On top of this I put AbstractTransformer.makeIndy(), which constructs the Apply() with a JCIndy and provides an API for supplying the BSM, the kind of invocation and some static arguments.

I've rebased the patches onto the latest compiler and ceylon.language, and you can see them here:

https://github.com/tombentley/ceylon-compiler/tree/indy https://github.com/tombentley/ceylon.language/tree/indy

And here's a gist I wrote about it at the time, sadly lacking in useful content. https://gist.github.com/tombentley/2859919

In summary: It's very roughly hacked together and may not be suitable for basing anything off, but it works at least for the one simple example I was playing with at the time.

[CeylonMigrationBot]

[@FroMage] OK that looks pretty complex. Any reason why you're not using the existing Code.emitInvokedynamic() method?

[CeylonMigrationBot]

[@FroMage] @glaforge tells me that Groovy implements JSR-223 (http://www.jcp.org/en/jsr/detail?id=223) which means that we can transform foo.bar in foo.getProperty('bar') and foo.baz() into foo.invokeMethod('baz'), though in most cases Groovy actually compiles to a reasonable Java Class with real typed methods that we can just invoke and JavaBean properties. It's only for dynamic attributes and methods that we need to use the dynamic part. Well, and for non-typed methods/fields I guess, since types are optional in Groovy, but when they do exist we can make use of this.

[CeylonMigrationBot]

[@tombentley] I remember finding it, and finding that it has no callers, but unfortunately I don't remember if there was a good reason why I didn't or couldn't use it.

[CeylonMigrationBot]

[@FroMage] The implementation in JDK8 seems to support generating INDY (see http://hg.openjdk.java.net/jdk8/jdk8/langtools/file/cfb65ca92082/src/share/classes/com/sun/tools/javac/jvm/ClassWriter.java). Perhaps I should backport this instead?

[CeylonMigrationBot]

[@tombentley] I would definitely backport that. I seem to recall there was more faffing about trying to get the bytecode right that there was in getting the earlier phases (e.g. resolve) to work right, and it's far from certain I got it right, as I was focussing on getting that one example to work.

[CeylonMigrationBot]

[@FroMage] OK.

[CeylonMigrationBot]

[@FroMage] So I've just added the JCTree.JCIndyIdent AST node to allow us to generate Indy calls. The AST part is heavily inspired by @tombentley's work but based on JCIdent rather than JCFieldAccess, because Indy is a static call that is never on something, so it's sort of like a toplevel function IMO, and with JCFieldAccess I got confused into thinking that the selected object would be passed as parameter somehow. We could make this transformation automatic, but I didn't see the point at this stage.

The backend part is backported from the latest JDK8.

I've tested it and it works. I didn't commit any tests because we don't have any mechanism that allows us to use it ATM.

So whether that is useful for interop with dynamic languages or not, I don't really mind, at least it opens up the possibility for us to use Indy for some things if we want :)

[CeylonMigrationBot]

[@FroMage] AND naturally I forgot to rebase before pushing, so da3e5b6509fa65820c3e91dcac36b2232de1ee12 and aa9706c0b2b1e4dd954e5a578cc34619f7578bda are actually the commits.

[CeylonMigrationBot]

[@dhoehmann] I'd definitely like to have dynamic blocks on the JVM. And the should work the same way as on JS, so the same code runs on both backends.

[CeylonMigrationBot]

[@quintesse] @dhoehmann and how do you imagine that? To me it seems highly improbable that you could ever write dynamic code that would work on both backends,

[CeylonMigrationBot]

[@dhoehmann] I imagine that dynamic values inside dynamic blocks are just not type checked at compile time. Either types fit at runtime or an exception occurs. This seams to be foolish in the first place as the goal is to avoid runtime type mismatch errors. But in some cases types are not predictable at compile time. My current use case is a script interpreter that I am porting from C# to Ceylon. And if the interpreted script wants to evaluate a>b my code must somehow check if a is larger than b. This is very easy with dynamic on JS:

Object? result; try { dynamic { dynamic op1 = <whatever is referenced by 'a' in the script>; dynamic op2 = <whatever is referenced by 'b' in the script>; result = op1 <=> op2; } } catch (e) {result = e;}

But there are other use cases. When you read data from a database, a csv file, json ... you could not in every case know the data types. And sometimes you don't even need to know the exact types. In pure Ceylon one need to narrow down the types - what gets complicated if you want to sort data and need to ensure all values satisfy Comparable. Because one can't use T as a placeholder it is necessary to either check for each and every possible comparable type or use reflection to bypass type checking. And for what result? To throw a runtime error when types do not match - so I get the same behaviour as with dynamics, but with (much) more effort and at the expense of readability. What I see is the point that this would - at least to some degree - depend on the runtime type checking of the backend. So the same code may behave differently on different backends.

[CeylonMigrationBot]

[@quintesse] Well to be able to compare anything a type would need to satisfy Comparable in any case, there's no other way to do comparisons, even when you write a < b that interface is used (that we might optimize that for Java's basic types is an implementation detail).

So either a < b would literally be converted to a < b in Java which won't work for almost all types (you can't do "foo" < "bar" in Java) or we'd convert it to a call on Comparable which still would give you the possibility of errors if the runtime type doesn't support that.

It's just not that easy on the JVM.

[CeylonMigrationBot]

[@dhoehmann] a < b must in fact be converted almost the same way as in non dynamic sections. Everything else would lead to confusion. I assume it becomes something like a.compareTo(b) == -1 and within dynamic blocks this must change to something like a.getClass().getMethod("compareTo", Object.class).invoke(a, b) == -1; Of course this would raise an error if a doesn't implement comparable or if its compare implementation refuses to compare with b. But so what? If I'm in need to compare values of unknown types it either works or it produces a runtime error. And it doesn't make a real difference if my code detects this error directly or by catching possible exceptions (except the programming effort ;-)).

[CeylonMigrationBot]

[@lucaswerkmeister] Comparable works differently in Ceylon and Java. In Java, compareTo returns a number smaller than, equal to, or greater than 0 (not just -1, 0, +1!). In Ceylon, compare returns a Comparison, that is, one of the objects smaller, equal, greater. So without knowing whether a and b are Ceylon types or Java types, it’s not clear what a < b should actually do.

[CeylonMigrationBot]

[@quintesse] @lucaswerkmeister well supposedly you could detect if a type came from Ceylon or not and handle things accordingly, so supposedly that could be handled somehow. But it's even worse, dynamic code runs according to the rules of the underlying language, so comparing "foo" < 2 gives a valid boolean response on the JS backend while it will throw an exception on the JVM. There's just no way that you could write code that works the same on both backends without having to deal with a whole bunch of weird corner cases.

I see more future in an approach where in normal typed code it's made easy to compare two objects of unknown type (not that I have any suggestions).

[CeylonMigrationBot]

[@quintesse] What I mean to say is that I see dynamic as an interoperability option for the underlying backend: it allows you to do things that otherwise would be impossible to do (interop with JS code on the JS backend for example). For the JVM we don't know yet in what way dynamic could be used. But I sincerely doubt it can be used in more than a few contrived cases to write the exact same code for both backends. (But I won't mind being proven wrong)

[CeylonMigrationBot]

[@dhoehmann] @lucaswerkmeister ah ok - I see that. The compiler produces different code depending on type. And for dynamic code this distinction must be taken at runtime. This is obviously not that easy - and may lead to very - hm - interesting results when comparing a Ceylon object to a Java object or vice versa.

[CeylonMigrationBot]

[@dhoehmann] After thinking this over again I must admit that we probably don't need dynamic stuff on the JVM. The problem just arises from the fact that SomeInterface<Float> is different from SomeInterface<String>. One could ensure that a and b are of the same type and both satisfy Comparable<T> (or SomeInterface<T>) so it's obvious to the programmer that a < b would work - but the type checker doesn't buy that. So what we need is:

• a way to make the type checker know that two (multiple) values are of the same generic type, even though the actual type parameter(s) is/are unknown (but we know they are identical on all of them)
• a modification to the compiler (I assume) to output code that decides at runtime whether to call Float.compare or String.compare or WhatEver.someMethod

Sounds like a quite challenging task ... ;-)

[CeylonMigrationBot]

[@quintesse] I'm no type-system wizard, maybe @gavinking knows of a nice way where you could ask if two objects are of the same Comparable type and then allow you to do o1.compare(o2) somehow.

[CeylonMigrationBot]

[@jvasileff] @dhoehmann, @quintesse, did you see my post to ceylon-users? I believe it's what you are talking about.

[CeylonMigrationBot]

[@quintesse] @jvasileff something like that indeed