promotion of flybytes to usethesource

[jurgenvinju]

Project

• https://github.com/cwi-swat/flybytes
• lead: @jurgenvinju

Motivation

This may be (is) quite an early proposal, for a project such as flybytes, but I see a not-so-distant future where the rascal compiler uses flybytes to directly generate (debuggable) JVM bytecode. For this to be possible flybytes needs to mature a bit more:

• support nested classes;
• support a flybytes extension library for generating code for lambdas and closures;
• finetune the code generator to generate even more idiomatic JVM code.
• write better documentation and create demonstration movies

Otherwise flybytes seems to be a perfect back-end for the Rascal compiler. It supports "block expressions", for example, which can make code generation considerably easier. Also other language features of Rascal (closures) could relatively easily be modeled as extensions to the flybytes compiler (i.e. flybytes to flybytes stages).

Flybytes also offers Mirrors which can be used to call any method or set/get any field in JVM bytecode directly from Rascal (using Java's reflection API and Rascal closures). It is used to test the flybytes compiler and decompiler, but it has many other applications in interfacing Rascal with arbitrary Java code. The Mirror bridge also marshalls basic types between JVM and Rascal.

Due to the (relatively complete) flybytes decompiler, it is easy to first write Java code to implement a Rascal language feature manually, then reverse engineer this into a flybytes AST pattern to extend the compiler. This feature can also be used when migrating from the Java code generator to a flybytes backend.

It would be cool if flybytes also supports an "intermediate code" debugger. Namely that one could step through the generated flybytes code instead of the original Rascal code that is associated with it. There is an idea on how to do this, by writing out indented AST values to a temporary folder and replacing the .src annotations with locations from these generated files, but it is not yet implemented either.

Another missing feature is an "API generator". Such a tool would read class files (using flybytes) and generate Rascal functions which will generate calls to the public API of such a class file. Currently users of flybytes must type in (copy/paste) the relatively complex method signatures into their code generators and use the "right method invocation primitive" from the JVM (invokeInterface, invokeStatic, invokeVirtual, invokeSpecial).

Question

I would like to invite you to have a look at flybytes and see what you think about the current quality and support. What would be required for you to accept and support this project at usethesource?

Vote

• [x] @DavyLandman
• [x] @PaulKlint
• [x] @tvdstorm
• [x] @jurgenvinju

[DavyLandman]

Can the mirror feature also avoid reflection but use lambda factory and the other new features added to make dynamic code without reflection? (Method handles and friends)

[jurgenvinju]

The mirror feature can not avoid reflection; something must be explained better:

• A lambda factory "macro library" can easily be written for flybytes. We can produce Rascal functions that will generate JVM bytecode that make use of the lambda factory and method handles. In fact most of the dyncall features are already supported by the current flybytes compiler. Only lambdafactory support should be added on top of it.
• The Mirror feature is to call generated bytecode from Rascal (directly from the Rascal REPL), for example after JVM bytecode generation you want to call the generated code and check the output. It also interacts nicely with the random test feature of Rascal. So the whole concept of the mirror feature is that it is a reflection of the reflection API.

[DavyLandman]

I'm for adding this as a library, if you think it has the right maturity.

Two questions I have:

1. What does moving support to java 11 for example entail?
2. Is there a good path for automated testing of it?

[jurgenvinju]

1. It runs on Java 11 on my machine already
2. It has many unit and regression tests written in Rascal for every little part of the compiler (unit-style) and for the whole of the decompiler (regression style). For these tests to be executed by mvn all we need is a wrapper class that calls the RascalJUnitRunner.

[jurgenvinju]

• maturity: it has a lot of potential; it is pretty complete compiler for "abstract Java" (except nested classes) and an 80% functional decompiler which is useful for getting inspiration on what to generate for your DSL.

[jurgenvinju]

If we use it as back-end for Rascal, some things have to be added; I think a library for generating implementations of lambda's would be nice. This does not require extending the Java code of the compiler.

As a back-end for little languages and DSLs it's already very functional. Especially the debugger support is where the finesse lies, and the proper handling of monitor blocks and nested try-finally blocks is nice. Also it has taken some time to generate code that is somewhat efficient, i.e. by partial evaluation of boolean comparisons and negations thereof, and avoiding code duplication.

[DavyLandman]

Sounds good to me. I indeed meant: java 11 features/class level support, but if you feel thats something you could add without too much redesign 👍🏼.

[jurgenvinju]

Definitely, most java 11, 13, 15, and 17 features are in the translation from Java source code to JVM bytecode, as far as I know, so such features could be added by "desugaring" to flybytes. If core compiler support is required for efficiency's sake, that can always be a follow-up implementation.

[jurgenvinju]

Everyone voted yes now; I'll move the project soon and start automating the test runner from maven.