NPE and CCE during parse or Javaslang

[l0rinc]

Hey,

I've tried compiling Javaslang to TypeScript using jsweet-maven-plugin version 1.2.0-SNAPSHOT and got lots of NullPointerExceptions and a ClassCastExceptions, e.g. at javaslang/collection/AbstractIterator.java:6:

java.lang.NullPointerException
    at org.jsweet.transpiler.util.Util.isImported(Util.java:1185)
    at org.jsweet.transpiler.typescript.Java2TypeScriptTranslator.visitIdent(Java2TypeScriptTranslator.java:2502)
    at com.sun.tools.javac.tree.JCTree$JCIdent.accept(JCTree.java:2011)

and

ERROR: internal transpiler error at /media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java(20)
java.lang.ClassCastException: com.sun.tools.javac.code.Type$ForAll cannot be cast to com.sun.tools.javac.code.Type$MethodType
    at org.jsweet.transpiler.typescript.Java2TypeScriptTranslator.visitClassDef(Java2TypeScriptTranslator.java:797)
    at com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:693)

[l0rinc]

cc @danieldietrich

[renaudpawlak]

Translating a complex Java code base to JavaScript would require to use the latest snapshot to make sure you have all latest fixes available. So please try using v2.0.0-SNAPSHOT. If you hit the same issues, I will be able to push a fix quite easily.

[l0rinc]

Thanks @renaudpawlak, tried the Maven plugin with 2.0.0-SNAPSHOT

2017-03-29 17:26:23.023 ERROR output:48 - internal transpiler error at /media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java(37)
dumping transpiler's strack trace:
   [JCIdent] Iterato... (/media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java:37)
   [JCMethodDecl] @Override()... (/media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java:37)
   [JCClassDecl] public abstract class Abstract... (/media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java:20)
   [JCCompilationUnit] package javaslang.collection;... (/media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java:6)
java.lang.NullPointerException
    at org.jsweet.transpiler.util.Util.isImported(Util.java:955)
    at org.jsweet.transpiler.Java2TypeScriptTranslator.visitIdent(Java2TypeScriptTranslator.java:3228)
    at com.sun.tools.javac.tree.JCTree$JCIdent.accept(JCTree.java:2011)

and

2017-03-29 17:26:23.023 INFO  JSweetTranspiler:861 - output file: javaslang/collection/AbstractIterator.ts
   [JCCompilationUnit] package javaslang.collection;... (/media/papus/TI31436700A/_Projects/javaslang/javaslang/src/main/java/javaslang/collection/AbstractIterator.java:6)
java.lang.ClassCastException: com.sun.tools.javac.code.Type$ForAll cannot be cast to com.sun.tools.javac.code.Type$MethodType
    at org.jsweet.transpiler.Java2TypeScriptTranslator.visitClassDef(Java2TypeScriptTranslator.java:1198)
    at com.sun.tools.javac.tree.JCTree$JCClassDecl.accept(JCTree.java:693)

I would gladly contribute, if you there there's a chance Javaslang can be compiled to TypeScript. BTW, using GWT we're already compiling to JavaScript, it might be enough to just generate the TypeScript type definitions.

[l0rinc]

cc @ruslansennov

[renaudpawlak]

Ok. For the first error, JSweet gives you a compile trace of where the error occurs... However, when I look a the code of AbstractIterator it does not look to be the same code... (https://github.com/javaslang/javaslang/blob/master/javaslang/src/main/java/javaslang/collection/AbstractIterator.java)

Are you compiling the master of Javaslang?

From what I see in the code, absolutely nothing looks like it would not compile with JSweet (I have compiled more complex case).

Hopefully, I'd could give it a try within the coming week if you tell me what to compile.

Besides, the null pointer exception in Util.isImported looks quite easy to fix (I can do it or you can contribute), but I would prefer to understand the root cause.

[l0rinc]

I've tried creating a test for AbstractIterator transpilation and I get the error:

INFO  JSweetTranspiler:1249 - org/jsweet/test/transpiler/AbstractIterator.ts(17,27): error TS2420: Class 'AbstractIterator<T>' incorrectly implements interface 'Iterator<T>'.
ERROR: class 'AbstractIterator<T>' incorrectly implements interface 'Iterator<T>' at src/test/java/org/jsweet/test/transpiler/AbstractIterator.java(21)
INFO  JSweetTranspiler:1249 -   Property 'forEachRemaining' is missing in type 'AbstractIterator<T>'.

which is a default method in java.util.Iterator#forEachRemaining. Are default methods not supported yet?

[l0rinc]

Update: if I add the two missing methods like:

@Override
public void forEachRemaining(Consumer<? super T> action) {
}

@Override
public void remove() {
}

the test passes, however if I try to call the super default:

@Override
public void forEachRemaining(Consumer<? super T> action) {
    Iterator.super.forEachRemaining(action);
}

@Override
public void remove() {
    Iterator.super.remove();
}

it crashes again with

INFO  JSweetTranspiler:1249 - org/jsweet/test/transpiler/AbstractIterator.ts(30,13): error TS2693: 'Iterator' only refers to a type, but is being used as a value here.
ERROR: 'Iterator' only refers to a type, but is being used as a value here at src/test/java/org/jsweet/test/transpiler/AbstractIterator.java(36)
INFO  JSweetTranspiler:1249 - org/jsweet/test/transpiler/AbstractIterator.ts(34,13): error TS2693: 'Iterator' only refers to a type, but is being used as a value here.
ERROR: 'Iterator' only refers to a type, but is being used as a value here at src/test/java/org/jsweet/test/transpiler/AbstractIterator.java(41)

It seems to me full Java 8 compilation (or parsing) is far from finished.

[renaudpawlak]

It seems to me full Java 8 compilation (or parsing) is far from finished.

I would not put it this way if you don't mind ;)

You have first to be aware that JSweet's primary goal was not to support any JDK/JRE APIs. JSweet's primary goal was to support only JavaScript APIs. JSweet was designed to be exactly like TypeScript, but with the Java syntax. In the first version of JSweet, the compiler would report an error when trying to access any Java type except for a very small subset (String, Numbers, ...). In counterpart, what you have in JSweet is full access to JavaScript APIs!

Now, because of popular demand, JSweet has evolved to support more and more Java constructs and APIs (especially through the J4TS project). But compiling an actual Java project remains difficult (Java 8 or not). It will remain difficult because JSweet is not a Java emulator and is not meant to fully respect the Java semantics (although it is getter closer). On the other hand, what you get is the ability to fully interoperate with TypeScript/JavaScript, because the generated code is actual JavaScript and not a Java emulation written in JavaScript.

With JSweet 2, JSweet now comes with an extension mechanism that allows the programmers to erase the Java APIs at compile time to bridge them directly to JavaScript APIs. For instance, with JSweet 2 you could write an extension that substitutes all the calls to the BigDecimal API to call the Big.js API directly (with no runtime overhead). This technique brings a lot of interoperability and is very light at runtime (you can use the best of JavaScript APIs and not emulated Java libraries).

Unfortunately, this is still WIP and not documented. So you may want to postpone a little your attempt to compile Javaslang. On the other hand, I have already used JSweet 2 to compile quite complex projects and that would be interesting to try it the right way just to evaluate what remains to be done. For instance, if you could share your JSweet/Javaslang compilation env in a github project, I would be able to give advice and adjust what would need to be adjusted.

So, I suggest that you try compiling Javaslang with JSweet 2 removing J4TS from your dependencies. If you remove J4TS from your Maven deps, JSweet 2 should automatically try to erase the JDK APIs. It might not work fully yet because all the JDK has not been tested, but it may give an idea of what remains to be done to have it working. Plus, the good thing with erasing the JDK is that you don't need the J4TS runtime anymore (which makes the generated code lighter and more efficient for most cases).

MORE INFO ON DEFAULT METHODS:

Looking closer at Javaslang, it looks like there will issues with wildcard imports (which is not a big issue), and with default methods indeed! But this last issue should not happen if you don't use J4TS.

FYI, default methods are supported but with limitations. The current implementation "copy-pastes" the default method implementations in the target type. This cannot work if the default method is not in the source base you are compiling. Here it cannot work probably because JSweet cannot access the source code of the default method declared in Iterator.

I believe the current implementation is bad and should be changed because the source code of an interface should not be required to compile. Once again, this problem should not happen if you don't use J4TS, unless I really miss something, which is alway possible at this point :(

[l0rinc]

Thank you for your detailed answer, I didn't mean any disrespect, it was just a realization from my part that I'm trying something in a way that it wasn't designed to do.

Is it worth though reinventing Java parsing and JavaScript transpiling, when GWT does it quite well? Could JSweet maybe build on that?

Will try it without J4TS, thanks for the hint :)

[renaudpawlak]

I have to answer this :)

When you compile with GWT, you are stuck with GWT. Your compiled Java code only works in a GWT environment, with the GWT's JRE emulation and whatever libs have been compiled with GWT. That's cool. But I have talked with some companies that are really in trouble for the long run because they have bet on GWT and they cannot interoperate with other really important JS frameworks they would need to use. GWT JSInterop makes things better but to me it is not sufficient and requires runtime adaptations in the generated code. IMO, when you want real interop with JavaScript frameworks, using GWT will most probably bring you trouble in the long run (performance and modularisation issues typically).

JSweet's goal is to be a transpiler only with no runtime at all and generating plain JavaScript-compatible objects. It means you can directly use JSweet-generated objects from any JavaScript framework. It also means that you can drop your Java base at any time and start coding in TypeScript because there is not much code pollution in the generated code.

Anyway, I wish I had more time to communicate on this and explain it better. I hope to be able to write more about it when JSweet v2 will be out.

[pedjapesic]

@Renaud can you please express your thoughts about J2CL or you will save your comments when this is available for public use ?

Thanks in advance.

On Thu, Mar 30, 2017 at 4:11 PM, Renaud Pawlak notifications@github.com wrote:

I have to answer this :)

When you compile with GWT, you are stuck with GWT. Your compiled Java code only works in a GWT environment, with the GWT's JRE emulation and whatever libs have been compiled with GWT. That's cool. But I have talked with some companies that are really in trouble for the long run because they have bet on GWT and they cannot interoperate with other really important JS frameworks they would need to use. GWT JSInterop makes things better but to me it is not sufficient and requires runtime adaptations in the generated code. IMO, when you want real interop with JavaScript frameworks, using GWT will most probably bring you trouble in the long run (performance and modularisation issues typically).

JSweet's goal is to be a transpiler only with no runtime at all and generating plain JavaScript-compatible objects. It means you can directly use JSweet-generated objects from any JavaScript framework. It also means that you can drop your Java base at any time and start coding in TypeScript because there is not much code pollution in the generated code.

Anyway, I wish I had more time to communicate on this and explain it better. I hope to be able to write more about it when JSweet v2 will be out.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cincheo/jsweet/issues/234#issuecomment-290422694, or mute the thread https://github.com/notifications/unsubscribe-auth/AJhKjl_2Lba-qOxLcBcg6CdBouXF-RF6ks5rq7gEgaJpZM4Ms1Ba .

[renaudpawlak]

Sure, I can do that. Need some time to write it though (time that I don't have this week). Don't hesitate to ping me if I don't give any news.

[pedjapesic]

ping (for comment)

[renaudpawlak]

:)

Well I don't have a lot of time but I hope I will be able to post more information on the blog soon!

In the meantime, let me talk not specifically about J2CL, but any approach that would consist of generating JavaScript code from Java, trying to respect the Java semantics (which is the case of J2CL, but also other transpilers like TeaVM). I probably don't know J2CL enough anyway. So maybe I misunderstand some things about it, but once again, I will talk general here.

Let us look at the following piece of code:

public class Test {
    public static int squareCount(int toSquare) {
        int i = 0;
        while (toSquare >= 1) {
            toSquare= toSquare * toSquare;
            i++;
        }
        return i;
    }

    public static int squareCount(double toSquare) {
        int i = 0;
        while (toSquare < Double.MAX_VALUE) {
            toSquare= toSquare * toSquare;
            i++;
        }
        return i;
    }

    public static void main(String[] args) {
        System.out.println(squareCount(10));
        System.out.println(squareCount(10L));
    }
}

Basically, the squareCount function, which is overloaded for ints and longs, counts the number of times the given number can be squared before overflowing.

If you want to fully respect the Java Language Semantics (JLS), you are in trouble because you need to fully emulate longs and ints, as well as overloading. Because of overloading, numbers cannot be pure JS numbers anymore, otherwise you cannot dissociate ints from longs and you cannot select the right overloaded method. Besides, you need to overload the multiplication behavior in order to respect Java ints and longs behaviors.

Since core JS objects are not modifiable in the JS VM, you need to wrap your ints and longs in wrapper objects. For example, when you have in Java squareCount(10), you would need to generate JS code that would look like: squareCount({ type: "int", value:10}). Since you also need to overload the multiplication behavior, the best way is to actually use a Java int object: squareCount(new java_int(10)). Then you can transpile toSquare * toSquare with toSquare.java_mul(toSquare).

This will work. You will have a full Java emulation. But it comes with many drawbacks. To show a few:

1. It is slower (much slower?). I bet that the VM cannot optimize as well as it would do for native JS numbers.
2. It requires a runtime, like the GWT's JRE emulation, that will implement the JLS for Java objects (with JSweet v2 you will not need any runtime).
3. It generates complicated to understand code because all the intention of the program is delegated within the emulation layer.
4. It is bad for JS interoperability: imagine you want to call the squareCount function from JS, you cannot call it with a native JS number directly. A solution would be that the generated code adds some code to map native JS numbers to emulate Java numbers with a default behavior, but it would add more noise in the code and make the interop slow. Another solution would be to explicitly wrap your JS objects in Java emulated objects when calling the function. That would be terrible too.
5. JavaScript frameworks will not work with such generated objects. Most JavaScript frameworks expect native objects. Imagine for instance that you want to JSON-serialize an object that would contain fields that are numbers... You would need a special serialization engine to make it compatible with regular serialization. ...

So, as far as I know, if you want JLS, there is no way to overcome these issues [NOTE: you could try using ES5 accessors but that would be complicated too I believe]. J2CL wants to fully emulate Java, so it will not escape these issues like all other transpilers trying to achieve the same goals. Of course, I am aware that J2CL should support the JSInterop annotations to allow defining/bridging native JS objects in Java, but at the end, you will get the same problems interoperating between emulated Java objects and pure JS objects.

JSweet does not try to fully emulate the JLS, because JSweet want full interop with JavaScript (to avoid all the aforementioned issues). As a consequence, if you look at the example, JSweet will generate the ints and longs as native JS number and will not care if it preserves the JLS or not. This is just an example, but also, with JSweet 2, Java collections are generated as native JS arrays when not using the J4TS runtime (now optional).

So, some statements will not behave like in Java, but like in JavaScript. Programmers need to be aware of it. This can be bad if you want to port legacy Java code to JavaScript, but the good news is that most code will be able to work as is, especially when using regular objects (only low-level calculations using primitive types pose complicated compatibility problems). The other good news is that if you manage to compile your Java programs with JSweet you will gain the following:

1. Your JSweet-compiled program will be as fast as a manually-written JavaScript program (for most cases)
2. Your JSweet-compiled program will be readable and you will be able to drop your Java code base if your want.
3. Your JSweet-compiled objects will be usable as is by JS frameworks.
4. Your JSweet-compiled program will be compatible as is will all the JS echo system (modules, Node.js, etc.). Remember that JSweet generates code going through TypeScript.

Because JSweet wants to match JS as much as possible, JSweet v2 will open the transpiler to allow in-depth tuning of the generated code depending on your use case. For instance, you can write a transpiler extension that will redirect all the calls to BigDecimal to the Big.js framework (it is quite easy to do). It means that you will use a JS framework that is specifically written for JS and with JS (instead of a BigDecimal emulation).

So, as you can understand, the main point here is that the more we try to emulate Java, the worse it will get for interop. In JSweet, we believe that it is more important to get interop/speed rather than emulation. First because most Java APIs already have similar implementation in JS (which are efficiently developed for JS). For example I have just transpiled a SAX-parser based implementation by redirecting the Java SAX to a JS SAX implementation (and it works great!). Second because most old legacy Java applications use outdated APIs (for instance Swing), that would gain being renovated anyway. It is an illusion to thing that you can transpile any Java code and have it work efficiently in the browser (what about threads, file system, ...??). At some point you will have to work adapting your code to target the web, and JSweet v2 aims at proposing a compile-time framework to do it efficiently and with maximized performance and interop.

I am probably still missing some points here, but I hope this will help understanding better the challenges and issues with Java -> JS transpilation (and where JSweet stands).

[pedjapesic]

Thank you very much on your answers @renaudpawlak. I do understand what you mean, the only thing I do not understand quite well is what are your plans about the future versions of JSweet e.g. what changes will come with v2 and does it mean that there will be no need for J4TS ? Do you have any time frame for v2 ?

Thanks again.

[renaudpawlak]

There are many small improvements in V2 and more support for Java semantics without compromising the JavaScript interoperabilty I have talked about. But if we speak about major improvements (justifying a new major version), there are 2.

The first major improvement for JSweet V2 is to make it possible to write your own definitions/candies in a more fexible way. It is very easy to bridge an existing JavaScript API by adding a def package in your program's source code. It was not possible that easily with previous JSweet versions. Also, candies source codes are now available for modification/adaptation in https://github.com/jsweet-candies. Anyone can modify and contribute. It is a more collaborative/open approach compared to the previous generative approach which was a black-box approach. Last but not least, we have open sourced the candy generator tool, which will allow anyone to translate a TypeScript definition file to a JSweet-compatible Java API. All this is explained here: https://github.com/cincheo/jsweet/tree/master/candy-generator

The second major improvement for JSweet V2 is to expose an extension API to customize the transpiler. An extension consists of subclassing the main components of the transpiler to adapt the generated code to your context. In particular, the "printer adapter" API will allow to write chainable extensions (so that you can compose extensions easily). For instance you can write two adapters:

• A "BigDecimalAdater" adapter that redirect all the calls to BigDecimal to the Big.js API.
• A "SAXAdater" adapter that redirect all the calls to Java SAX to a JS SAX API.

Then you will be able to use any of these extensions independently or together. [ATTENTION: adapters are compile-time entities, they require no runtime... for instance an adapter can translate a statement new BigDecimal("1000") in your Java code to a statement new Big("1000") in your generated JavaScript code... all this in a safe and composable way]

JSweet 2 comes with an adapter (RemoveJavaDependenciesAdapter) that overrides the default extension in order to replace all the calls to J4TS by native JS code, so that the generated code does not need any runtime anymore. [for example: List<String> l = new ArrayList<String>() translates to let l : string[] = []. The cool thing is that JSweet detects if J4TS is in the classpath and chooses the right root adapter for you. So if you have J4TS in your classpath, JSweet will assume that you use a runtime and will select the default root adapter (and conversely, JSweet will use the other adapter). On the top of the root adapter, you can add up your own list of adapters.

The adapter API starts being quite stable. A good thing is that it does not depend on the javac API but on the standard javax.lang.model.element API. I have been using it for various customers to develop custom JSweet extensions. Some will be Open Source so I will advertise them when the time will come.

The plan is to release asap, but having full-time projects while developing JSweet makes the release cycle quite long. Probably some time before or during the summer. In the meantime using the snapshot is fine for most usage because the main refactoring have been done and the API is now stable.

[pedjapesic]

Thank you @renaudpawlak.