search

google / j2cl

Java to Closure JavaScript transpiler
Apache License 2.0
1.22k stars 143 forks source link

J2CL Now & Next #93

Open gkdn opened 4 years ago

gkdn commented 4 years ago

By popular request, I am listing the things that J2CL team is actively working on apart from continuous improvements and maintenance and what to expect in the next quarters:

Last Edit: Jul 26, 2023

Now

Next

Later

Completed in previous quarters

Note:

The J2CL code is production ready and battle tested with Google Apps for years. Only open-source tooling is missing some workflows. All of our improvements to the released code are in sync with Google and you are getting the improvement at the same time as Google gets.

At the moment for J2CL, we are only planning to do dated releases (instead of semantic versioning like v1) and make more tools work in open source over time.

rov63rus commented 3 years ago

Are you planning gradle support ? Bazel is not known to any IDE.

tbroyer commented 3 years ago

https://docs.bazel.build/versions/master/ide.html ? (my main problem with the IntelliJ IDEA plugin though is that it's currently not compatible with latest IDEA 2020.2: https://github.com/bazelbuild/intellij/issues/1998)

sgammon commented 3 years ago

i would love to see some gRPC support on this list :)

treblereel commented 3 years ago

Are there any workarounds how to run tests, before j2cl_test ll be ported ? It's a critical feature

rjahn commented 2 years ago

any updates here?

gkdn commented 2 years ago

I update it every now and then but comment date doesn't reflect that (you need to click edited to see the change dates).

LifeIsStrange commented 2 years ago

Any update on Kotlin support? That would be amazing and java projects are increasingly hybridized with Kotlin

gkdn commented 2 years ago

We just started Kotlin work. Our goal is to have something usable by end of the year.

LifeIsStrange commented 2 years ago

@gkdn that is amazing to hear but Kotlin JS is officially developed and has an excellent and mature support for transpiling Kotlin to JS (and generating typescript types) However while kotlin multiplarform/js is great, and while j2cl allow to transpile Java to js, I feel like the main area for improvement would be for J2CL to leverage existing Kotlin js transpiling ability and to be interoperable with the generated js from Java. If you could work on making it easier to have hybrid Kotlin-Java projects that target js that would be amazing, maybe that the biggest issue is about IDE supports at the boundaries (Kotlin js will not allow calling java code even if this java code transpile to J2CL). Improving this IDE interop between Java and Kotlin JS should be possible in theory, either by contributing directly to the Kotlin js compiler or by making a Kotlin compiler plugin (note that the replacement API for the new Kotlin compiler (K2) is in active development https://youtrack.jetbrains.com/issue/KT-49508 ) Also you might consider using arrow-meta which is a library that helps building compiler plugins. https://github.com/arrow-kt/arrow-meta BTW since you work at Google, maybe you could contact one of the official Google kotlin devs, as you can see here: https://kotlinlang.org/docs/kotlin-foundation.html#current-personnel for some guidance. The best would be to contact Roman Elizarov or to open a youtrack ticket for a collaboration :) I really hope there is the possibility of making a synergy with JetBrains! Anyway just my two cents on how to achieve the best of both worlds.

BTW: I concur that Gradle support would be a step in the right direction for adoption.

gkdn commented 2 years ago

Our high demanding customers relies on very Closure style centric Google JavaScript stack (which J2CL is optimized for) and they count every single byte for performance. Unfortunately Kotlin/JS output doesn't match our needs and sometimes in a way that might be contradicting with the Kotlin open-source users needs who rely on different JavaScript stacks.

So for now our main focus we will be enabling Kotlin to those customers. Being said that I already met with Roman on this and we are in contact with JetBreains. We will be still looking into bringing the two worlds together in the long run.

LifeIsStrange commented 2 years ago

@gkdn I suppose you could apply a post processing step to the generated kotlin-js by feeding it to Closure. You could also work on improving the efficiency of the IR backend. But sure feel free to implement your own incompatible transpiler.

treblereel commented 1 year ago

@gkdn just in case, do you have any updates about testing support for Open Source ?

gkdn commented 1 year ago

We have an intern working on it at the moment but we will see how it will turn out.

treblereel commented 1 year ago

We have an intern working on it at the moment but we will see how it will turn out.

ok, and i suppose you have a deadline right ?

gkdn commented 1 year ago

Should be within 2 months; if the project is successful.

treblereel commented 1 year ago

Should be within 2 months; if the project is successful.

It looks like it failed ... So pity :(

gkdn commented 1 year ago

Actually basic unit testing support is released, sample app and getting started code is updated accordingly.

I haven't closed the task since it still doesn't support test suites but I will update the notes.

Edit: updated

treblereel commented 5 months ago

@gkdn Do you plan to add support for the following methods from Java 9-11? Stream: takeWhile, dropWhile, ofNullable, and iterate String: isBlank, lines, repeat, strip, stripLeading, and stripTrailing List/Set/Map: copyOf Collectors: toUnmodifiable*

And if not, can we make a PR?

rluble commented 5 months ago

We do accept contributions for missing jre methods.

Note that the implementations need to :

Isn't String.repeat is already in the JRE?

niloc132 commented 5 months ago

String.repeat is indeed already implemented - but can you direct us how to run its tests? (I assume tests are required for such submissions, even though it wasn't on your list ;). )

I see that jre/javatests/com/google/j2cl/jre/java/lang/StringTest.java has a testRepeat method, but there don't seem to be any bazel targets to run individual test classes. https://github.com/google/j2cl/issues/93#issuecomment-1416544473 says (and seems to still be correct) that suites cannot run, and only jre/javatests/com/google/j2cl/jre/LangSuite.java references StringTest.

Here's the other things I tried, just in case I missed the obvious: I noticed that LangSuite is referenced by the Lang j2cl_multi_test in jre/javatests/BUILD`, but running that results in

$ bazel test //jre/javatests:Lang                                          
ERROR: Skipping '//jre/javatests:Lang': error loading package 'jre/javatests': Label '//javascript/tools/jscompiler/builddefs:flags.bzl' is invalid because 'javascript/tools/jscompiler/builddefs' is not a package; perhaps you meant to put the colon here: '//:javascript/tools/jscompiler/builddefs/flags.bzl'?
ERROR: error loading package 'jre/javatests': Label '//javascript/tools/jscompiler/builddefs:flags.bzl' is invalid because 'javascript/tools/jscompiler/builddefs' is not a package; perhaps you meant to put the colon here: '//:javascript/tools/jscompiler/builddefs/flags.bzl'?
INFO: Elapsed time: 0.207s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: jre/javatests

Typically when we see //javascript labels, or references at all to jscompiler, it means that this isn't available to be run outside of Google. Removing that import line

load("//javascript/tools/jscompiler/builddefs:flags.bzl", "VERBOSE_WARNING_FLAGS_STRICT", "WHITESPACE_ONLY_FLAGS")

and all of the conformance test wiring that depends on it at least advances us to a new error:

$ bazel test //jre/javatests:Lang
ERROR: Skipping '//jre/javatests:Lang': error loading package 'jre/javatests': Label '//third_party/bazel_rules/rules_closure/closure:defs.bzl' is invalid because 'third_party/bazel_rules/rules_closure/closure' is not a package; perhaps you meant to put the colon here: '//third_party:bazel_rules/rules_closure/closure/defs.bzl'?
ERROR: error loading package 'jre/javatests': Label '//third_party/bazel_rules/rules_closure/closure:defs.bzl' is invalid because 'third_party/bazel_rules/rules_closure/closure' is not a package; perhaps you meant to put the colon here: '//third_party:bazel_rules/rules_closure/closure/defs.bzl'?
INFO: Elapsed time: 0.068s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading: jre/javatests

Fortunately this is also not terrible to fix: there are two separate load() calls for closure_js_binary - removing the second one still leaves the first (though to just run Lang, I've commented out both of them).

Next, I was getting this error:

$ bazel test //jre/javatests:Lang
ERROR: Traceback (most recent call last):
        File "/home/colin/workspace/j2cl/jre/javatests/BUILD", line 85, column 16, in <toplevel>
                j2cl_multi_test(
        File "/home/colin/workspace/j2cl/jre/javatests/j2cl_multi_test.bzl", line 50, column 25, in j2cl_multi_test
                j2kt_native_test(
        File "/home/colin/workspace/j2cl/build_defs/internal_do_not_use/j2kt_test.bzl", line 16, column 21, in j2kt_native_test
                j2cl_test_common(
        File "/home/colin/workspace/j2cl/build_defs/internal_do_not_use/j2cl_test_common.bzl", line 274, column 13, in j2cl_test_common
                fail("Unknown platform: " + platform)
Error in fail: Unknown platform: J2KT-NATIVE

It looks like j2cl_multi_test itself isn't able to run in open source due to J2KT-NATIVE platform not being populated? The code only has branches for CLOSURE and WASM. Next I took the drastic step of removing all j2cl_multi_test (getting past the point of what I could comfortably suggest might even make sense to commit...), and instead wrote a custom test target:

j2cl_test(
    name = "Lang",
    test_class = "com.google.j2cl.jre.LangSuite",
    runtime_deps = [":emul_tests_lib"],
)

This hit compile errors, oddly - it seems that emul_tests_lib itself isn't set up properly? How can this run inside of Google at all? Or do you have entirely separate BUILD files (not just BUILD vs BUILD.bazel), and the checked in BUILD is never used? To fix, I changed my approach, not trying to run tests at all any more, but just get them to build, via bazel build //jre/javatests:emul_tests_lib. First I had to add //third_party:jspecify_annotations-j2cl to the deps list. Next, it seems that the list of disabled errorprone checks in emul_tests_lib is inconsistent with currently checked-in code - TreeMapTest.ConflictingKey either needs a SuppressWarnings on it, or a flag passed to errorprone - I opted for -Xep:ComparableType:OFF, as that seems to be the convention.

At this point, emul_tests_lib can succeed in building.

Next, I tried to build (not test) the modified Lang rule above - this failed with a confusing message:

ERROR: /home/colin/workspace/j2cl/jre/javatests/BUILD:129:10: Executing genrule //jre/javatests:genLang_test.js failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

        FAIL: j2cl_test currently supports testing with a single testsuite only.
        IF YOU HAVE MULTIPLE TESTSUITES, WE DO NOT KNOW IF ALL OF YOUR TESTS PASS OR NOT!

Is it accurate to say that suites don't work, but a single suite would? Regardless, I changed my custom rule to only run StringTest:

j2cl_test(
    name = "StringTest",
    test_class = "com.google.j2cl.jre.java.lang.StringTest",
    runtime_deps = [":emul_tests_lib"],
)

This passed a bazel build finally, but tests fail (specifically testIndexOfNull).

-> 02:03:46.792 : Running test: testIndexOfNull
-> $module$exports$junit$framework$Assert$impl$m_fail__java_lang_String__void$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:1832:421
-> $testIndexOfNull$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4879:179
[native code]
$invokeFunction_$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3869:42
$safeRunTest_$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3855:32
[native code]
$goog$testing$TestCase$Continuation_$run$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4226:99
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3780:167
[native code]
$goog$testing$TestCase$Continuation_$run$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:4226:99
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3778:47
$goog$Promise$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:2631:32
$JSCompiler_StaticMethods_runTestsReturningPromise$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:3777:22
$JSCompiler_StaticMethods_JSC$2497_execute$$@http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:5582:159
http://localhost:50362/filez/com_google_j2cl/jre/javatests/StringTest_bin.js:5714:53
-> 02:03:46.793 : testIndexOfNull : FAILED

Is there an easier way to go about running tests, so that we can migrate already-working java9-11 stream/optional/collection code into j2cl, complete with tests?

Would any/all of this be suitable for a contribution, so that a CI script could exist that runs these emulation tests, in anticipation of emulation patches?

gkdn commented 5 months ago

The JRE unit tests are not ported to open-source so they would not work out of the box.

We are currently swamped with couple of a big projects but if you like to commit to the JRE emulation, the best I can offer at the moment is:

treblereel commented 5 months ago

@gkdn Thanks a million; it works like a charm.

niloc132 commented 5 months ago

Great, thank you @gkdn - different changes than I had made, but clearly solving the same issues. I appreciate the pointer to the _debug test target. @treblereel has submitted #222 with changes from GWT, verified in j2cl by running tests as you described.