Importing annotated bytecode

[sgammon]

Describe the bug I'm trying to import a JAR of JsInterop-annotated Java bytecode, and it isn't working :(

To Reproduce

Setup/inputs

I'm trying to compile the following Kotlin object via J2CL:

package javatests.language

import jsinterop.annotations.JsType

@JsType
class KotlinObject {
  fun hello(): String = "Kotlin"
}

I set it up via these Bazel rules:

def _cross_kt_lib(name, srcs, deps = []):
        kt_jvm_library(
            name = "%s-kt" % name,
            srcs = srcs,
            deps = [
                "@com_google_j2cl//:jsinterop-annotations",
            ] + deps,
        )

        j2cl_import(
            name = "%s-j2cl" % name,
            jar = ":%s-kt" % name,
        )

cross_lib(
    name = "KotlinObject",
    srcs = ["KotlinObject.kt"],
    deps = [":JavaObject"],
)

Then, I invoke it via JS:

goog.module('main');

const KotlinObject = goog.require('javatests.language.KotlinObject');

/**
 * Main function, dispatched on page load.
 */
function main() {
  console.log(`Hello from ${KotlinObject.hello()}!`);
}

// Bind `main` to page load.
window.addEventListener('load', main);

Which I build via:

closure_js_library(
    name = "main",
    srcs = ["main.js"],
    deps = [
        ":KotlinObject-j2cl",
    ],
)

j2cl_application(
    name = "example",
    entry_points = ["main"],
    deps = [":main"],
)

Outputs

The problem is, the outputs end up empty:

KotlinObject-j2cl.i.js:

(empty)

KotlinObject-j2cl.js.zip: Does not exist

Also, the build fails with the following error:

ERROR: /.../javatests/language/BUILD.bazel:29:1: Checking 1 JS files in //javatests/language:main failed (Exit 1)
javatests/language/main.js:5: ERROR - Namespace not provided by any srcs or direct deps of //javatests/language:main.
const KotlinObject = goog.require('javatests.language.KotlinObject');
                                  ^
  ProTip: "CR_NOT_PROVIDED" or "strictDependencies" can be added to the `suppress` attribute of:
  //javatests/language:main

1 error(s), 0 warning(s)

Target //javatests/language:example failed to build

Bazel version

Bazelisk version: development
Build label: 2.0.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 19 12:33:30 2019 (1576758810)
Build timestamp: 1576758810
Build timestamp as int: 1576758810

Expected behavior Since the Kotlin source would have been converted to Java bytecode by the time it is loaded by j2cl_import, I would imagine it would work to produce J2CL-driven JS, with Kotlin as the original source?

[sgammon]

It is worth noting that, of course, an identical sample works for Java.

[sgammon]

Alternative 1: Using the generated JAR via j2cl_library

j2cl_library(
        name = "%s-j2cl" % name,
        srcs = ["%s-kt.jar" % name],
        deps = [
            "@com_google_j2cl//:jsinterop-annotations-j2cl",
        ] + ["%s-j2cl" % i for i in deps],
)

This, curious enough, builds fine:

INFO: Analyzed target //javatests/language:KotlinObject-j2cl (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //javatests/language:KotlinObject-j2cl up-to-date:
  dist/bin/javatests/language/KotlinObject-j2cl.js.zip
  dist/bin/javatests/language/libKotlinObject-j2cl.jar
INFO: Elapsed time: 0.198s, Critical Path: 0.02s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

Contents of the produced outputs, which are there but all empty:

KotlinObject-j2cl.i.js:

(empty)

KotlinObject-j2cl.js.zip:

➜  kotlin unzip KotlinObject-j2cl.js.zip 
Archive:  KotlinObject-j2cl.js.zip
warning [KotlinObject-j2cl.js.zip]:  zipfile is empty

Alternative 2: Importing the generated JAR via j2cl_import

j2cl_import(
        name = "%s-j2cl" % name,
        jar = "%s-kt.jar" % name,
 )

(I.e. depending on the generated JAR from Kotlin). This produces the error:

ERROR: /.../javatests/language/BUILD.bazel:23:1: in jar attribute of j2cl_java_import rule //javatests/language:KotlinObject-j2cl: generated file '//javatests/language:KotlinObject-kt.jar' is misplaced here (expected no files). Since this rule was created by the macro '_cross_java_lib', the error might have been caused by the macro implementation
ERROR: Analysis of target '//javatests/language:KotlinObject-j2cl' failed; build aborted: Analysis of target '//javatests/language:KotlinObject-j2cl' failed; build aborted
INFO: Elapsed time: 0.101s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

No .js.zip artifact is produced at all, and the .i.js artifact is empty.

[niloc132]

I'm far from a bazel expert, but it looks like you are trying to use j2cl_import to turn the kotlin-originating bytecode into JS. This isn't what that task is supposed to be for, according to its docs: it is only intended to bring the bytecode of annotations over the fence into j2cl, which works since annotations don't result in .js output, and are only needed to that the rest of the source, bytecode makes sense.

J2cl itself can only transform .java sources, not jvm bytecode in .class files - it will take bytecode as an input (so that it can compile your .java sources against that bytecode, same as javac itself), but it will not generate .js from plain bytecode.

[rluble]

J2CL compiles Java into JavaScript from source (not bytecode). J2CL does not compile kotlin code, and @niloc132 is correct in pointing out that j2cl_import is for annotation which do not need to be transpiled to JavaScript.

[sgammon]

@rluble / @niloc132 thank you that does clarify things

being that Kotlin operates on the bytecode level, this would make J2CL fully incompatible for the foreseeable future with any meta languages that also operate that way. would you guys happen to know if Lombok or other solutions all work at the bytecode level, or is there a way to use some terser syntax with J2CL using any of those?

[sgammon]

also thank you for responding so quickly to this.

[rluble]

being that Kotlin operates on the bytecode level, this would make J2CL fully incompatible for the foreseeable future with any meta languages that also operate that way.

Kotlin is a different language and is compiled by the Kotlin compiler targeting the Java Virtual Machine. The Java Virtual Machine (JVM) has its own language (a low level stack based assembly like language) which is known as bytecode.

J2CL compiles Java source to JavaScript (not bytecode). Compiling bytecode to JavaScript is very different than compiling Java source and a different compiler needs to be written. There is a different set of challenges, as there are different tradeoffs in each approach. When we started J2CL we evaluated the different approaches and settled on going from source code.

Kotlin could be also compiled to JavaScript directly in the same way that Java is through J2CL.

would you guys happen to know if Lombok or other solutions all work at the bytecode level, or is there a way to use some terser syntax with J2CL using any of those?

So in short to either compile Kotlin or bytecode to JavaScript requires a different compiler. Such a compiler could reuse parts of J2CL (in particular what is called the backend which does the transformations from Java like structures and semantics to the corresponding JavaScript ones) but would need a completely different front end to be written.

Lombok does actually operate in bytecode targeting the JVM, and operates by changing the underlying bytecode driven by annotations. So it is not part of the Java to JavaScript pipeline and not usable for the purpose you want.

[tbroyer]

Lombok doesn't operate on bytecode, but during compilation to bytecode. You can however use delombok to generate Java source code that could probably be compiled to JS by J2CL (it used to be the way to use Lombok with GWT)

[sgammon]

@rluble / @tbroyer thank you for sharing your expertise on this, that all makes sense. i'll experiement around

of course yes Kotlin itself can natively be compiled to JS, but that leaves the entire Closure framework on the table which we're hoping to leverage.

thank you again everyone this was really helpful

[niloc132]

I wasn't aware of the delombok method for GWT 2, but the lombok site used to (and may still) list that since GWT 2 uses JDT, you can set up lombok as a javagent and let it change the JDT AST as it is built, manipulated, so that GWT actually sees the transformed AST. As long as J2CL retains the JDT frontend and as long as you use it instead of the JAVAC implementation, this could potentially work here too, albeit with a custom j2cl_library() target to include that javagent?

Edit: https://projectlombok.org/setup/gwt

[rluble]

Lombok doesn't operate on bytecode, but during compilation to bytecode.

I meant that it does not operate as a "normal" apt but does alter the bytecode that javac produces by ingeniously intercepting the way the bytecode is output, so in a way as opposed to most APTs that generate source code, lombok transforms the bytecode.

You can however use delombok to generate Java source code.

This is interesting, lombok as a source transformation....

As long as J2CL retains the JDT frontend and as long as you use it instead of the JAVAC implementation

The plan is to switch to javac and stop supporting JDT since there is no need to have duplicated frontends. That avoids the few incompatibilities between both and makes it easier to evolve as javac is much better maintained and the reference implementation for java.

That said delombok seems to operate as a source transformation and seems to me that it could be used regardless of the frontend.