Spin the Java Starlark interpreter into a stand-alone library

[johnynek]

there are many use case for the skylark and the build language.

It would be great to have the BUILD language for many configuration problems on the JVM. It would be cool for this code to be available as a maven jar, or if that is not possible as a bazel repository we can cleanly depend on.

I imagine the user plugs in the standard library of functions that can be called. So, bazel proper would give access to all the built-ins of skylark's current implementation.

Basically, at this point skylark looks like a total, dynamically typed programming language (one could even imagine a project to add type inference to skylark, which I think may be possible pretty easily since it lacks recursion).

/cc @jart @laurentlb

[ittaiz]

@mikelalcon is this something that is interesting for copybara as well?

[jart]

Congratulations @laurentlb it looks like you're going to be the next Guido van Rossum.

[mikelalcon]

We already use Skylark as a standalone library :). Well not really standalone, but almost :) https://github.com/google/copybara/blob/master/WORKSPACE#L90

[laurentlb]

We're working on it. :)

We're doing some experimentation (e.g. with copybara), but it's not ready yet for prime-time.

There's a standalone binary here if you want to play with it: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/skylark/Skylark.java Please don't depend on it or on the libraries, we're likely to change the interface at some point.

Thank you for the request/feedback, this is very useful for us to prioritize the work!

[laszlocsomor]

@laurentlb : Could you prioritize this issue please?

[ttsugriy]

I also like the idea of making skylark a standalone library and I'm very much interested in using it. I used a hacky way to get a skylark standalone jar:

• apply patch https://gist.github.com/ttsugriy/cf1d8eccd7c45e0b8298cc205796bf75 to turn skylark-lang into a java_binary
• bazel build //src/main/java/com/google/devtools/build/lib:skylark-lang_deploy.jar
• use jarjar tool to shade or third-party jars like guava

[laurentlb]

For the record, there is now a separate implementation of the interpreter in Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

Edit: i.e. we still want to provide a standalone Skylark library, to be used by Bazel and other projects.

[ittaiz]

So you're deprecating the java interpreter? Copybara is jvm based and we were also interested to depend on it inside the jvm On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun notifications@github.com wrote:

For the record, there is a separate implementation of the interpreter in Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b .

[dslomov]

We are not deprecating the Java implementation. Skylark-in-Go is a parallel effort implementing the same language.

On Oct 4, 2017 01:08, "Ittai Zeidman" notifications@github.com wrote:

So you're deprecating the java interpreter? Copybara is jvm based and we were also interested to depend on it inside the jvm On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun notifications@github.com wrote:

For the record, there is a separate implementation of the interpreter in Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334080693, or mute the thread https://github.com/notifications/unsubscribe-auth/AExrX7jxVjywy6bNxjGv4xmevIwmALVyks5soz0XgaJpZM4LmE4b .

[ittaiz]

Great. Is splitting the java interpreter out to a standalone library still a goal? On Wed, 4 Oct 2017 at 12:50 Dmitry Lomov notifications@github.com wrote:

We are not deprecating the Java implementation. Skylark-in-Go is a parallel effort implementing the same language.

On Oct 4, 2017 01:08, "Ittai Zeidman" notifications@github.com wrote:

So you're deprecating the java interpreter? Copybara is jvm based and we were also interested to depend on it inside the jvm On Wed, 4 Oct 2017 at 10:59 Laurent Le Brun notifications@github.com wrote:

For the record, there is a separate implementation of the interpreter in Go: https://github.com/google/skylark/

We still plan to remove Bazel dependencies from the Java interpreter.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334078465 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ABUIFzxpOz4QnmyDALCkrgm2C5svetNgks5sozrggaJpZM4LmE4b .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334080693 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AExrX7jxVjywy6bNxjGv4xmevIwmALVyks5soz0XgaJpZM4LmE4b

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334105668, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIF5S0cGjHeWmYhd-RPIzrdQkkZ_u8ks5so1UBgaJpZM4LmE4b .

[laurentlb]

Yes, that's what I meant (sorry if I was unclear). We're still cleaning and improving the Java code.

[johnynek]

I wonder if at some point the speed of interpreting skylark is an issue. I wonder if it ever makes sense to use a JIT like V8 or luajit to improve skylark speed.

On Wed, Oct 4, 2017 at 05:33 Laurent Le Brun notifications@github.com wrote:

Yes, that's what I meant (sorry if I was unclear). We're still cleaning and improving the Java code.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-334195544, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEJdslC2yAL050syBATijtES29C11Ajks5so6UsgaJpZM4LmE4b .

-- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[ttsugriy]

@johnynek, JVM already has a JIT, so hot Skylark code should be optimized as long as it's executed enough times, which depends on the complexity of your project. In practice, most likely most of the Java functions will be compiled, leaving only Skylark functions not compiled, since it's unlikely that they will reach JVM's second tier threshold.

[johnynek]

@ttsugriy I'm not sure that's really true. The interpreter code is jitted, but the skylark code itself won't be very effectively optimized by the JIT. If it were, then interpreter patterns seen in FP (see free monad, etc...) would not suffer a slow-down on the jvm.

I hear you that there should be a small amount of code in skylark and that a high fraction of build time should be in actions that are not skylark related. I have seen posts where people had O(n^2) work in skylark and things took a long time. Certainly O(n^2) work is no good, but the geek in me wonders if any off-the-shelf JITs (pypy, v8) could be leveraged to make skylark actually fast.

[laurentlb]

There are many ways to improve the performance. If we want JIT, we can compile the Skylark code to JVM bytecode and run it. Using LLVM or V8 doesn't make sense for Bazel, since the data structure should be shared with the rest of Bazel.

To be honest, we haven't optimized the interpreter yet. There are many things we could do (do faster variable lookups, optimize method resolution, etc.), but that's a separate discussion. Feel free to file a separate issue for that.

I keep this issue open for providing a nice, self-contained library. This will be useful for Buck, among others.

[ittaiz]

Just as another data point- we built a small WORKSPACE parser and BUILD parser we'd love to replace with the above On Tue, 10 Oct 2017 at 14:29 Laurent Le Brun notifications@github.com wrote:

There are many ways to improve the performance. If we want JIT, we can compile the Skylark code to JVM bytecode and run it. Using LLVM or V8 doesn't make sense for Bazel, since the data structure should be shared with the rest of Bazel.

To be honest, we haven't optimized the interpreter yet. There are many things we could do (do faster variable lookups, optimize method resolution, etc.), but that's a separate discussion. Feel free to file a separate issue for that.

I keep this issue open for providing a nice, self-contained library. This will be useful for Buck, among others.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-335443274, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIF52ZN74tt1-JN_Dj1NlYJSnAzK1nks5sq1UqgaJpZM4LmE4b .

[ittaiz]

cc @or-shachar (as our maintainer of a very poor WORKSPACE + BUILD file parsers)

[dtretyakov]

Any progress on providing Skylark Java language library? It would be very useful for parsing BUILD/WORKSPACE files.

[ittaiz]

+1 On Wed, 19 Sep 2018 at 22:13 Dmitry Tretyakov notifications@github.com wrote:

Any progress on providing Skylark Java language library? It would be very useful for parsing BUILD/WORKSPACE files.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/2367#issuecomment-422924499, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUIF9KhhLcI8UsghqTQeC7yrJKgPElOks5ucpc8gaJpZM4LmE4b .

[laurentlb]

@alandonovan has recently made a lot of cleanup and some progress in this area. Do you have any update to share, Alan? Maybe you could associate your future commits with this issue.

[alandonovan]

I've been working towards splitting the interpreter out for over 6 months, primarily for the health of the Blaze and Copybara code bases, not to support other clients. There remains much to do in terms of breaking dependencies and cleaning up API mistakes---another quarter I suspect.

Once that's done, we can discuss the bigger question of whether to publish the package, move it into a separate repository, and permanently freeze its API. I don't think any decision to do that has yet been made.

[brandjon]

Update:

The Go Starlark implementation (google/starlark-go) exists and is relatively stable. Use this if you're looking for a reliable Starlark implementation, are ok with writing your application in Go, and don't need strict bug-for-bug compatibility with Bazel's Starlark interpreter (which is unlikely, since at that point you'd probably want to run Bazel itself).

The Java interpreter is still hosted inside Bazel's source repository (bazelbuild/bazel). However, it has been moved to its own path (src/main/java/net/starlark) outside of Bazel's core logic, and does not depend on "Bazel-isms" anymore. That said, we are not encouraging new uses of this interpreter at this time. Its API is still not stable, and we will make breaking changes as needed (coordinating them with the only supported clients, Bazel and Copybara). Alan has done a tremendous job cleaning up the interpreter's API, but finalizing that work at this point is not a priority, thought it's conceivable it could still happen at some point in the future.

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen (or ping me to reopen) if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

[ittaiz]

@brandjon @alandonovan almost 3 years later- any chance that the java interpreter API has stabilized? We're working on an internal tool that aims to give you feedback when you change a semantic macro in a backward incompatible way. Part of the need is to parse the BUILD.bazel file and that macro into java.