Added Java support

[peterwilli]

Hi Microcule devs,

I've began adding Java to Microcule use your awesome tool on the JVM :+1: . It's probably not ready yet, as streaming isn't tested and tests are yet to be written (in microcule-examples)

I just wanted to post this pull request in advance, so you can evaluate and give feedback on my work. So far, it's been a great experience to work in your code. Have a nice day!

Details of what it's doing:

• run ./bin/microcule ./helloworld.java from source (create the said java file)
• it detects files with .java extension
• compiles down to .class in a temporary file
• sends the class path as compiled source to micro-java (JVM can't run source directly, so a path was needed)
• runs the code and shows the output in the reply.

TODO:

• Add support for (JSON?) parameters
• Add tests
• Support for other JVM languages like Kotlin

[Marak]

Very awesome!

I will review these changes over the next few days and make comments.

Looking forward to getting this fully working!

[peterwilli]

@Marak Thanks! We just figured out that it might not be a good fit for us since we need to include maven libraries with our 'microservice', but I'll keep in touch on this :) Wouldn't be much work anymore to finish this up anyway :)

[Marak]

The general approach would probably be to include maven in the worker environment where microcule was being run.

I don't do much Java development so I'm not entirely sure.

Would maven act the same as npm or pip or gem? We do have a way of supporting package managers.

[peterwilli]

@Marak Maven is indeed a package manager, but works by bundling the dependencies into the resulting JAR file (jar is a java archive like working like .exe, in case you didn't know). The only way I see that working either by supporting maven during the compilation step, or if we support using JAR files directly.

Also, you mentioned that Microcule has support for package managers like NPM and PIP. Can you tell me more about that? I currently use Microcule for a node app which basically has microcule installed locally and then being launched using ./node_modules/microcule/bin/microcule -w true. This was the only way I found out to use local node packages within microcule.

How is this done generally? i.e with pip?

[Marak]

There's no trick with using npm or pip. You just need to run microcule in the same environment / path where the package manager is also available.

Here is an example of using gm module https://github.com/Stackvana/microcule-examples/blob/master/javascript-image-resize/package.json

It's possible that microcule might have some pathing issues, let me know. Is probably a minor fix.

I'm continuing to review your PR. Thank you for your contribution. Will update soon.

[Marak]

@peterwilli -

This is a good commit. I will use this to help get JAVA support going. Thank you.

The problem ( which you have identified ) is that this kind of implementation is naive. I know, because I did a similar implementation recently for OCaml. It's a good first step, but JAVA ( and other languages ) need to be compiled.

We've got basic support for a compilation step in microcule, but it's not very well developed yet. We also need to be careful in how microcule accesses the disk, because in production we do not have write access to the disk ( this means we have to be mindful of how and when we are writing the compiled source files to disk ). We need to iterate on this design and find a good pattern for storing compiled source on disk or to DB provider.

You mentioned supporting JAR files directly? I think that might be a good first step. How would that work? Would the JAR file be listening on STDIN / STDOUT?

[peterwilli]

@Marak yeah JAR could do that, since it's essentially a executable that can be run using java -jar <jar> similar to running a single .class file.

I wanted to add direct support for JAR or .class files instead of compiling them when I began this quest, so Kotlin and other popular languages on the JVM would work as well. When I started, the branch was called 'jvm-support'. But I couldn't find a nice way to run JAR files directly, since JAR is a binary file and (as far as I can tell) Microcule reads the full source of the files you tell it to run, and then passes it on through a command line argument.

[Marak]

@peterwilli -

Probably the best idea here will be to provide an API that allows microcule to mount an arbitrary binary to a url route. This way, we could have a compiled binary of any language which is sitting and waiting to be spawned with STDIN / STDOUT API.

Would it make sense to standardize on Java_servlet API for Java? The spawned process needs to able to send HTTP response methods over the STDERR channel as JSON. In order for Java to write HTTP headers, we'd have to create the producer API in Java for the existing consumer. Then any Java binaries would most likely want to include a microcule SDK.

We could also not use servlet and simply standardize on STDIN / STDOUT. We'd still have to include some sort of SDK for the Java app though, to give a sugar syntax for sending HTTP response methods over STDERR as JSON.

Let me know if you have any ideas.

[peterwilli]

@Marak I think that is one of the best options we got here. A separate SDK also makes it very easy to support other languages on the JVM, since they can read any other sdk or library.

You read my mind re: servlet-like API :+1: The use of a java servlet-like API to allow sending/receiving json is a great idea. I was actually looking for something like this to implement, similar to how we currently use Node's HTTPRequest pattern (hook.res.json({bla: 1})), but couldn't remember the exact name :)

Probably the best idea here will be to provide an API that allows microcule to mount an arbitrary binary to a url route

Would that be a possibility? When I look at the example here (https://github.com/Stackvana/microcule#examples) I can see that source code is directly being embedded here.

[Marak]

@peterwilli

I think yes. I am going to create a new branch with experimental support for mounting and spawning arbitrary binaries. The expectation would be STDIN / STDOUT API contract with argv containing incoming JSON of request params and ENV variables.

I'm not really great at Java. What would be a good way to implement the Servlet API? Is there existing tooling for mapping STDIN / STDOUT to servlet that we can re-use?

[peterwilli]

@Marak Great! Glad I can help. Uhm, I have to look into it a bit more, will get back to it in this comment section when I figured it out. I've been doing a great deal of java in my life, but not with servlet directly. (I know it's existence but that's how far it goes :))

[Marak]

@peterwilli -

I'm half way into adding the functionality for spawning arbitrary binaries with arbitrary argv.

It occurs to me that this functionality already exists if developers use our bash support? What is stopping us from spawning a JAR file from a bash script?

I guess it would be cleaner to not spawn a bash sub-shell ( which in turn would spawn an additional child process ), but I'm not certain of the actual implications. Since it seems like it will be easy and non-obtrusive to add support for arbitrary binaries, I will attempt to add it.

[Marak]

We can now run commands like this in the any-bin branch:

microcule echo "hello world"
microcule cat ReadMe.md
microcule tail -f ReadMe.md

This should give basic support for mapping any JAR file ( or any other binary on the system ). Will merge into master soon.

[peterwilli]

@Marak that's great! But wouldn't treating every jar file like any other binary stop us from implementing a servlet-like API? I still think this is a great start, definitively what I needed.

[Marak]

@peterwilli -

This is a good start for general purpose binaries.

I'm still not 100% certain on how the Servlet API would work with microcule. I understand how we can do WSGI for python and streaming middleware functions for Node.js, but I'm not sure how the Servlet API would work / be implemented.

Any code examples or additional information on how to setup Java Servlet API ( or something similar ) would help. I found http://sparkjava.com/ which has a similar API to node, but I'm not sure if this would help at all. Maybe it could be a good source of inspiration.

[Marak]

FYI, we now have initial support for compiled Java in the compiled-langs branch.

https://github.com/stackvana/microcule/tree/compiled-langs

[peterwilli]

@Marak wow didn't expect that you would be all over java and other compiled languages so quickly after this pull request - great job!

[Marak]

I believe basic Java support is now completed in the compiled-langs branch.

I'll write up a full description as part of the release, but the gist of it is that Java is part of the new compile plugin and currently works on STDIN / STDOUT / STDERR. HTTP params will be passed in over argv ( for now ).

This will leave room in the future to implement a better request / response contract to STDIN / STDOUT.

Should be included in the upcoming release.

[Marak]

@peterwilli -

We now have basic support for Java services released with microcule v5.x.x and higher.

Please try out the latest version with:

microcule ./examples/services/hello-world/hello.java

Or maybe try to find / add Java examples to: https://github.com/stackvana/microcule-examples

Our current implementation is not perfect, but I think it's a very good start! Thank you for your contributions and helping push this forward. Closing for now.

[peterwilli]

@Marak your welcome. Thank you too, now Microcule can be used for all our microservices :+1: