Support for openWhisk

[jeromesimeon]

Some of our targets (notably JavaScript and Cloudant) could be deployed on openWhisk (https://openwhisk.incubator.apache.org) instead of being run through 'query runners'. This would provide for an easy way to run queries compiled by Q*cert.

Deploying the Q*cert compiler itself on openWhisk could also be of interest to openWhisk user who wants to use query languages to create their own actions more easily.

[joshuaauerbachwatson]

To deploy the Q*Cert compiler on openWhisk we decided at a meeting today to use a sequence of two actions. The second action is simply QCertJS.js running as a Node action. It accepts a JSON input defined by the TypeScript type QcertCompilerConfig. It requires a preliminary action only when Java processing is needed.

The first action will be written in Java and be an augmentation of the existing Java service, incorporating some of the decision logic presently in QCertPreCompiler.ts. Basically, it figures out if any Java preprocessing is needed and, if so, it does it. This action also accepts (and returns) a QcertCompilerConfig.

The network-connected demo would simply use this new support in lieu of what it does now. But, we would want to preserve the ability to run locally either (1) when there is no reachable Java service at all, in which case we would only support languages that don't need pre-processing in Java or (2) when there is a reachable Java service running on 'localhost'. To make this maximally compositional, we would change the long-running "Java service" to have exactly the calling conventions and semantics of the remote whisk action (we could do this by running a whisk instance locally but that seems like overkill). That is, it analyzes the requests, does Java pre-processing if needed, then uses nashorn to run QCertJS.js so that what is returned is equivalent to what the whisk sequence would have returned.

[joshuaauerbachwatson]

I removed a few sentences from the previous comment because the issue of how to support non-network-attached demos is a little trickier than I thought.

One reason we run on 'localhost' for demos is to avoid network issues but another reason is to incorporate the jrules support which (I still assume) wouldn't be on any public deployment, whether in whisk or not. For this use case, there still needs to be a "search order" with 'localhost' at the start and public URLs after that. The best design would make the invocation of 'localhost' mimic the invocation of the remote whisk action sequence.

Another goal here is to support users who didn't install Java. If such users have network access, they are fine because all the Java stuff is remote. But, if such a user is not on the network, our current ability to run something on 'localhost' won't work since we use Java to do that; we could substitute something else (e.g. Node) but it would beg the issue since that is a pre-requisite just as difficult as installing Java. So, the final fallback has to be that the demo itself loads QcertJS into the browser and executes it.

I think it still makes sense for a server running on 'localhost', if present at all, to use Java and, in fact, to run QcertJS using nashorn (this gives uniformity with the remote case and makes the search order traversal simple). But, if the demo exhausts the entire search order without finding a usable service, it has to invoke QcertJS itself.

[joshuaauerbachwatson]

There is now a candidate whisk action (untested) which is designed to be the first of two in a sequence, the second being QcertJS running under Node. I have still to develop a testing strategy and to incorporate the new action. The old action (design to be used by itself as a "Java service" in the older sense) is still operative, as are the Java service running in AWS and ability to start a Java service on localhost.

[joshuaauerbachwatson]

There is now a way to test the two-action whisk sequence. It has been (very lightly) tested and it appears to work! The demo is still using the old setup, so we now have the union of all the different ways we have done things. The artifact qcertJavaWhiskAction.jar actually supports both a simple Java service running as a single whisk action and the two-action sequence (deployed differently, with different mains).

Once I've gotten the new support runnable on 'localhost' and also modified the demo, the plan would be to retire some of the different ways we have of running things:

• qcert cmdline will still do fork and exec as before (we could change it but there isn't a strong motivation and best to leave it working as it was for now)
• the ability to run a long-running Java service, either on localhost or in AWS will be retired
• the ability to run just the Java service with its old linkage as a single whisk action will also be retired; it was just a step along the way
• there will be a new way to run the two-action sequence (simulating a tiny piece of the whisk runtime) on 'localhost' (in theory, on other URLs, but I don't see much motivation)
• the two-action sequence running under whisk on bluemix will be the preferred way of running the demo

One loose end concerns how the actions are deployed. Currently, the Java action in the two-step sequence is deployed using wsk in the javaService Makefile. Both the QcertJS action and the sequence itself are built using the deployment tool out of the openwhisk/compiler/Makefile. For now, the latter makefile invokes the former to ensure an up-to-date result. Ideally, all actions should be deployed the same way. I am unclear on the degree of support for Java actions in the deployment tool.

[joshuaauerbachwatson]

I have pushed support for running a simulation of the whisk two-action sequence on localhost. It appears to work. Caveats:

• the demo is still not updated; consequently (1) there is still an old JavaService running in AWS and (2) there are actually two ways to run the "JavaService" on localhost; the documented way gives the old linkage and supports the demo. The new way (read the comment at the top of org.qcert.javasvc.Main in 'javaService/src') simulates the remote whisk deployment but can include ODM support. All of this will get simplified once the demo is changed and debugged running in the new say.
• there is still the slight mess in the way the two different whisk actions are built and deployed. We could decide this is ok or we could fix it; worst case, fixing it might require improvements to the deployment tool.
• I have observed anomalies while testing whereby something that works fine when directed to the remote whisk service in bluemix will fail with a StackOverflowException (in nashorn executing qcertJS.js). I am not sure what test case exhibited this and I haven't reproduced it since building everything the Kosher way and retesting it so it could turn out to be a red herring

[joshuaauerbachwatson]

The demo (demo.html and its scripts) has been updated to use the new stuff. Remaining issues:

• The old way of running JavaService (-server vs -whiskserver) persists and won't be removed until we're satisfied that all copies and versions of the demo are switched over.
• I'm confused about how many versions of the demo exist and so maybe others need to check their status
• When I run the new service configuration on localhost I get StackOverflowError in nashorn for some queries. For example (for me) test01.arl works but test02.arl gets the error. I can't tell whether I would get the same error running under whisk because I don't deploy ODM jars under whisk. I previously found with some large SQL queries that things that worked remotely under whisk failed on localhost under nashorn.
• The JavaService feature to convert CSV files no longer supports JavaScript execution because the CSV conversion happens in the whisk action and no longer in the browser. However, while we do evaluation of other languages in the whisk action we are still evaluating JavaScript only in the browser. I could fix this in three possible ways (or neither, and declare it a feature):
  1. Do JavaScript evaluation in one or the other whisk actions. If I do it in the first action I would have to use nashorn to do it. To do it in the second action will require changes in the qcertJS.js build because the evaluation must coded in JavaScript source, not OCaml source.
  2. Allow CSV conversion as a special case of a remote service. This would be pretty trivial to do in the first action (it is already doing it, but assumes this in the context of a compilation). A small change to qcertJS.js (which could be written in OCaml) simply passes through the result.
  3. Stop using the JavaService to do CSV conversion and do it entirely in the browser as before, except using JavaScript code. It is the way it is now since it allows me to use a respected Apache library to do CSV (it supposedly adheres to an RFC on the topic); I couldn't find a similarly vetted library in JavaScript. It wouldn't be hard to write something that does the conversion and is "usually correct."