search

openhab / openhab-core

Core framework of openHAB
https://www.openhab.org/
Eclipse Public License 2.0
906 stars 420 forks source link

GraalVM JavaScript replaces Nashorn, breaking Blockly and existing Nashorn rules #2433

Closed rkoshak closed 2 years ago

rkoshak commented 3 years ago

Is there a way to have Nashorn and the GraalVM JavaScript add-on live side by side? Now it appears to completely replace Nashorn which breaks things.

I've had to deal with two problems in two days caused by the fact that GraalVM JavaScript is installed and then the users are trying to use a Nashorn rule or to use Blockly.

If it is not possible, I'll move this issue over to the webuis repo to have Blockly removed as an option to build rules when the add-on is installed. However, that's not a very satisfying approach. Installing an add-on shouldn't replace or break stuff that is built into openHAB be default in my opinion.

wborn commented 2 years ago

Is there a way to have Nashorn and the GraalVM JavaScript add-on live side by side?

I don't think so currently. But it would be nice if the core would support multiple versions of script engine languages. That way you could slowly migrate your rules from one major version to another. But it would also mean that script engine versions need to be stored for rules created in the UI and there needs to be a way to determine the script engine version for scripts in files.

Because Nashorn has been removed from Java 15, there won't be any ECMAScript 5.1 script engine available in newer Java versions (unless an Automation add-on is created that provides Nashorn (https://github.com/openhab/openhab-distro/issues/1316#issuecomment-922436661)).

Eventually the Blockly integration will also need to support generating code for newer ECMAScript versions. So it wouldn't hurt if the UI enables Blockly depending on the available ECMAScript engine version. It would be even nicer if the Blockly integration supports generating code for multiple ECMAScript versions. :slightly_smiling_face:

rkoshak commented 2 years ago

Given how different the GraalVM add-on implements the interactions between rules and openHAB itself compared to Rules DSL, Nashorn, and Jython, this might be a chance to step back and think about how we want it to work. To a large extent, it's going to be way more than just migration from ECMAScript 5.1 to ECMAScript whatever GraalVM supports. It's going to be reimplementation in how everyone needs to write rules.

In the traditional languages all sorts of things are imported and available to the rules coder by default:

There's more but those are the most used. In GraalVM JavaScript a design decision was made to import nothing. And it's not clear in the current set of documentation how to access that stuff. But to access that stuff means each and every rule/Script Action/Script Condition is going to require a whole bunch of import statements/load statements just to do simple stuff like logging or sending a command to an Item.

It might be a good idea to bring this up as part of any conversation when moving to Java 15+. GraalVM isn't just an updated JavaScript support.

ghys commented 2 years ago

I downloaded the newest distribution, installed the GraalVM JS add-on, and tried a new Blockly script and indeed it doesn't work but this is the actual error:

20:39:21.003 [ERROR] [ript.internal.ScriptEngineManagerImpl] - Error while creating ScriptEngine
org.graalvm.polyglot.PolyglotException: Error: Invalid CommonJS root folder: I:\dev\OPC442~1.0-S\conf\automation\lib\javascript\personal
        at org.graalvm.polyglot.Context.eval(Context.java:345) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.evalInternal(GraalJSScriptEngine.java:319) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSBindings.initGlobal(GraalJSBindings.java:94) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSBindings.initContext(GraalJSBindings.java:90) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSBindings.requireContext(GraalJSBindings.java:84) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSBindings.put(GraalJSBindings.java:127) ~[?:?]
        at javax.script.SimpleScriptContext.setAttribute(SimpleScriptContext.java:246) ~[java.scripting:?]
        at org.openhab.core.automation.module.script.internal.ScriptEngineManagerImpl.addAttributeToScriptContext(ScriptEngineManagerImpl.java:254) ~[?:?]
        at org.openhab.core.automation.module.script.internal.ScriptEngineManagerImpl.createScriptEngine(ScriptEngineManagerImpl.java:144) ~[?:?]
        at org.openhab.core.automation.module.script.internal.handler.AbstractScriptModuleHandler.createScriptEngine(AbstractScriptModuleHandler.java:92) ~[?:?]
        at org.openhab.core.automation.module.script.internal.handler.AbstractScriptModuleHandler.getScriptEngine(AbstractScriptModuleHandler.java:88) ~[?:?]
        at org.openhab.core.automation.module.script.internal.handler.ScriptActionHandler.execute(ScriptActionHandler.java:59) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.executeActions(RuleEngineImpl.java:1183) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.runNow(RuleEngineImpl.java:1035) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.runNow(RuleEngineImpl.java:1051) ~[?:?]
        at org.openhab.core.automation.rest.internal.RuleResource.runNow(RuleResource.java:327) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[bundleFile:3.4.3]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[bundleFile:3.4.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.3]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.3]        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[bundleFile:3.4.3]        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.3]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
        at java.lang.Thread.run(Thread.java:834) [?:?]
20:39:21.063 [WARN ] [re.automation.internal.RuleEngineImpl] - Fail to execute action: script
java.lang.NullPointerException: null
        at com.oracle.truffle.js.scriptengine.GraalJSBindings.put(GraalJSBindings.java:128) ~[?:?]
        at javax.script.SimpleScriptContext.setAttribute(SimpleScriptContext.java:246) ~[java.scripting:?]
        at org.openhab.core.automation.module.script.internal.handler.AbstractScriptModuleHandler.setExecutionContext(AbstractScriptModuleHandler.java:117) ~[?:?]
        at org.openhab.core.automation.module.script.internal.handler.ScriptActionHandler.lambda$0(ScriptActionHandler.java:60) ~[?:?]
        at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
        at org.openhab.core.automation.module.script.internal.handler.ScriptActionHandler.execute(ScriptActionHandler.java:59) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.executeActions(RuleEngineImpl.java:1183) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.runNow(RuleEngineImpl.java:1035) ~[?:?]
        at org.openhab.core.automation.internal.RuleEngineImpl.runNow(RuleEngineImpl.java:1051) ~[?:?]
        at org.openhab.core.automation.rest.internal.RuleResource.runNow(RuleResource.java:327) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[bundleFile:3.4.3]
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[bundleFile:3.4.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[bundleFile:3.4.3]
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[bundleFile:3.4.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[bundleFile:3.4.3]
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[bundleFile:3.4.3]
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[bundleFile:3.4.3]        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298) ~[bundleFile:3.4.3]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[bundleFile:3.4.3]        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[bundleFile:3.1.0]
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273) ~[bundleFile:3.4.3]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:294) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[bundleFile:9.4.43.v20210629]
        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:82) ~[bundleFile:?]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [bundleFile:9.4.43.v20210629]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [bundleFile:9.4.43.v20210629]
        at java.lang.Thread.run(Thread.java:834) [?:?]

So from my point of view, maybe we could focus on this error and try to fix it before bailing out and saying installing the GraalVM add-on breaks Blockly because of compatibility issues? In terms of the ECMAScript language, if Blockly generates code targeting a certain language version, subsequent versions should be able to interpret it still. It seems that the JSScripting add-on expects some directories and will fail if they are not present, this seems easy enough to fix?

rkoshak commented 2 years ago

It's significantly more involved than just an upgrade to the ECMAScript issue. In addition to using a different folder structure (which is the root of the error above), the GraalVM JavaScript add-on does not import anything from openHAB. In Nashorn we are used to having:

GraalVM JavaScript makes none of that available. You have to go and manually import all of those things manually if you want to use them in your script.

So even when that error gets fixed, Blockly would still need to be significantly changed to produce code that will work for the GraalVM JavaScript and that code will not be compatible with Nashorn.

For all intents and purposes Nashorn JavaScript and GraalVM JavaScript may as well be two completely different languages.

ghys commented 2 years ago

I see, thanks for the explanation. I would have thought these were provided by DefaultScopeProvider and available regardless of the script engine.

Still, it's not that far of a stretch to adapt Blockly code to make these imports when needed - it's the case already when you use the "log" block (https://github.com/openhab/openhab-webui/blob/d8a16db016b149d007ef31a320fcdc000b8c51dc/bundles/org.openhab.ui/web/src/assets/definitions/blockly/ohblocks.js#L124-L132).

My point is that with Nashorn deprecated, Jython unmaintained (and a EOL 2.7 version of Python), DSL Rules being legacy, and Groovy being kind of niche, GraalVM might be the only way viable option left (with JRuby?) as a embedded scripting engine going forward, so we should try to make it work.

rkoshak commented 2 years ago

The developer went out of his way to avoid polluting the namespaces so rules could be pure JavaScript. There are legitimate reasons for doing so but it's not clear to me how the DefaultScopeProvider is used but it's not in the same way as it happens in Nashorn. In Nashorn the variables are just there thanks to the provider but in GraalVM JS rules they are not.

I don't disagree that we should try to make it work with something that will have support in the long run. But in the mean time GraalVM JavaScript will continue to break anything that is written for Nashorn which will trip up any user who wants to use Blockly or existing Nashorn code. If it's going to be months or years before Blockly can be

Given that GraalVM JavaScript has almost no documentation yet, pretty much only those who have the skills and willingness to study @jpg0's library are able to really build anything with it which makes moving Blockly over to support it somewhat of a challenge. I also don't know if anyone is trying to build rules in the UI with it so there may be gotchas we are unaware of. I know the jRuby folks too are ignoring UI rules. I haven't tried to keep up with their documentation though. They might be a bit more well documented but I don't know if they've even submitted jRuby to be considered as an official add-on yet.

I filed an issue on openhab-core to see if we could get Nashorn and GraalVM to exist side-by-side which would be the best way to bridge between what we have now to where we need to go but that doesn't seem to be possible. It's either or, not both.

I'm worried about months and months or even years between now and when Blockly can support something more future proof where users can't figure out why stuff is broken and not working because they installed an add-on.

I'd love to hear @jpg0's opinion on how to best to proceed.

jpg0 commented 2 years ago

Firstly, I'd like to apologize for the slow progress on the add-on and docs; I've recently moved house (well, moved the family from one side of the world to the other) so have had other priorities.

As for work on the GraalVM add-on, the plan is to update the folder structure (to make it compatible with the JS ecosystem), then write the docs.

@rkoshak is correct that the add-on diverges from the Nashorn by not importing things by default. Default things can be imported explicitly if required - they are at @runtime, as opposed to @runtime/<something>. It would be possible to add some boilerplate at the top of GraalVM scripts to import default things at which point they would probably be mostly compatible with Nashorn (but not 100%, as Nashorn is not 100% compatible with JS or GraalVM).

Anyway, regarding side-by-side installs - when you install the GraalJS add-on, in fact it doesn't replace the Nashorn engine (that engine is actually still used for JS transformations), however when openHAB attempts to find an engine for a script it looks it up by file extension, and only one engine can be returned.

To address your concern, it seems that the challenge is that neither Nashorn nor Graal can accept each others' scripts. Ideally we would have a backwards-compatible upgrade. The options that I can see are:

Of course there is also the option to never move to this 'everything explicit' model, but that forever shuts out the ability to rely on and integrate with existing 3rd party JS.

rkoshak commented 2 years ago

I don't think you need to apologize. We all have life commitments. It is certainly not the only place where openHAB docs are lacking.

or just deferring the issue?

Deferring the issue though could be a good thing if it keeps people from being confused and breaking their OH in the time between now and a final solution is implemented. It really is causing problems for all sorts of users, even OH users who have been using OH for years.

explicitly allow concurrent execution, either by allowing switching of the engine within openHAB core, or by choosing 2nd file extension to disambiguate engines

That only addresses text based rules, not UI rules. There's no file in that case. The engine is chosen, presumably, based on the "type" property of the action configuration. However, the version of the language doesn't seem to be a part of that. It's just application/javascript for a Nashorn Script Action/Condition.

This does seem like it might be an OK approach long term perhaps. I can imagine that there might be similar incompatible versions of languages that might gain support sometime in the future.

But that's all stuff that has to happen over on openhab-core I think.

ghys commented 2 years ago

So in the Blockly case, after reading https://github.com/openhab/openhab-addons/issues/11221 and creating the missing directory (something that should IMHO be needed but probably easy to fix), given that the code generated by Blockly only uses a couple objects for now (namely itemRegistry and events), we could either:

1) prepend this code or similar to all scripts to make these objects available as global variables when using GraalVM:

var runtime = (typeof(require) === "function") ? require("@runtime") : undefined;
if (runtime) { 
  var itemRegistry = runtime.itemRegistry, events = runtime.events; // etc.
}

2) prepend this code or similar to all scripts to make these objects available under runtime in both GraalVM and Nashorn (and use runtime.itemRegistry/runtime.events instead of itemRegistry/events in generated blocks:

var runtime = (typeof(require) === "function") ? require("@runtime") : {
  itemRegistry: itemRegistry,
  events: events,
  // etc.
};

The trick is that require is not defined in Nashorn so that way we can detect which engine is used to run the code.

Then I confirmed that (in the 1st alternative):

print(itemRegistry);
print(events);

resp. in the 2nd alternative:

print(runtime.itemRegistry);
print(runtime.events);

yields the same result in both Nashorn & GraalVM:

org.openhab.core.internal.items.ItemRegistryImpl@d5a9c50
org.openhab.core.automation.module.script.internal.defaultscope.ScriptBusEvent@1c23f2fd

With that simple addition to Blockly-generated code we could then still offer it when the JSScripting add-on is installed (the rest of the generated ECMAScript code being simple enough should hopefully work in both ECMAScript versions). Maybe ithis add-on could even become installed as part of the standard package in the future, like RRD4j persistence is today.

jpg0 commented 2 years ago

@ghys you may also want to merge everything from the runtime into the current context (with something like Object.apply(this, runtime) so as to ensure everything is available, but either way, your approach should work fine.

@rkoshak Another thing I was thinking about was the option to explicitly version, possibly via adding to the mime types in this case (I realised that if we do change the root directories for JS with GraalVM, then we can use this to indicate the version for files). I believe that the mime-types are simply strings, so it would be perfectly possible for the new implementation to register with something like application/javascript;version=2021. Additionally, the script engine manager can actually determine the language version for each engine (it does so when it first sees one), although it currently doesn't actually do anything with it. This would mean that it would technically be possible for language versioning to be added to the ScriptEngineManager, so that it could choose to register script engines with a versioned mime type in addition to their normal one, at which point consumers would be able to use versioned mime types to specify which script engine to use. Note however that this versioned string is not standard; whilst it's fine to use as a Content-Type parameter in HTTP, it's not usually part of the actual mime-type. (Note that it would be possible therefore to also add explicit language versions to the script engine manager directly, but that would require consumer changes to specify another parameter if they want an explicit version.)

rkoshak commented 2 years ago

@jpg0 I like the idea of being able to define the version number somehow. It seems to me that this is only the first time that we will run into this problem but indeed it will come up again and again. For example, what happens when ECMAScript 2022 or what ever comes out? Having the version that a given rule/script was written for can also help with your third proposal with error messages. "I see you used foo. In the new JavaScript XXXX you should now use bar."

@ghys, this is fantastic news. I would really really like to be able to move towards selecting a "default" language that isn't already deprecated and will disappear as soon as OH needs to move to a new JVM and it makes sense that the default would be the same thing backing Blockly. GraalVM JavaScript is a good fit for this, I've just not had the time to figure out how to use it yet. But I agree, having it installed as a default like rrd4j seems to be a good idea once we can ensure that doing so doesn't break Nashorn.

Doing so in the near term will give users time to migrate to GraalVM JavaScript (or maybe other languages too, Python would be greatly appreciated by a lot of users). Heck, I'll even write some tutorials for how to do it and contribute to some docs if I can. The trouble for me right now is time but I've got some off work time coming up so maybe I can look into it then.

I've been relatively unhappy with the thought of writing rule templates for the marketplace using languages that are basically deprecated and will need to be rewritten.

ghys commented 2 years ago

@ghys you may also want to merge everything from the runtime into the current context (with something like Object.apply(this, runtime) so as to ensure everything is available, but either way, your approach should work fine.

Yes, thanks, it makes sense! For now as I've mentioned earlier only 2 objects are needed from the openHAB API in Blockly so it's also reasonable to expose only those two but as we eventually add functionality it may become the go-to option. The good thing is that the actual JS code generated by Blockly can be changed on a whim because it's regenerated in full whenever the source blocks change.

creating the missing directory (something that should IMHO be needed but probably easy to fix)

I obviously meant not be needed, at least manually.

I would really really like to be able to move towards selecting a "default" language that isn't already deprecated and will disappear as soon as OH needs to move to a new JVM and it makes sense that the default would be the same thing backing Blockly

Completely agree with this and all of your points. I'm almost ready to add rule template support in the UI and eventually open the marketplace to the public, but I'm actually worried about the stability of the script languages that might be used in them - not only languages but also API differences to address OH objects since we have 2 JS engines and they are incompatibilities between them. So before we do that we might have to settle on a common API and stick to it - or even maybe mandate GraalVM for rule templates that have JS scripts in them so every script would have to require(@runtime) - fine by me. Again I don't see another viable option other than GraalVM for a future-proof way to go for embedded scripting in openHAB, since it's the only one that is actively maintained and offers a modern language.

digitaldan commented 2 years ago

Sorry if this is has been discussed before (I'm sure it has) , but is there a reason we are not injecting the runtime objects by default into the GraalVM instance? Or at least making it an option (turned on by default)? My assumption is that nearly every user (90%+ maybe?) switching to use GraalVM would need to add those to their scripts , seems like a lot of forced boiler plate code to make users copy and paste. I'm sure there's a reason not too, hence maybe making it configurable. I for one really struggled with this when i was trying it out a few months ago, it made what is a super awesome feature feel not production ready.

rkoshak commented 2 years ago

but is there a reason we are not injecting the runtime objects by default into the GraalVM instance?

There's a balancing act that has to be taken between injecting stuff and potentially polluting the namespace with the ability to support third party libraries which might use the same variables. My understanding is that @jpg0 choose the side that maintains the maximum ability to use any third party libraries (e.g. like libraries installed using npm, not just OH specific libraries) over making less work for rules developers who can rely on openHAB stuff just being there.

I personally am not sure where the best balance is. I do agree that it does seem like a lot fo boiler plate that has to be added, especially when working in the UI where one could have more than one Script Action and Script Condition per rule and each one would have to re-include the boilerplate. I'd almost suggest that with UI scripts that stuff be included by default no matter what through some behind the scenes magic or make it configurable as you suggest. People writing rules in the UI are unlikely to ever install a third party library via npm or any other way and the boiler plate is most glaring there.

But even with Nashorn we are stuck with a bunch of boiler plate already. I can't call logInfo, I have to first Java.type(var Log = Java.type("org.openhab.core.model.script.actions.Log"); to even get access to the logger and then I have to call Log.logInfo(). And how do I know that's the right class to type? Because I'm aware of the JavaDocs and know enough programming for how to use them. Most of our users don't though.

So even though it's a lot of boiler plate, it's not really any more boiler plate than we already have to do. To anyone switching from Jython or Nashorn JavaScript it's just a different set of boiler plate, not really more boiler plate. And anyone switching from Rules DSL to any other language already has to add a lot of boiler plate to their rules.

And really, this is what the Helper Libraries are all about. A way to abstract and consolidate all of that boiler plate and even more boilerplate once you look at what it takes to actually create a rule in a text file in any of these languages. Without them (or their equivalent in GraalVM JavaScript or jRuby) writing even simple rules requires a level of understanding for openHAB internal classes and structure that is beyond what should be needed for our users. It has always been my opinion that the automation add-on by itself has never been sufficient on it's own. A helper library for that language is also required to make the interactions between OH and the script seamless. To date, none of the helper libraries that have been written are even part a part of the OH project though. And while I think it's theoretically possible to package a helper library as separate add-on there are none who have done so yet. @jpg0, correct me if I'm wrong but your helper library is installed through NPM. The jRuby helper library is installed through ruby gems. The Nashorn/JavaScript/Groovy? helper libraries are installed by cloning the github repo and copying the files to the OH conf folder. Why do I need to resort to an external tool to install the library that makes the language usable to write rules?

Were I king for a day, the helper libraries for these would be a part of the add-on so you get both. Then the docs become easier to write, the code gets easier for end users to write, and the overall experience is much nicer. But every time I bring this up I seem to get pushback which I don't understand.

Anyway, I'm off my soapbox. Rant over, for now. ;-)

digitaldan commented 2 years ago

I wonder if it would make sense to have a generic "Javascript" binding which is more like a "meta" package, so mostly empty, but depends on both the GraalVM binding and a "GraalVM Runtime Helpers" binding which simply injects the runtime objects, logging ,etc... Most user users would install this generic "Javascript" binding (and get these two as dependencies) and be happy. Advanced users could install the raw GraalVM binding and have more control over the namespace.

jpg0 commented 2 years ago

@rkoshak spot on with your explanation. Helper libraries are essential, and the standard JS module loading (require("mylibrary")) is so much better than the non-standard, fragile and overly complex code required in Nashorn.

@digitaldan My fear is that bifurcating script runtimes will mean that we have two 'flavours' of script which will lead to greater confusion in the future when users share their scripts etc. From many years of development experience I firmly believe that too much 'magic' always bites you in the end.

As a pretty trivial example, there is nothing stopping there being new variables added to the default context - in fact any part of openHAB core or any add-on can do it (and it is designed this way) - what do they choose as the name for the variable? Because if it has already been used in a script somewhere, this can cause nasty problems (and with existing names like event, I expect the chance of collision is high). My belief therefore is to keep things explicit, but as short as possible.

All you need at the top of your script is a statement like Object.assign(this, require('@runtime')); to choose to merge all defaults into the context. This would mean that existing scripts would continue to work, although TBH I would discourage it because it's explicitly choosing the dangerous path of defining who-knows-what.

Other, better, standard options:

(Note that it is even possible to put your code in a with statement that merges this context explicitly for a subset of the code: with(require('@runtime')) { itemRegistry.<something> }) but note that (as in the link above) this is deprecated (and I'd advise against it) because it's a source of bugs/ambiguity/confusion - although less that the current openHAB approach!)

rkoshak commented 2 years ago

To plays devil's advocate here a little though. While I agree that too much magic can become a problem, and in fact I think it is a problem in Rules DSL, too much boilerplate can also be a problem.

When one is writing rules in text files where you might have 100+ lines of code with two or three rules defined adding one or two lines to the top of the file to import stuff you will almost always need is no big deal. It's just the cost of writing code that will work with openHAB.

However, when it comes to UI rules it becomes much more onerous. In my experience a typical Script Condition in Nashorn right now is about three lines of code that does real work plus another four to five lines of code to import the stuff I need even with all the magic that Nashorn provides with items and event and whatnot.

It's not quite so bad in Script Actions as those, again in my experience, average around 20 lines of code that do real work again with another four to five lines of code to import stuff. But that still means roughly 1/5th of the code I need to write for each and every Script Action (and there can be more than one Script Action in a single rule) is boilerplate. I'm always gonna want to log. I'm always gonna want to interact with Items. I frequently need to interact with the ItemRegistry and openHAB core Actions (which are harder to use than binding provided Actions today). So I'll have to import these for each and every Script Action I write.

If you assume one Script Condition and one Script Action per rule and a modest 15 rules, that's a lot of boiler plate. On the low end that's 120 lines of code just to get started. We haven't even begun to actually implement anything useful, we've just imported the stuff we need to interact with OH.

At least in the context of UI written rules, anything but the "dangerous path" described above is placing a pretty big burden on our least technically adept users. Even with the dangerous path we are looking at 30 lines of code out of 375 lines of production code being boilerplate. That's almost 10%.

This sort of thing is also why I always bring up "have you thought about how it will work in UI rules?" and why I get so frustrated when the answer is invariably "don't know. haven't thought about it. probably but that's not my focus." These kind of choices impose a pretty big burden on the OH users who are least able to bear it.

If one is writing rules in the UI one is most likely not an advanced coder. What one is doing is relatively simple and the likelihood they'll ever need to import a third party library is pretty small. So why impose a 10-20% burden on these users if we can avoid it?

jpg0 commented 2 years ago

An area which is maybe an extreme version of your example is JS transformations (defined inline, not in a distinct file). My belief is that for many of these, users just want a simple line of JS, such as input.name.toUppercase() or something which relies on openHAB such as itemRegistry.getItem(input).getTag('foo') or whatever. This seems like an extreme case of needing something as concise as possible, where we really only want a single line.

One thing that I would note is that with ES6 you can import everything you need on a single line: let { thing1, thing2, ... thingN} = require('my_library'). You cannot do that with ES5, so I would hope that general imports are vastly reduced. It would be interesting to see if you existing Nashorn scripts would grow or shrink with the current GraalVM version.

Maybe one option to help this would be to allow users to define a custom header for all their scripts in the UI? It could default with a load of things imported. They could add things to the list if they want, for all their scripts. I would just want to ensure that openHAB never updates the list itself (which it does today) as that can break existing scripts.

rkoshak commented 2 years ago

I would expect them to shrink quite a bit, though it would still be irksome. We will find out soon I suspect, especially if Yannick's proposal to limit rule templates to GraalVM JS comes to pass.

Even though I focused on lines of code, that's only part of the problem though. The users also need to know what to import in the first place. That too is a burden, especially for the less skilled users. It can be greatly alleviated with a Helper Library but without it one needs to know of and how to use the Java and OH Javadocs just to know what to import and from where and how to use them.

One thing that I would note is that with ES6 you can import everything you need on a single line

One question out of ignorance. That imports the OH Java classes we might need, JS libraries, or both? Could I use that same line to import say java.time.ZonedDateTime as well as my personal metadta.js library in $OH_CONF/automation/blah/blah/blah?

I like any approach that makes creating rules in the UI better for the less technical users.

jpg0 commented 2 years ago

To answer your last question, it allows importing any symbols that have been exported by the JS library that is being referenced. It does not allow importing Java classes directly (that could not be part of standard JS), however there is nothing preventing the JS library exporting Java classes for the script to import. It would probably even be possible to create a library allow importation of arbitrary java classes, which could be used like: let { java_time_ZonedDateTime } = require('js-library-to-access-java-classes').

Note that this style of import requires the name of the thing being imported, and in this case a . is not part of a valid JS identifier (hence my switch to _, which is probably not safe as Java also allows it in class names, but I'm sure there would be another option). It is possible to rename the local reference though - let { zonedDateTime:java_time_ZonedDateTime } works.

ghys commented 2 years ago

When making https://github.com/openhab/openhab-webui/pull/1170 I found that Java.type() still works in GraalVM like in Nashorn to import Java types.

i.e.

image translates to:


if (typeof(require) === "function") {
  ctx = this;
  var runtime = require("@runtime");
  itemRegistry = runtime.itemRegistry;
  events = runtime.events;
}

function mathRandomInt(a, b) {
  if (a > b) {
    // Swap a and b to ensure a is smaller.
    var c = a;
    a = b;
    b = c;
  }
  return Math.floor(Math.random() * (b - a + 1) + a);
}

var logger = Java.type('org.slf4j.LoggerFactory').getLogger('org.openhab.rule.' + ctx.ruleUID);

events.sendCommand('NewItem', (String(mathRandomInt(1, 100))));
logger.info(itemRegistry.getItem('NewItem').getState());

...and the logging works.

rkoshak commented 2 years ago

So it's looking like my original assertion that an automation add-on without corresponding helper libraries is going to be pretty unusable for many OH users still. The average OH rules developer shouldn't have to know that ZonedDateTime needs to be imported one way and personal library handled a completely different way. They shouldn't have to spend time looking through the JavaSE Javadocs and the openHAB Javadocs to know what needs to be imported from where and then figure out how.

I know this complaint is bigger than this particular issue but my biggest concern with Jython, Nashorn JavaScript, GraalVM JavaScript and all the rest is that we require the end users to know and understand far too much about differences in types (just today I had to explain why someone can't just multiple a Number Item's state by 1000 because the state is a java.lang.Number but 1000 is a Python primitive int). They have to know far too much about openHAB internals and they have to pay close attention to know when they are working with a Java Object or a language native entity. Helper libraries can greatly help with these but, as I said, those are all outside of the OH project.

Take creating a Timer for example. First I have to find where createTimer is defined. OK a search of the openHAB Javadocs shows its in org.openhab.core.model.script.actions.ScriptExecution. Now I need to know that I have to Java.type that class to bring it into scope so I can use it. But createTimer requires a ZonedDateTime. So most of the time (but not always) I'll have to also do a Java.type on java.lang.ZonedDateTime so I can get access to now. That's already a good deal of research and two "imports" required just to create a Timer. Of course a Helper Library would be able to hide that. You could even pass it something like "2h3m5s" as the "time" and the helper library can handle all that conversion.

That's what's missing and what I think is absolutely required to make these languages usable. Without them it's just too hard. You have to be a pretty experienced programmer to get anywhere or be a pretty lucky cargo-cult-programmer.

So I wonder if the discussion about imports and such is heading down the wrong path. Maybe we need to focus on making these helper libraries a part of the OH project itself and making them as easy to install as an add-on, or even better make them a part of the automation add-ons themselves. A whole lot of these problems can be handled in the helper libraries.

digitaldan commented 2 years ago

So I wonder if the discussion about imports and such is heading down the wrong path. Maybe we need to focus on making these helper libraries a part of the OH project itself and making them as easy to install as an add-on, or even better make them a part of the automation add-ons themselves. A whole lot of these problems can be handled in the helper libraries.

I completely agree with this, our focus should be "What's the best experience for the majority of users?" Right now, this is not an ideal solution for the vast majority, which is a shame, as i think this could be one of our best features!

My fear is that bifurcating script runtimes will mean that we have two 'flavours' of script which will lead to greater confusion in the future when users share their scripts etc. From many years of development experience I firmly believe that too much 'magic' always bites you in the end.

So i would agree more choices is not desirable here, my suggestion was under the assumption that there is a use case for having a runtime without global namespace pollution . But i believe this only benefits a minority of users, so to quote Rich, if i were king for a day, there would be one javascript binding, and that would have sane global objects that would allow me to access runtime features like logging, items, etc... Without this, the GraalVM binding is not useable, even to experienced users.

Could this conversation not pivot to what that global injection namespace looks like? Could we not restrict this to a few common ones that would stay constant, but whose internal functions may grow over time (like a runtime object) ?

I am fully aware many here have vast previous experience discussing this, and i am not suggesting anything new or revolutionary, but i'm also feeling like we have been stuck for awhile in this situation, and I for one would really, really like to start using this binding.

jpg0 commented 2 years ago

So it's looking like my original assertion that an automation add-on without corresponding helper libraries is going to be pretty unusable for many OH users still.

Interestingly, when I first started looking at using JS with openHAB, this was a point of contention between myself and Scott (at the time). I did agree with him that we should push most functionality into Java, however there are too many rough edges in the Java-to-other-language conversions that makes using Java objects directly fraught with challenges (often due to type issues). I do agree that we should have helper libraries for each language, and that these should be the primary interface to openHAB for script writers. I couldn't agree more that script writers should not have to know Java! Personally I have no problem with helper libraries diverging between languages, because this allows idiomatic usage of each language. As for whether these are bundled in the add-on, or installed via the package manager for the language in question, I don't really mind. I'm aware that this isn't a new discussion. (Oh, and I'm not sure about libraries being part of automation itself, as this would make updating them really hard and they would likely fall behind.)

Could this conversation not pivot to what that global injection namespace looks like? Could we not restrict this to a few common ones that would stay constant, but whose internal functions may grow over time (like a runtime object) ?

This is certainly something which has been raised before. However we could not use anything else that could clash with existing names, and I'm sure that runtime is widely used elsewhere. Typically languages use nasty things like __openhab_runtime__ for symbols like this, although I know that won't be popular! Note that even when injecting something with a non-clashing name, you are still going to cause problems for skilled JS developers because it will cause problems with their toolchains (linting, typescript, even auto-completion).

My view is that we can use a combination of the all the discussed approaches. What I would love to see is:

My hope is that this should be able to serve both beginner and advanced users. I do believe that advanced users are worth supporting because I expect them to generate community libraries. For example, I have my system to calculate and adjust light temperatures throughout the day, for which I use chroma.js, and it would be great to make this available to the community. (And maybe a supporting point is the lack of libraries for ES5 because the dev experience is so poor.)

the GraalVM binding is not useable, even to experienced users

I agree with this! Although maybe not for the same reasons - there are bugs, a broken directory layout and no docs, all of which need fixing before I would declare it ready. (I am confident in it's stability and capability due to having using it extensively over more than a year.)

ghys commented 2 years ago

I think it's a sensible approach if we do the transition in phases and make proper announcements - something like:

Also I think there could be a case for the new marketplace to distribute scripting libraries as add-ons, and even custom Blockly blocks made by the community (described declaratively, like UI widgets or rule templates - defining that could prove challenging but is not impossible). A dedicated developer tool in the UI would allow defining the block's structure and the code they generate, in YAML, like widgets. These blocks would make Blockly have even more an "educational" value to less experienced users, as they would be able to learn from the code that was generated and have a better sense of how things are done.

J-N-K commented 2 years ago

The suggested timeline ties openHAB to Java 11 for nearly 1.5 years, since Nashorn was removed in Java 15.

lewie commented 2 years ago

I'm essentially still scratching around with my 3 year old very rudimentary scripts that I handed over to the openhab-helper-libraries back in the day. Since then I'm waiting for the firm integration of the really incredible great work you've done with GraalVM in openHab. A few times I set up a test system with it and I had tears in my eyes, when I finally will be able to use these things in production systems. Disappointed I am back to my old systems from many years ago.

Unfortunately I and many people can't use this ingenious GraalVM work of yours, because the basic architecture of the integration in openHab is still in flux after now three years and even the contradictory documentation drives me crazy - and I'm tough. For an own clean implementation my dusty Java architecture knowledge is not enough.

Get rid of the nashorn-engine and agree on fixed rules for the connection and include the GraalVM as standard. The resentment of some users will surely come, be limited and quickly subside. The gratitude of the newcomers should outweigh by orders of magnitude.

The once rewrite of no matter how many scripts is not the problem! I think it's not just me. The horror is the constant adjustment with several systems because constantly something changes or something is missing or there it is missing, then it is again outdated in the next version, because nashorn this or that etc..

Shorten the changeover. Finally make a cut with the old nashorn-engine, please. Preferably today and here.

Yes, I know, that is much too heretical... ;-) Or isn't it?

Please do not be angry at my so clear words.... I hope I can dedicate myself as soon as possible again much more with the further development of openHab and make a contribution. But life is sometimes curious.

digitaldan commented 2 years ago

in openHAB 3.2 (end 2021) the built-in Nashorn engine will expose all openHAB objects under openhab

Agreed , this sounds very reasonable and what i was thinking as well.

in openHAB 3.3 (mid-2022) the global objects will be removed from the Nashorn built-in engine

I don't think we have to go this far, i would rather us keep it there until we decide Nashorn is no longer included , I know there will be some migration woes as people need to change things like events.sendCommand to openhab.events.sendCommand when we remove Nashorn, , but i think this is minor inconvenience for the very large benefit of moving to GraalVM, and as @J-N-K mentions, we may want to get off Nashorn sooner and not be stuck with a old version of Java (again ). We have pretty long release cycles, so i feel we do a better job than most at communicating breaking changes.

I also want to clarify something , there was a mention of only injecting when using the openHAB UI? I'm not clear if thats actually the intention, but If that is the case, i think that would be a huge mistake. For one, it means one could not freely copy scripts to and from the UI to files, leading to all sorts of confusion. Also, while i don't think injecting a global object to interact with openHAb as "magic" by any means, doing so when only using the OH UI would be magic, as i would have a hard time explaining why that choice was made . I think we need our engine to behave one way, consistently, no matter how you configure openHAB (UI, files, or both). If this was not the intent, then please disregard this whole paragraph :-)

Not to jump the gun here, but if we can agree on using openhab as "the" global namespace object, can we move forward defining what this objects holds (event, item registry, logging, etc...) ?

jpg0 commented 2 years ago

I also want to clarify something , there was a mention of only injecting when using the openHAB UI?

My suggestion is to never inject, rather to allow the UI to provide an explicit header which declares all the things that a script writer is likely to need. When viewing the script in the UI, this should be present.

Something that I'd also like to point out regarding a 'global namespace object': When I was porting much existing JS to the new engine (I think much written by @lewie - thanks very much), I found it much easier to not use a global object. The porting (or maybe: updating) process goes as follows:

This did not require going to each reference of an openHAB object and modifying it to reference via some namespace. This could absolutely be done, but it's simpler to not do it (and leaves more concise code) because you don't need to find each reference.

rkoshak commented 2 years ago

I had a really long reply with responses to everyone but this is a more concise rewrite.

@jpg0, what makes including the helper library with the automation add-on hard (an honest question, not being rhetorical)? Is there something that could be done to make it less hard? If we are proposing the helper library to be the main interaction with OH anyway, shouldn't it be at least a little hard to make changes, particularly breaking changes?

I would argue that the more similar that the helper libraries can be between the languages, within the constraints of the languages, the better because it will make it easier:

Personally I have no problem with helper libraries diverging between languages, because this allows i Typically languages use nasty things like __openhab_runtime__ for symbols like this, although I know that won't be popular!

That symbol doesn't bother me one bit. Users who are not programmers will just treat it as one more arcane symbol they need to use. I don't think that's a problem.

I agree with @jpg0's suggestions as being reasonable and achievable.

@ghys, the proposed timeline seems reasonable. However, only if we can get some docs written for GraalVM and sort out the helper library issue for it relatively soon. I don't want to deprecate Nashorn until we've given the tools to the users to start the transition.

I don't know that we can fully remove Nashorn until OH 4 though. That would count as a pretty big breaking change, right? Those are usually reserved for full version updates. Maybe it's an issue to bring up to the AC? We need to not forget that removal of Nashorn also has implications for the JHS Transformation too.

And I really really really want the marketplace to support distributing scripting libraries. There are lots of cases where one might have a library that isn't really a helper library but isn't really a rule either.

+1 to Blockly libraries.

For one, it means one could not freely copy scripts to and from the UI to files, leading to all sorts of confusion. Also, while i don't think injecting a global object to interact with openHAb as "magic" by any means, doing so when only using the OH UI would be magic, as i would have a hard time explaining why that choice was made .

That's not really all that easy to do now. The structure of how a rule is defined is so radically different between the UI and text files for all the languages, and the UI supports the conditional clause which, to my knowledge doesn't exist in text based rules. So an easy copy in the UI to text files direction is not going to be so easy anyway. And the existing Helper Libraries makes it challenging to go in the other direction too because the annotations do so much for you that isn't possible to do in the UI (e.g. @rule in Python scripts creates a logger for the rule for you).

In fact there will be rules created in the UI that simply cannot be copied as is to a text based config. For example, I can have a rule in the UI with more than one language used. Really simple rules don't even involve code at all.

So while I'm Mr. Everything Must be Consistent, in this case I don't think it's that big of a deal. It's going to be challenging to go from the UI to text files anyway.

NOTE: TIL that you can extract individual rule's JSON and put that as a file under Automation. So technically everything I just said is wrong except in the case where one wants to move from JSON to a native code file. I'm not sure what that means but it felt worth mentioning.

The porting (or maybe: updating) process goes as follows:

try to run the code, get a " is not declared" error in file import at the top of file repeat

This sounds like a pretty miserable experience for non-experienced programmer who just wants to write a rule that waits five minutes and adds a value to an Item (for example). That's going to be a lot of loops to do something relatively simple. And it doesn't address where/how the user is supposed to figure out how to import that thing that isn't defined.

Not to jump the gun here, but if we can agree on using openhab as "the" global namespace object, can we move forward defining what this objects holds (event, item registry, logging, etc...) ?

I think that's moving away from @jpg0's set of suggestions. If the user's primary interaction with OH is through the helper library then all the interaction with that global openhab Object is going to take place in that helper library, not in the rule itself. Doing anything else leaves us back in the situation where users have to care about type since everything in that Object is going to be Java. So we'd need to inject (I agree with @jpg, don't do it secretly, just automatically add the lines to when creating the script, maybe even make that optional in case someone wants to use an alternative or personal helper library instead of the "officail" one) some sort of basic part of the helper library.

Having said that, what should be included (I'm mainly talking about the UI here):

What Why Comment
event That's what triggered the rule and there is a ton of necessary information in there.
events That's how we interact with Items, sendCommand and postUpdate. I think it's really poorly named. With a helper library we'd have postUpdate and sendCommand that can convert from native stuff to openHAB State Objects as needed though so events wouldn't be used in a rule directly.
items Dict (or language concept) of Item names and states. I cannot express too much how much this helps to create concise and easy to understand rules. However, the states are all Java Objects so perhaps they need to be converted to something native, maybe put a native wrapper around it.
ir There are times when you need more than just the Item's state. Again, the ir interactions should be adjudicated through the helper library.
ThingRegistry To get the status of a Thing. Same as ir. Maybe this is an advanced thing that should be explicitly imported instead of made available by default.
Core OH Actions createTimer, sendHttpXRequest, etc. It makes no sense that the core actions have to me accessed differently from binding actions. However, in at least createTimer and executeCommandLine's cases, they need to be wrapped by the helper library so we don't have to mess with java.time.ZonedDateTime and java.time.Duration to call them.
Binding Actions Access to Actions provided by Things. The Helper Library can wrap these together with the core actions perhaps.
Logging Technically it's a Core OH Actions so see above.

Those are pretty core I think and used in 90% of all rules. There are some advanced things that the helper libraries should support too like access to the metadata registry, ability to create and delete Items (important for unit tests and perhaps rule templates), ability to create rules dynamically (important for unit tests and can be useful in some other circumstances).

I'm sure there's more. This is just off the top of my head.

ghys commented 2 years ago

I'm sure there's more. This is just off the top of my head.

There's the context variables that get defined with outputs from the triggers e.g. item, state, newState etc. I'm not exactly sure how these can be accessed in GraalVM, maybe this.ctx?

About the timeline, this was more an example than a suggestion. Removing Nashorn was only an idea, once and if we achieve feature parity, to avoid ending up with several JS versions of the same thing (one with const openhab = require('@runtime') and the other where openhab is implicitely defined). But I agree it could wait until the next major version.

rkoshak commented 2 years ago

There's the context variables that get defined with outputs from the triggers e.g. item, state, newState etc. I'm not exactly sure how these can be accessed in GraalVM, maybe this.ctx?

Those are all encapsulated in event. They might also exist independently but all the examples I've written and seen get at them using event. For example,

Rules DSL Jython/Nashorn
newState event.itemState
oldState event.prevState (going from memory, here, might be event.oldState
receivedCommand event.itemCommand
triggeringItemName events.itemName

I don't know how they are accessed in GraalVM JS either. It's one of the things keeping me from playing around right now. I've not spent the time to look into what's there and how to get at it.

digitaldan commented 2 years ago

NOTE: TIL that you can extract individual rule's JSON and put that as a file under Automation. So technically everything I just said is wrong except in the case where one wants to move from JSON to a native code file. I'm not sure what that means but it felt worth mentioning.

This is what i am referring to when sharing and copying scripts (and how i started migrating a few things).

I agree with @jpg, don't do it secretly, just automatically add the lines to when creating the script, maybe even make that optional in case someone wants to use an alternative or personal helper library instead of the "officail" one)

I'm still confused by this, so if i create a file and put it in the automation directory, i need to specifically include these helper libraries (so boiler plate code in every script), but if i use the UI its going to do that for me? This seems super hacky to me, if we are insisting on doing this every time in the UI, what not make a consistant experience? I'm trying to come around to this idea, but if i put on my "user" hat, it just feels like a poor experience. Could we make this an option of the binding? I would think by default most people would want this, but give the option for power users to turn it off.

I'm not going to dig my heels in on this topic as its more important to get the ball rolling and i may be in the minority , but i am trying to advocate for having a really clear automation environment that works the same (as much as we can) for files or the UI , if i'm the lone voice here, then i will go along with this and assume i'm overthinking it :-)

rkoshak commented 2 years ago

I'm still confused by this, so if i create a file and put it in the automation directory, i need to specifically include these helper libraries (so boiler plate code in every script), but if i use the UI its going to do that for me?

That's part of the discussion here I think.

If we want to address the problem where users have to mess with both Java and JavaScript inside their rules and continue to deal with the problems where, for example, a java.lang.Number is not compatible to do math with a JavaScript int we need a helper library to basically abstract and adjudicate the interactions between the user's rules and OH.

If we need a helper library to do that it should be there be default, or at least very easily available through an add-on.

If we want to support using third party libraries, we need to avoid automatically injecting a bunch of stuff into a script as that will lead to name conflicts.

If we want to make UI created rules less painful for non-developers to use and write, we need to limit how much proforma (i.e. imports) is required in each and every script action and script condition. It's not just the added lines of code, it's the knowledge and research to figure out what needs to be imported in the first place. "symbol not defined". doesn't tell me I need to import the ScriptExecution class in order to call createTimer.

Given all of the above I think what is being proposed, or at least discussed is:

If you exported the JSON and wanted to use JSON files, those imports would still be there so the experience would be consistent. It's only if one wants to abandon the JSON and write everything in the native language that they would need to know and make the imports manually. And by making the line visible and editable we give the advanced users the ability to override it if they want to (and know what they are doing). But because it's inserted automatically less capable users can proceed without needing to add it themselves.

I would think by default most people would want this, but give the option for power users to turn it off.

I think we are on the same page here. When I create a Script Action there might be some lines of code already inserted into it that makes the commonly needed stuff available to me. But if I'm an advanced user, I can delete or modify those lines of code. Less work for the average user with all the control needed by the advanced user. And if we are worried about consistency between UI and .json files it's the exact same JSON in either case so that's consistent.

jpg0 commented 2 years ago

@rkoshak to answer your question about whether helper libraries can be bundled with add-ons: I don't see any technical reason why they cannot. It's certainly possible with GraalVM/JS. I do know that Scott was looking into this as one point but I think he struggled to get decisions made.

Also, regarding

That symbol doesn't bother me one bit.

I realised that I was caught out my github markup formatting; what I meant was __openhab_runtime__. Languages often use prefix and suffix double-underscores to try to avoid clashes.

Also, I'm not sure about whether my statement of

try to run the code, get a " is not declared" error in file import at the top of file repeat

was fully understood - I mean this is the experience for porting existing code running in Nashorn. I am not advocating that this should be the experience for new script writers (even if it is for more developers!).

Regarding what symbols are actually available, these are all defined already in Java. I would not expect us to change this.

rkoshak commented 2 years ago

I realised that I was caught out my github markup formatting; what I meant was __openhab_runtime__. Languages often use prefix and suffix double-underscores to try to avoid clashes.

Yes, that's what I understood you to mean. The double underscores don't bother me and I don't think it'll bother many if any non-programmer user.

I mean this is the experience for porting existing code running in Nashorn. I am not advocating that this should be the experience for new script writers (even if it is for more developers!). :+1 OK, I missuinderstood and this makes more sense.

lewie commented 2 years ago

I hope I understood correctly and try to summarize:

This way the inexperienced can get started right away and the more experienced has no restrictions. If I have understood correctly, this sounds perfect for me.

jpg0 commented 2 years ago

This is how I understand it, with the clarification that

A helper library uses this header

Is that the header provides access to the helper library for the script

digitaldan commented 2 years ago

Agree with @lewie ( and others) this sounds great. Out of curiosity , i know there are helper frameworks/scripts maintained externally from openHAB, is there one that would be the basis for an official helper library, or is this something thats needs to be created form scratch?

rkoshak commented 2 years ago

I believe @jpg0 has a pretty good start on one for GraalVM JavaScript. Alternatively the Nashorn Helper Libraries which are based on @lewie's original helper library could potentially be refactored to work with GraalVM JavaScript. I don't think we need to start from scratch. @CrazyIvan359 is working on a rewrite of those to bring them to parity with the Jython libraries. https://github.com/CrazyIvan359/openhab-helper-libraries/tree/js-rewrite

For the Jython the Helper Library is already pretty complete and is perhaps the most mature so I'd say let's just use that until such time as Jython gets deprecated and dropped. https://github.com/CrazyIvan359/openhab-helper-libraries/tree/ivans-updates

NOTE the different branches in those links.

The jRuby helper library was approaching a pretty complete stage the last time I looked too, but I don't know when/if jRuby would be submitted as an official library. For now their library is installed via Ruby Gems.

My understanding of Groovy is that it doesn't support libraries in quite the same way as the other scripting languages so I'm not sure what, if anything is needed there.

There is a Java automation add-on in work too but because that requires compilation, I don't think it will ever be a candidate for use in the UI. Also, it's Java so the issues with type difference between languages isn't as much of a problem. I think that in that case the helper library is already built in.

Rules DSL doesn't support libraries anyway but the language already supports most of the interactions in an almost seamless way so no helper library is possible or needed.

That covers what I know about the state of helper libraries right now.

J-N-K commented 2 years ago

For Java it depends on the approach. I'll add a Java module to the marketplace in the next week that allows scripting in the UI.

lewie commented 2 years ago

I just think we've been discussing this without even knowing who is going to implement it. If nobody from our round here finds itself, I fear, the thing peters out again. That would be a real pity!

Does anyone have an idea who can do this, who feels addressed? Who has the skills and hopefully possibly the time to program the structures for header and auxiliary library? (I think I should not have asked for the time... 🥴) The connection of the GraalVM seems to run already useful @jpg0, or? Can this structure be taken over in the current state, or what is missing? Is possibly even someone already programming?

If the structures are ready, all objects in openHab can be accessed (here and what else is missing) and it is approved, then the conversion of the existing auxiliary libraries and default headers is no more witchcraft (I would also spend time for the Javascript part to adapt, no matter how...😉).

What do you think about it?

jpg0 commented 2 years ago

I plan to modify the GraalVM JS add-on to change the JS directory to <openHAB-Home>/automation/js (as in https://github.com/openhab/openhab-core/pull/2408#issuecomment-872506675), and library directory to node_modules under this. I also plan to write docs. Just haven't yet found the time.

As for helper libraries, they need to be written and possibly bundled into the add-on. I am not going to volunteer for this, however I am happy to offer the helper library I wrote (https://github.com/jpg0/ohj) as a starting point (or even a fairly completely 1st version!) for anyone who wants to do this.

digitaldan commented 2 years ago

however I am happy to offer the helper library I wrote (https://github.com/jpg0/ohj) as a starting point (or even a fairly completely 1st version!) for anyone who wants to do this.

I think that looks like a great starting point, and as you pointed out could be v1 on its own. Just spent some time looking at it, nicely organized and easy to follow. Where would this live in the OH repo? Would that be in the GraalVM addon ? I also love the fluent part, but wonder if maybe for v1 we should focus our time and testing OH basics, and maybe talk about pulling this in later?

rkoshak commented 2 years ago

I don't have the skills nor the time to learn the skills to do the work by myself in the short term. I can help test and I can probably contribute some code and docs for the helper library if given a bit of a boost but I've not even the slightest idea how to make anything part of an add-on or make a library an add-on by itself (though the latter is something I'd really like to figure out eventually).

If it's going to be bundled with the add-on it makes sense that it exist and be maintained as part of the automation add-on itself.

If it's going to be a separate add-on, I'd suggest it either be maintained in openhab-addons or a new repo created for the helper libraries.

I personally like the first idea best as it's less work for everyone all around. It's less work for end users because they only have the one add-on to install to get everything they need. It's less work for the developers as they don't have to generate two sets of docs, one for the automation add-on and another for the helper library. Since the helper library is to be the primary interface for rules developers anyway, the helper library is probably all that really needs to be in the docs, or at least the primary focus of the docs. Anyone who needs more details about the internal workings can look at the source code.

I agree the @jpg0's library is a great candidate for a first version based on a quick looking at it. It seems pretty complete already though it would probably be worth while to see if there is anything in the Jython helper libraries that is missing, just to be certain something important isn't left out. A lot of this stuff grows gradually and the Jython helper library has had the longest time to grow in this way.

I've not had chance to actually use it yet primarily because all my rules are currently written in Nashorn and until this thread and the hints from Yannick I would have to redo all my rules all at once which I don't have time for right now. With the ability to more easily migrate it should be less effort.

lewie commented 2 years ago

@jpg0, ufff! That's finished and ready to use, including documentation for the javacript part. This is indeed, a complete starting point! Fluent and classic possibilities as I can see. There is not much more todo. Great!!!

@digitaldan, what do you mean by "testing OH basics", they are just in there. Do you mean a special test library shortened to the essence to test access to OH objects and maintain them as a complete list and keep them in sync with openHab?

Such a test library that contains only the basics in the respective script language and shows how openHab objects are accessed from a script language. All the peculiarities of the individual languages would be minimized. That would be then a solid basis, from which the information for more complex higher developed auxiliary libraries as for example that of @jpg0 can be written and extended. If something changes, the Java programmer can add or change this quite easily in these test libraries and thus provide it to the script people. I did this in my first version for javascript, very rudimentary: (https://github.com/CrazyIvan359/openhab-helper-libraries/blob/js-rewrite/Script%20Examples/JavaScript/AccessibleFromAutomationTest.js).

For javascript I can do that. Provided I can get a GraalVM addon to run.

@jpg0, do you have a GraalVM addon that runs in the current openHab 3? And a small description of how I put things together to test and understand and install your helper libraries?

When this works (working GraalVM addon, 1st version helper libraries JS, test library accessible objects from openHab), next step would be to find a programmer for integrating the general header we spoke about?!

After all, we have gone far beyond the topic of this issue, should we open a new one?

digitaldan commented 2 years ago

If it's going to be bundled with the add-on it makes sense that it exist and be maintained as part of the automation add-on itself.

this is what i was thinking as well

what do you mean by "testing OH basics", they are just in there. Do you mean a special test library shortened to the essence to test access to OH objects and maintain them as a complete list and keep them in sync with openHab?

I meant focusing our time to include everything in @jpg0 's repo but the fluent part (for now), which is an abstraction over scheduling rules, it actually looks quite nice, but i'm not sure its "core" like accessing the item registry, events, logging , which would be common among other scripting environments. I was not talking about specific testing frameworks or strategy. My intention is to keep the first release focused, so testing that this works well , we have docs around it, and gets included as a first milestone. Please feel free to disagree if you think it does make sense to include fluent, I only worry about additional scope and our time.

@jpg0, do you have a GraalVM addon that runs in the current openHab 3?

Maybe i missed this, but i thought it was working once you created the right directories it was expecting?

After all, we have gone far beyond the topic of this issue, should we open a new one?

I think that makes a lot of sense!

digitaldan commented 2 years ago

I spent some time trying to get the ohj library working with GraalVM (jsscripting) on a SNAPSHOT version of openHAB without luck, i'm not sure if its me, jsscripting or the ohj libraries that are at fault. Here was my process:

  1. Clone, compile and install https://github.com/jpg0/ohj-support.git binding
  2. Clone https://github.com/jpg0/ohj.git into $OPENHAB_CONF/automation/lib/javascript/personal/node_modules/ and run npm install inside ohj
  3. Alternatively install ohj using npm install ohj from inside $OPENHAB_CONF/automation/lib/javascript/personal/ (i tried both 2. and 3.)
  4. Load a test script: 
    const log = require('ohj').log('js_scripting');
log.info("TEST");

I keep getting the following error:

 Failed to execute script:
org.graalvm.polyglot.PolyglotException: TypeError: Cannot load CommonJS module: 'ohj'
    at <js>.:program(<eval>:1) ~[?:?]
    at org.graalvm.polyglot.Context.eval(Context.java:345) ~[?:?]
    at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:379) ~[?:?]
    at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:356) ~[?:?]
...
digitaldan commented 2 years ago

Ok i spoke to soon, i'm not sure what changed, maybe reloading the script triggered something, but now i'm getting a different error

 Failed to execute script:
org.graalvm.polyglot.PolyglotException: TypeError: Access to host class org.slf4j.helpers.MessageFormatter is not allowed or does not exist.

This is actually encouraging as at least the library is loading, although now i think i am running into https://github.com/openhab/openhab-addons/issues/11222

jpg0 commented 2 years ago

The current version should, or at least did, work with openHAB 3.x.

The error above happens if a script tries to load a Java class which it cannot find - either because it doesn't exist, or it does not have access to it (either OSGi or the JVM preventing access). Possibly there is some other incompatibility here that has been introduced since openHAB 3.0.0, but I have not looked at it yet.