Kotlin Scripting Support

[ligee]

https://github.com/Kotlin/KEEP/blob/master/proposals/scripting-support.md

replaces https://github.com/Kotlin/KEEP/blob/master/proposals/script-definition-template.md

[DaanDeMeyer]

This looks really promising!

Would this allow me to distribute a single self contained script template which could then be opened in Intellij without any other configuration being required? Ideally I could distribute a script template with a reference to the script definition and Intellij would automatically download this definition from Maven and provide rich code completion.

[ligee]

@DaanDeMeyer this is not yet in this KEEP, but it could be an interesting addition. I'll think about it. Thanks!

[artem-zinnatullin]

@ligee 2 questions:

• How is Spek related to scripting?
• What about unit testing for Kotlin Scripts? (tests for scripts themselves)?

Thanks!

[holgerbrandl]

Great that you guys are planning to push scripting to a new level. It's an amazing KEEP and I especially liked the bits about declarative bootstrapping of an IDE dependency and runtime environment. It's also great to read how you envision to cover the whole range from build-files to shell scripts with a unified framework addition.

Though, it reads like a very ambitious plan to me including lots of sub-projects. So I'd be curious about the indented time-frame.

1. You mentioned the difficulties of parameterized shebangs, and indeed they don't work across shells, so using a declarative approach is the way to go I think.

2. It would be helpful to spec out the declarations first, so that third party tools like my kscript (see https://github.com/holgerbrandl/kscript) could support these declarations rather soon in order to fill the gap until an official JB solutions is ready. Once the latter is done, kscript would be obsolete for good.

Looking forward to kontlinconf so that we can chat more about it!

[ligee]

@artem-zinnatullin: About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl. Basically, it means using scripting infrastructure to produce regular project class files from script-like sources with reduced boilerplate, like necessity to place spec tests inside a class. About unit testing scripts - it is, of course, possible to use script runner inside tests, but we do not plan any additional helpers/facilities to simplify scripts testing. Maybe we'll consider it later though.

[ligee]

@holgerbrandl, thank you for the feedback. We will try to keep the size of the work here under control. Significant parts of the infrastructure changes described here are, in fact, reshapings and unifyings of the existing scripting infrastructure. But there should be, of course, some particular implementations on top of that, and implementing them will take time, but with clearly defined interfaces we may be able to outsource it partly to the community. But at the moment it is work in progress, so the plans could change. As of the specs for declarations - we'll try to keep it mind. But first, we have to finish more general infrastructure specs.

[artem-zinnatullin]

@ligee

About Spec - it is described here - https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md#script-based-dsl

Yep, I saw Spek there, that's why I asked :)

like necessity to place spec tests inside a class

Spek is a pretty complex test framework that requires test engine/runner to work (currently Spek uses JUnit5 infrastructure). Also, usually you run tests through build system, not sure how scripting will handle that.

This can save one nesting level by removing top-level class (which doesn't look as a pain point atm), but on the other hand running tests seems to become much more complicated. Technically everything is possible, but I don't think this is direction we want to move Spek into at the moment.

cc @raniejade, @hhariri PTAL

[apatrida]

@ligee for script support, it looks like you have something roughed in for whitelist/blacklisting of functions/classes that can be accessed. This will be important for us to embed Kotlin scripting in Elasticsearch or other engines like Arango, Neo4j, etc. Ensuring what can be called at any given moment, or even what classes can be loaded. Lastly, time limiting of the script running, or loop iteration limiting protections as well (endless loops kill these type of systems). The more integrated this protection can be, the better.

[unlimitedsola]

[gtnarg]

Is there anywhere to follow the progress on this?

[ligee]

I'm working now on the updated proposal, along with the series of the prototypes. As soon as it will be ready, I'll publish the updated version. Hopefully, it will happen within a few weeks from now.

[sdeleuze]

I am interested to leverage these improvements in Spring and Riff.

[apatrida]

@ilya-g please add the requirement: Between compilation and class loading, there should be the option for a verifier that can approve or reject a script. This allows secure scripting to be tied into anything that executes those steps concurrently without needing to re-write every permutation of script runners that might be created. Allowing something like Cuerentena integration https://github.com/kohesive/cuarentena ... There should never be a hard-coded "compile and load class" without the ability to interject between those steps, before class loading.

[bhtek]

@apatrida Thanks. In cases where the script content came from outside of development, e.g. some simple formula for calculating P&L etc. It is critical to validate before loading for the feature to be usable in production.

[ligee]

The 1.2.50 is out with the new experimental scripting support. The KEEP is updated too, reflects now the approach taken with 1.2.50 and actual implementation status. Not all feedback is incorporated (yet) into the KEEP, but I believe that what we have now is a good foundation that could be developed in the right direction relatively quick and easy. Any feedback will be highly appreciated!

[ligee]

@apatrida, the current approach allows to write a scripting host that will get a compiled script before instantiating/evaluating it and can perform any checks at that point. Could it be an acceptable solution for your case, or you rather believe that we should introduce another entity - a Checker - with a predefined interface, that could be used with standard hosts?

[apatrida]

@ligee it should be after bytecode is generated but before any class is loaded to prevent attacks during static initialization. And would be nice if many implementations that might exist of scripting hosts don't all need rewritten to add verification. So some plugin point that can be exposed in the common hosts and hopefully whatever wrapping people do around those.

I'll look at what is in 1.2.50 and see how it is currently set up.

[ligee]

@apatrida it could be done now exactly at the moment you described, but having an extension point for it in the standard host interface looks like a good idea, I'll think about it.

[udalov]

@ligee As noted by @rgoldberg in #124, kotlin-scripting-common is listed twice in the "Implementation status" section, which is probably a typo

[yschimke]

It would be great if this optionally supported continuations via an implicit surrounding runBlocking call. Avoid nasty boilerplate and allow suspending functions to be called from top level (without nesting).

[LouisCAD]

@yschimke You can already have top level suspend fun in .kts files

[yschimke]

@LouisCAD to clarify, I really meant calling them at the toplevel e.g. without runBlocking here

https://github.com/yschimke/oksocial/blob/master/src/test/kotlin/commands/text-to-speech.kts#L71-L78

[cmorriss]

I've been playing around with this and noticed that the cache interface for the compiled scripts doesn't seem to accept a script source reference when storing the compiled scripts. I'm curious then how the get() is able to return an instance based on a script source?

[ligee]

@cmorriss, seems a bug, thanks for pointing out! It was lost during the last refactoring when the source was removed from the configuration.

[jetersen]

I haven't found any good examples of how to actually setup a script project while having full IDE support and make it run. 😢

[cmorriss]

I've noticed that this seems to require access to the full kotlin compiler. Our use case is to run it as an embedded component, allowing scripts to be incorporated as sandboxed plugins in a backend service. It would be great if this could run utilizing kotlin-compiler-embedded as a dependency. It pulls in fewer dependencies and would allow this to more easily run as a library.

[cmorriss]

Just to add more context, when I try to run using the embedded compiler, I get the following exception:

java.lang.NoClassDefFoundError: com/intellij/openapi/util/Disposer
    at kotlin.script.experimental.jvmhost.impl.KJVMCompilerImpl.compile(KJVMCompilerImpl.kt:99)
    at kotlin.script.experimental.jvm.JvmScriptCompiler.compile$suspendImpl(jvmScriptCompilation.kt:43)
    at kotlin.script.experimental.jvm.JvmScriptCompiler.compile(jvmScriptCompilation.kt)
    at com.amazon.fuselets.kotlinscript.FuseletScriptCompiler.compile(FuseletScriptCompiler.kt:19)
    at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.doResume(FuseletScriptingHost.kt:28)
    at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.invoke(FuseletScriptingHost.kt)
    at com.amazon.fuselets.kotlinscript.FuseletScriptingHost$eval$1.invoke(FuseletScriptingHost.kt:14)
    at kotlin.script.experimental.host.BasicScriptingHost$runInCoroutineContext$1.doResume(BasicScriptingHost.kt:30)

It looks like this class, among others, has a modified package name (I'm guessing to prevent collisions when running as an embedded component) of org.jetbrains.kotlin.com.intellij.openapi.util instead of the normal com.intellij.openapi.util. Maybe these classes can be encapsulated and have the system catch the first ClassNotFoundException and pick the right one under the covers for future calls and instance creation needs. Just a thought.

[ligee]

@cmorriss In fact, there is no difference in size and dependencies between "full" compiler and embeddable one. The latter is intended for usages in the environments already containing some of the libs packed into the jar, e.g. IDEA's openapi. And it also requires the other components to be remapped similarly, that's why you're getting the error above. We do not provide yet the appropriate scripting components for using with the embeddable compiler (and I'm not sure we want to do it, taking into account related problems and misunderstandings). In short, I do not recommend to do it. Please use the "full" compiler instead.

[cmorriss]

@ligee Thanks for the response. While it is possible to use the full compiler, we'll have to figure out a way to do so in a contained classloader as our particular use case does fall under the "environments already containing some of the libs packed into the jar" scenario.

In particular, this goes beyond just things like Google Guava. It actually includes the Kotlin standard libraries themselves. Our use case is to utilize Kotlin scripting as a backend service plugin language running in a tightly contained sandbox. This allows the logic of our services to be tweaked through these scripts by providing functional business logic in well defined extension points.

These services all depend on Kotlin at a 1.2.x or 1.3.x level instead of a specific version, like 1.2.60. This allows for backwards compatible updates to be deployed without the need to update a ton of version numbers throughout our infrastructure. However, the compiler is clearly tied to a very specific version of Kotlin.

The problem is that by including the full compiler as part of our plugin library, it can cause conflicts as the Kotlin libraries that the service depends on may diverge from the version that is included in the compiler since they can be deployed and updated independently.

The only way we can avoid this issue currently would be to load the compiler in a separate classloader, perform the compilation there, and then run the classes in the classloader of the primary service. This can work (I hope!) as we can at least be sure that the Kotlin compiler is at or below the version of the Kotlin standard library for the service. We just can't be sure they're at exactly the same version level.

I believe that running with the embeddable version of the Kotlin compiler would solve these issues as it's embedded libraries would not conflict with those of the service embedding it.

In the mean time I'll work on the classloading solution, but would greatly appreciate if using the embeddable version of the compiler could be investigated further. I'd wager we're not the only use case for this, but I could be wrong.

Would be happy to hear I'm wrong about this as well!!!!

[jerves]

@cmorriss,@ligee, Hello, is anyone can tell me how can I define my kotlin script that has some functions with parameters, and how can I visit that functions in my custom project? With the new experimental scripting support. Thanks!!

[cmorriss]

@jerves Not sure how "correct" this is, but it works.

https://github.com/cmorriss/kotlinscript-sandbox/blob/master/src/main/kotlin/KotlinScriptSandbox.kt

Essentially, the script has a function that matches the signature of a function in the abstract script class. The script is compiled to a class file, instantiated, and cast to the abstract script class.

At that point, you can call the function, passing in whatever parameters are needed, and it will execute the code in the script and give you back a return value.

Edit: Note that this is written against the 1.3.0 RC-80 version of the scripting support, so subject to change as the scripting support evolves.

[ligee]

@cmorriss this looks like you're exploiting a compiler bug here. So I wouldn't recommend this approach. If overriding the base class functions is something important, let's discuss how to support them properly. Otherwise, I guess we should fix the bug and disallow accidental override.

[ligee]

@jerves, could you, please, elaborate. Are you looking for something similar to what @cmorriss proposed (then please see my previous comment, before starting to use it). Or you simply need to iterate over functions in your script - in this case maybe the reflection could help you.

[cmorriss]

@ligee Thanks for the reply. Good to know. I definitely want to utilize a supported path and iterating over the functions and using reflection can achieve the same result.

It would be a "nice to have" feature to supply an interface to the scripting host, in addition to the script source, such that it would return an instance of that interface that could be called to execute corresponding functions in the script. If the script does not provide functions that match the signatures of the interface, an exception is thrown.

I'm planning to provide this using a dynamic proxy to forward the calls to the script instance functions, although it would be very convenient for others who have a similar use case be able to access some similar functionality that's baked into the scripting support.

[ligee]

@cmorriss I created an issue to track this topic - https://youtrack.jetbrains.com/issue/KT-27349

[cmorriss]

Thanks @ligee . I also created a separate issue to track the possibility of creating an embeddable version of the jvm scripting host that would work with the embeddable compiler here - https://youtrack.jetbrains.com/issue/IDEA-199942

[jerves]

@cmorriss I created an issue to track this topic - https://youtrack.jetbrains.com/issue/KT-27349

Thanks！It is exactly my use case. And I want to know in which version can support this use case?

[apatrida]

@apatrida it could be done now exactly at the moment you described, but having an extension point for it in the standard host interface looks like a good idea, I'll think about it.

Did this happen as described @ligee , I'm about to dig in to update my scripting libs

[ligee]

@apatrida I haven't implemented any extension points yet to simplify such checks. Please try to make a custom host for that and tell me if there is anything missing now. Alternatively, there is a compiler cache interface that receives the compiled bytes, so maybe it is possible to hack something there. And finally, you can try to wrap the default script compiler implementation and make checks there. I'll try to add missing pieces fur such functionality on the next API update iteration.

[balage1551]

I am pleased with this API, especially since Oracle declared Nashorn deprecated without replacement. In our project, we use intensely scripting to add customer specific calculations to our engine. As I read, 1.3 will contain only a subset of all the features. Is there some kind of priority or roadmap of adding the not-yet-implemented features. Namely, I am looking for two features to take kotlin scripts as a replacement of our Nashorn JS: the ability to run scripts from string (not from file) and most importantly to restrict access for library classes and functions (to avoid code injection attacks).

Is the "compile from string (or stream)" possible or will it be possible in the future? What is the internal priority of "restrictions"?

I know a developer don't like to drop in release dates of unfinished features, but I need to know whether it is months or years. (We have time to make the migration, but have to pick a new solution soon, java 8 is going to be unsupported in 2019.)

[NikkyAI]

Scripting support so far is awesome, good job to all parties involved

The Feature i think i need next would be import/include for extra source files without compiling them into a jar in a seperate step (because thats a hell of a lot of complication and things to go wrong) The files I want to include contain constants generated at setup by a gradle plugin/task in the project source, i am aiming for somwhat of a moving target so it cannot be compiled into the library one solution i found that works.. (for now) is to concat all the files i need into one big string and move imports to the top with regex needless to say this is not the prettiest code or easy to reason aboutor easy to track down compile errors since line numbers will not even match up anymore

and thats about all the complaints i have i guess.. anything else this api handles or will be bale to handle with ease it seems

[ligee]

@balage1551, the compiling from a string was supported from the beginning. The restrictions are planned, but the timeline is not yet clear.

[abelsromero]

What's the status of implemented features? I don't mind dealing with beta and things that may change but I am finding the documentation is not clear, and all the examples I have found don't seem to work. For instance, I am trying to run the example here https://blog.jetbrains.com/kotlin/2018/09/kotlin-1-3-rc-is-here-migrate-your-coroutines/ and the one from https://github.com/Kotlin/KEEP/blob/scripting/proposals/scripting-support.md

@file:DependsOn("junit:junit:4.11")

org.junit.Assert.assertTrue(true)

println("Hello, World!")

But the resolution simply does not seem to work.

[ligee]

@abelsromero there is a section on the status in the KEEP, it might be not 100% accurate, but you can get a picture. The example from the blog post definitely works, I checked it many times. Possible problems might be either you're not pulling the right script definition (e.g. do not add kotlin-main-kts.jar to the classpath), or the script "type" is not recognized (e.g. in the example from the blogpost, you have to give your script the extension ".main.kts", this is the extension by which kotlin-main-kts recognizes its scripts). Unfortunately, the diagnostics and proper tutorials and examples are lacking right now, we're working on it.

[dobrakmato]

Will this feature be supported on Kotlin/Native or it is planned to be JVM only?

[abelsromero]

Thanks @ligee, totally my fault for rushing to copy-pasting the examples without paying attention. I got it working now! Keep up the good work! Those stuck with Windows are looking forward to have a standard scripting tools.

[ligee]

@dobrakmato the API is designed to be platform-agnostic as far as possible, but current implementations are JVM-only. We're investigating possibilities to make it available in Kotlin/Native too, but it is too early for conclusions.

[SillyMoo]

Any tips on how to get intellij code completion working for scripts? I'm able to dynamically load files, and all works well, but code completion is not working (as intellij can't seem to resolve the base class from the @KotlinScript annotation. I get the impression it has something to do with the intellij kotlin compiler properties "Script template classes", but I can't figure out how.

[ligee]

@SillyMoo there are several possibilities to supply Intellij with everything needed for the resolution/completion/etc. The simplest one is, as you mentioned, kotlin compiler properties "Kotlin Scripting". You need to put the FQN of the base class into "Script Template Classes" field and the classpath needed to load this class into "Script templates classpath". These settings are intended to be more like workaround though. If your script is a part of the regular project sources, then you just need to add a properly prepared jar with the script definition into the project dependencies. And finally, it is possible to write a simple Intellij plugin that will supply needed info for your scripts. This is how e.g. "build.gradle.kts" files are supported now. We're also considering other methods of configuring Intellij for scripting, but so far haven't come up with good enough solutions.

[SillyMoo]

@ligee, so I have an uber jar with the @KotlinScript annotated script class in it, and also a file in /META-INF/kotlin/script/templates with the name consisting of the FQDN of the annotated script class. But intellij does not seem to recognise my script file extension (it does do so in the original project from in which the scripting class is generated). Is there any debug I can enable to see if there is a parsing issue or something?