Feature Suggestion: Self-bootstrapping scripts

[EugeneSusla]

I was thinking if there was a way to make the scripts I write more standalone in a sense that I could share it with someone without either having to explain how to install kotlin&kscript or producing a binary they can't read/edit. Much like you can do today with shell/python scripts.

I believe I have found a fairly practical way to achieve this: by wrapping an existing script with a specific header and footer it can be made into a valid shell script and kotlin script at the same time. No unpacking step needed! See example below.

That means original script can be rewritten once, replacing it, and further iteration can happen with the bootsrapping footer in place. IntelliJ support is available for both kotlin and bash parts of the script with kscript --idea script.kts thanks to the intellij's language injection feature. Having the modifying happen only once also has the benefit that the shell code despite being auto-generated is ok to modify, as it won't be overwritten. See example below where I choose to propagate the script file name to kotlin.

So I was thinking, that it would be great to have this built into kscript on an "opt-in" basis, e.g. kscript --bootstrapify script.kts ^ prepends/appends things to script.kts making it shareable

P.S.: Happy to work on a PR if this seems ok to add

Example of self-bootstrapping script.kts: (try opening with kscript --idea to see how IDE experience is uncompromised; installing "BashSupport" pluging may be required for bash support)

#!/bin/bash
//usr/bin/env cat << '__EOF_KOTLIN' >/dev/null

import org.intellij.lang.annotations.Language

println("Hello from kotlin!")
if (System.console() == null) {
    print(generateSequence(::readLine).joinToString("\n") + " | ")
}
println("${System.getenv("KSCRIPT_FILE")} ${args.toList().joinToString(" ")}")

//DEPS org.jetbrains:annotations-java5:16.0.2
@Language("sh")
private val __SHELL_BOOTSTRAP = """
__EOF_KOTLIN
    function in_path() { command -v "$1" >/dev/null 2>&1; }
    in_path kscript || {
        function echo_and_eval() { echo "$ $@" 1>&2; eval "$@" 1>&2; }
        in_path sdk || {
            curl "https://get.sdkman.io" | bash 1>&2 && \
                echo_and_eval source "$"SDKMAN_DIR/bin/sdkman-init.sh
        }
        in_path kotlin || echo_and_eval sdk install kotlin
        in_path gradle || echo_and_eval sdk install gradle
        echo_and_eval sdk install kscript
    }
    export KSCRIPT_FILE="$0";
    kscript $0 "$@";
    exit $?
"""

[holgerbrandl]

That means original script can be rewritten once, replacing it, and further iteration can happen with the bootstrapping footer in place.

That's a very clever way of doing it. :-)

Much like you can do today with shell/python scripts.

Just works for python, because Linux OS vendors often choose to bundle the python runtime because it is popular. Fingers crossed that kotlin will be honored in the same way some day. :-)

Just in case you missed it: There is --package already, which allows to share scripts in a similar way. After --package you do not even need gradle or kotlin anymore. Clearly being able to edit the shared solution like in your suggestion is a big plus.

Although the rewritten scripts are legit kscripts, the kscripty nature is feels a bit hidden to me when looking at your example. That is it took me almost a minute to figure the used interpreter.

On a technical level, I think your suggestion does not do process substitution, whereas kscript does so. But maybe by doing exec kscript $0 $* at the we could fix this.

I also wonder about stdin handling. Can we still pipe into these rewritten script? Like in cat foo.txt | kscript bar.kts >> result.txt

Anyway, the suggestion is great and I think we should add it.

For sake of conciseness I'd think that we may want to leave out

//DEPS org.jetbrains:annotations-java5:16.0.2
@Language("sh")

since it should work without.

Also, the bootstrapping block feels a bit long and is not really interesting to the user, so we could potentially try to shorten it to something like:

private val __KSCRIPT_BOOTSTRAP = """
__EOF_KOTLIN
    function in_path() { command -v "$1" >/dev/null 2>&1; }
    in_path kscript || source <(curl https://shortened.url/install_kscript.sh 2>&1 2>/dev/null)
    exec kscript $0 "$@";
"""

So essentially we could/would move the bootstrap part to a script in the repo. What do you think? Maybe we could even inline in_path

It would be great if you provide a PR including

1. the new option

2. during rewrite

   • stripping shebangs if present
   • add the footer
   • prevent that --bootstrapifying twice creates a corrupted script

3. regression tests to ensure that kscript script.kts and ./script.kts are supported (which seems to be the case already) and that stream redirection still works with bootstrappified scripts

4. Readme update (I guess it should go into the package section).

I know this is a lot, but I'm also happy to help if needed.

I'm not sure about Windows support. A strappified script would not longer run under windows shell (see #194) whereas a old-school kscripts will after #194 has beed merged. But since kscript original and main intention is to support bash, I think we can live with this limitation.

Final question, could there be a more descriptive name for the process and the option instead of --bootstrapify? I have no sound idea yet, but bootstrapify may be to obscure for some/most users.

[EugeneSusla]

Just in case you missed it: There is --package already

Yep, knew about it. Btw, just tried it and got an error, will post separate bug with output.

Can we still pipe into these rewritten script? Like in cat foo.txt | kscript bar.kts >> result.txt

Yes! Works with the original example too (you'll see it echoed back), just need to save it to a file first

may want to leave out [language injection]

feels a bit hidden

I have a previous iteration of this that happens to trade language injection for having all shell in header (no footer). Sounds like it may be better then - see example below

curl https://shortened.url

Great idea!

A strappified script would not longer run under windows shell (see #194) whereas a old-school kscripts will

Hmm, will it not? I thought windows will just look at the extension(and ignore shebang) and send it directly to kscript, which will just treat is as valid kotlin script. Looking forward to #194 to actually try it out!

Anyway, the suggestion is great and I think we should add it.

provide a PR including

Awesome! I'll get started on a PR with these then! (thanks for enumerating the corner cases/potential regressions by the way!)

more descriptive name for the process and the option instead of --bootstrapify

--embed-bootstrap perhaps? I was struggling with the name too as you can tell :)

leave out [Language]

One caveat worth mentioning with any approach that doesn't add any imports, is that intellij will try to put first import in-between shebang and second line, breaking the validity of shell script. Easiest workaround I could think of is to add dummy import kotlin.Unit as part of the header - if there's already at least one import, IntelliJ will place new imports next to it instead. I guess in a refined version I could inspect if script already has imports and only add dummy one if it doesn't

Example 2: (note that intellij allows folding bootstrap lines, which looks nicer. I wish there was a way to programmatically trigger that particular fold...)

#!/bin/bash
//usr/bin/env echo '
/*********** Bootstrap *************************\'>/dev/null
 command -v kscript >/dev/null 2>&1 || curl "https://gist.githubusercontent.com/EugeneSusla/910750a9eb5f9cd77ee5765650c223e3/raw/9400952357600c66303c69b3cc7ceb59f396b10a/get-kscript.sh" | bash 1>&2
 export KSCRIPT_FILE="$0"; #optional

 exec kscript $0 "$@"; exit $?
\*********** End Bootstrap *********************/

import kotlin.Unit

println("Hello from kotlin!")
if (System.console() == null) {
    print(generateSequence(::readLine).joinToString("\n") + " | ")
}
println("${System.getenv("KSCRIPT_FILE")} ${args.toList()}")

[holgerbrandl]

A strappified script would not longer run under windows shell (see #194) whereas a old-school kscripts will.

Hmm, will it not?

afaik there's neither support for shebang nor bash in the windows shell. So, no it won't.

One caveat worth mentioning with any approach that doesn't add any imports, is that intellij will try to put first import in-between shebang and second line, breaking the validity of shell script.

Indeed, but it's imho a minor limitation which we could just document in the boostrap section (see my revised version below).

Easiest workaround I could think of is to add dummy import kotlin.Unit as part of the header

IJ will optimize it away quickly, so I don't think that this would help.

Example 2: I clearly prefer the example2 way because the boostrap comes first and as a single block (like a preamble) which has a more consistent flow and logically more intriguing. It can be simplified even further

#!/bin/bash

//usr/bin/env echo ' / BOOTSTRAP kscript - DO NOT TOUCH! \'>/dev/null command -v kscript >/dev/null 2>&1 || curl "https://git.io/fpVLG" | bash 1>&2 exec kscript $0 "$@" *** IMPORTANT: Any code including imports and annotations must come after this line ***/

//DEPS com.github.holgerbrandl:kscript-support:1.2.4

import kscript.text.print import kscript.text.resolveArgFile import kotlin.system.exitProcess

println(args.toString()) resolveArgFile(args).filter{it.contains("YYTIPAPSLLGIQDFVLYQK")}.print()

exitProcess(1)

There's no need for `echo $?` in the header because exec substitutes the process. Just try the example with smthg like `cat text.txt | ./script2.kts -` followed by `echo $?` 

I also ditched `KSCRIPT_FILE` because we do no support it elsewhere in `kscript`.  If at all it could/should be a different ticket/PR.

> --embed-bootstrap perhaps? I was struggling with the name too as you can tell :)

or maybe `--add-boostrap-header`? Should it write the modified script to stdout or do an inplace file replacement (similar to `sed -i`)? The former approach would be easier to grasp, but the latter more convenient I guess.

>> Can we still pipe into these rewritten script? Like in cat foo.txt | kscript bar.kts >> result.txt

> Yes! Works with the original example too (you'll see it echoed back), just need to save it to a file first

IMHO there is no need to save stdin to a file first, just checkout my example from above. Which is good, because this allows to stream data also through a boostrappified kscript.

> I wish there was a way to programmatically trigger that particular fold...)

You can fold block comments in general when using IDEA but I guess that's not intended in most situations.

[EugeneSusla]

It's nice to see it that short! Will use the latest one as the template when implementing then

DO NOT TOUCH!

I was thinking it should maybe say the opposite (or just don't say either way), as we'll never overwrite custom changes there. There are some things that can be only done in bash, capturing script name being one example.

Should it write the modified script to stdout or do an inplace file replacement?

I think in-place replacement is the only right way - we should encourage people to not duplicate their script to a new file as much as we can. Without clear guidance I can imagine thinking this is basically just another flavor of --package and redirecting output to a new file is expected of me.

--add-boostrap-header

Seems pretty clear, if a bit long. I'll go with it then

no need to save stdin to a file first

yeah, meant to say the script itself, not stdin :) we're on the same page here

[EugeneSusla]

Closing as the implementation is now merged