Dynamic build commands

[Jermolene]

This PR extends the format for build targets in tiddlywiki.info files to allow command tokens to be generated dynamically. It currently supports wikification, filter evaluation, and prompting the user for a value.

The hope is that these features will make it possible to for --build commands to implement the kind of operations proposed in #8294 instead of using npm commands which have proved difficult to get working across all our supported platforms.

This PR also introduces a new --echo command to make it easier to test dynamic command tokens.

Here's a contrived example:

"build": {
    "dynamic": [
        "--echo","testing",
        "the following argument is wikified",
        {
            "type": "wikify",
            "text": "<<version>>-prod.html"
        },
        "the following argument is a filter result",
        {
            "type": "filter",
            "text": "[<version>!match[5.3.6-prerelease]then[text/html]else[text/plain]]"
        },
        "the following argument was provided by the user",
        {
            "type": "prompt",
            "prompt": "Please enter some text and type enter",
            "default": "Nothing"
        }
    ],
...

Here's the output it produces:

$ echo "demo" | tiddlywiki editions/tw5.com --build dynamic-example
Please enter some text and type enter (Nothing): demo
[
    "testing",
    "the following argument is wikified",
    "5.3.6-prerelease-prod.html",
    "the following argument is a filter result",
    "text/plain",
    "the following argument was provided by the user",
    "demo"
]

[github-actions[bot]]

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

[Jermolene]

There's a few ideas for further development that I have been thinking about:

• Support for prompts with numbered options
• Support for passwords so that the text typed by the user is not echoed to the terminal
• Support for executing external command line programs and piping their output. Useful for pulling defaults from the system. The security risks would be mitigated by the fact that build scripts can only be modified through direct physical access to the TW server

[pmario]

How can I access the value of the user input in following commands?

Eg. The user enters tw5.com -- How can I use to create a command like .editions/<user-input>-server --listen using the value from the user?

[Jermolene]

Eg. The user enters tw5.com -- How can I use to create a command like .editions/<user-input>-server --listen using the value from the user?

At the moment user input can only be passed as a complete, independent command token.

I have considered adding an optional filter that is used to transform the user input into the command token. You could then perform your example by setting that filter to [<user-input>addprefix[./editions/]addsuffix[-server]]

[pmario]

Support for executing external command line programs and piping their output. Useful for pulling defaults from the system. The security risks would be mitigated by the fact that build scripts can only be modified through direct physical access to the TW server

Think about the following situation: A future core or user published tiddlywiki.info file contains elements that can execute external executables. eg: cleanup, which is intended to delete several files before creating new ones with the TW build script. IMO that's a reasonable usecase for this type of functionality. right?

Let's assume I am a new user to TW. I do install TW globally. Then I do initialize the TW edition, that contains the tiddlywiki.info file that does execute "cleanup".

On my machine "cleanup" is an executable which can be invoked through path environment settings but it performs actions, that are completely unrelated to TW. But now they are triggered as a side-effect if I run tiddlywiki ./test --build

I think that is a problem, since TW --build triggers all actions that are configured under the "build" section. So cleanup may be executed unintentionally and it can cause damage, that may not be reparable.

Just some thoughts.

[Jermolene]

Think about the following situation: A future core or user published tiddlywiki.info file contains elements that can execute external executables. eg: cleanup, which is intended to delete several files before creating new ones with the TW build script. IMO that's a reasonable usecase for this type of functionality. right?

That would be an unsafe scenario for exactly the reasons you go onto outline. The problem is that there's a dependency on the command line tool "cleanup" without the distributed wiki being able to verify which program is actually being executed.

The sort of scenario I was thinking about was a script that wants to include the operating system username in a command. In other words, this would be for very customer specific hacks. These shell invocations wouldn't be cross platform, so we would never include them in the core.

I think that is a problem, since TW --build triggers all actions that are configured under the "build" section. So cleanup may be executed unintentionally and it can cause damage, that may not be reparable.

The fact that --build runs all the visible build actions is definitely a problem that I hadn't considered. Maybe we need to make the current structure be a shortcut for a more elaborate structure that allows us to add metadata:

"encrypted": {
        "tokens": [
            "--password", "password",
            "--render", "$:/core/save/all", "encrypted.html", "text/plain",
            "--clearpassword"],
        "mode": "interactive"
    },

My own position on this PR is that it is not yet clear whether the changes are sufficient to permit us to package useful end user scripts. My enthusiasm diminishes each time we need to introduce more complexity to support common scenarios but I still think it's an interesting area for exploration.

[pmario]

My own position on this PR is that it is not yet clear whether the changes are sufficient to permit us to package useful end user scripts. My enthusiasm diminishes each time we need to introduce more complexity to support common scenarios but I still think it's an interesting area for exploration.

That's right. I also was very optimistic about the "now closed" extensions to package.json, just to find out that I did miss an OS specific "placeholder", which only would be solvable by introducing a huge amount of node based dependencies. Which basically is a complete no-go from my point of view.

I think I can create "very minimalistic" and generic extension to package.json -- But it will only be of value for core devs. The advantage will be it should be simple :)

[pmario]

@Jermolene -- Would it be possible to extract the echo command into a new PR. I think we should definitely ship this one with v5.3.6. It would also make the package.json script section even simpler.

[Jermolene]

@Jermolene -- Would it be possible to extract the echo command into a new PR. I think we should definitely ship this one with v5.3.6. It would also make the package.json script section even simpler.

Yes I am happy to cherry pick the echo command, and possibly the terminal colour improvements.