executablebooks / mystmd

Command line tools for working with MyST Markdown.
https://mystmd.org/guide
MIT License
172 stars 50 forks source link

:computer: Add support for IPC-backed plugins #1329

Open agoose77 opened 2 weeks ago

agoose77 commented 2 weeks ago

This PR fixes #1298 in a different way to #1325 by adding a new "executable" type of plugin (the approach taken in https://gist.github.com/agoose77/66700b2a172b4c11d4801f742534e4f7). Unlike #1325, this PR adds support for Python via an IPC mechanism (stdout, stdin).

1325 and this PR are not mutually exclusive; #1325 adds support for an in-process Python executor, which supports sharing stateful proxies between JS and Python. This PR, meanwhile, is fully decoupled and allows for further languages. Note that, unlike #1325, we do not vendor Python, so it is possible that users may define plugins whose environment is not sufficiently reproducible and thus see build errors.

[!NOTE]

1325 will need to be rebased and updated to use this new mechanism for declaring plugin types.

To implement such a plugin, the executable needs to support the following invocation modes:

Right now, I didn't add the validate method. We could; it's not much extra work. But do we need it? An example of a directive and transform is given in the docs of this PR.

[!CAUTION] Syntax Changes from this PR in myst.yml:

```yaml project: plugins: - test.mjs ``` ```yaml project: plugins: - type: javascript path: test.mjs - type: executable path: python.py ```

N.b. this change is opt-in; existing plugin declarations will work just fine.

rowanc1 commented 1 week ago

Can we auto detect that in the transform and expand plugin.py to the dict?

e.g.

project:
  plugins:
    - test.mjs
    - plugin.py

Which is basically what you wrote, but slightly nicer UX / overhead for the user. :)

rowanc1 commented 1 week ago

Note that the way you wrote it should also work, as I see now the javascript could also be an executable.

agoose77 commented 1 week ago

@rowanc1 yes, I think the forward-looking preferred syntax would be the verbose kind. For single strings, we should only support known extensions but IPC plugins don't have a specific extension. If we added support for Python as in #1325, then we'd be able to write

  plugins:
    - foo.mjs
    - bar.py

Otherwise, we expand non-.mjs plugins to executable plugins, and then down the road this breaks when we add support for e.g. .py.