Document providing rendering context to plugin directives via transforms

[sneakers-the-rat]

Proposal

I am using myst to write a paper asynchronously with a few people, and during the drafting phase it would be nice to be able to write a mention directive where we can make notes that mention another author - e.g. i'm writing a draft section, and i want to ask another author to check me on something, or to assign some section to someone else. Then I would like to make a second directive like a mentionsList where I can render all the mentions in one place.

This is reminiscent of the canonical "Developing a TODO extension" for sphinx.

This is also a general pattern, where one might want to have a directive that can be aware of and use nodes that are outside of its argument and body, or in another document entirely. Searching i found this prior issue re: bloglike/gallery behavior, and one could see how it might be nice to keep bloglike functionality in-universe with existing myst concepts by just having a blogpost directive that can render a list of posts by having access to the render context: https://github.com/jupyter-book/mystmd/issues/840

other examples include indexes, backlinks/wikilinks/etc.

Some approaches

this might be trickier than it seems, i have not yet taken a deep look at the myst rendering process and am raising this first as a feasibility check.

There's the basic issue of timing - when does the directive get called. it would probably be massively overcomplicating if one had to pause the render, leave a placeholder, render everything else, and then at the end render all custom directives. then there's perf, where needing to copy and pass an entire humongous document through standard streams, parse, etc. when doing any minor incremental build of a single page. and so on

But some possible approaches, lmk which of these might be appealing, if any, and i'd be happy to collaborate on a patch.

Build Phase Hooks - this is probably the most complex to get right, but most flexible approach. a cheap and easy version would be just to add a hook method between the tokenizing phase before the directives are applied where a plugin could modify the tree and inject the information it needed into the unprocessed directive. that would be probably pretty tricky to write and error prone without a lot of helpers, but it would be extremely general.

Pass the whole AST to a directive that requests it in its spec - easy but expensive, especially with larger multipage documents.

Dependencies between directives - it could be possible to just declare that one directive must be applied after all the directives of another type/name have been processed. Then you could pass only those directives that the dependent directive has requested to it as context. This would be intuitive to write, probably more performant, but a little fragile because as far as i'm aware there isn't any namespacing for directives, which would be a problem if someone was using a bunch of 3rd party plugins that weren't mutually deconflicted. It would be relatively easy to implement, just topo sort the directive nodes by dependencies.

Events/listeners - Another case aside from a big index directive might be one that wants to be dependent on the value from a neighboring directive. So that same dependencies idea from above at a type level could be applied at the instance level, where a directive could declare itself an emitter of an event type, and another could declare itself a listener for an event type. So in the case of global indexing-style directives, you just have a single key, but then you have a superset of behavior where pairs/groups of heterogeneous directives can depend on one another and reuse values. This would probably be on the more complex side of things to implement, but it would be ~ cool ~ and probably ~ overengineered ~.

Phased directives - this is probably way less desirable, but instead of build phase hooks one could also imagine having an enum in directives that allow them to declare that they should be rendered at a particular point in the build phase. just listing options, i don't think this is a particularly good one.

Additional notes

Anyway just sketching, hopefully i made the distinction between the actual pitch/need and the potential ideas part clear enough. "i want to be able to have context-aware directives and there are a bunch of different ways that could happen and i want to help out with the one you like" is what i'm going for here, so lmk ttyl <3

[rowanc1]

There is something that is pretty close to your use case here:

https://github.com/myst-ext/myst-ext-points/blob/main/extension.mjs

This defines a points role and pointReport directive, and then a transform to make sure they are in sync. The initial parsing is completed by directives and roles, and they leave a specific type (or subtype) of a node that can be transformed later in the process. The transform comes in later (following the "pass the whole ast and phased directives approaches you mention) and actually does the addition and final display. Hopefully that makes sense - at transform time, you do have access to the whole AST, and also things can happen async, whereas the directive/roles parsing is constrained to be both synchronous, fast and local.

I think you lay out a lot of the tradeoffs we are thinking about as we make these transformation chains a bit more complex, as well as user friendly. However, what you are talking about is possible today, at least in the context of a single document.

[sneakers-the-rat]

oh that rocks!!!! I was wondering how to do this with a transform but was thinking about it in terms of passing values to the directive which rendered it, rather than doing the rendering in the transform and the directive is just a placeholder node.

I'll test this out, my initial question is whether or not i get the context of the whole project (like all the different pages in a multipage book) or just the current page, this comment is making me think i might need to do something else, but rather than taking your time asking you to tell me i'll just try it out ;)

since this came up in terms of making a blog index before, and i imagine it may come up again, would it be helpful for me to draft a docs PR with your example? Reading the current docs now, the pattern seems obvious in retrospect, but it might be helpful to have as another worked example on how to combine custom directives, roles, and transforms, and that one is a good one (and also looks like fun! i wish my teachers had a sweet document-driven points system)

[rowanc1]

Currently the transforms are document specific, certainly something to change in the future:

• https://github.com/jupyter-book/mystmd/issues/1616

We would absolutely LOVE help with the docs and examples to make this clearer. Maybe if you are turning this into a mini-tutorial that is sharable and we can help you out along the way that would be fantastic. :)