PROPOSAL: Bootstrap governance and simplified MyST EP process

[choldgraf]

This is a proposal for a bootstrap governance model for the executablebooks organization, as well as a simplified MyST Enhancement Proposals model. Its goal is to give us the basic building blocks of explicit governance and decision making while being as simple as possible, in the hopes that we increase the likelihood that we can follow these processes.

It defines two different sources of truth.

• One for organizational policy, which would now live at a new repository called team-compass. We make changes to this repository via issues and PRs.
• One for the MyST specification, which lives at myst-spec/. We approve changes to this spec via a process carried out in myst-eps. This takes strong inspiration from the MyST EPs proposal that @chrisjsewell put together in this HackMD and slims it down / restructures considerably in the hopes that it is more realistic for our (relatively) small team to follow.

In addition, it defines two decision-making groups:

• A steering council, that has decision-making power over organizational policy. Currently comprised of the three Principle Investigators of the original grant that funded this work.
• A core team, that has decision-making power over MEPs (and most decisions in most repositories) as delegated to by the Steering Council. This is roughly the @executablebooks/ebpteam (the folks defined on the team page)

My goal for this PR is just to get us on the same page, and to give us the basic building blocks we can use to start defining formal policy, strategy, etc.

My plan is to:

• Wait a bit for feedback, and incorporate ideas into the proposal
• Ask for approval from @gregcaporaso and @jstac
• Once I have approval, I'll merge this in, and then:
• Create a team-compass/ repository
• Move over a subset of information to encode there (e.g. team structure).
• We can then begin to iterate on more policy changes from there.
• Simultaneously, we will need to make the necessary changes to myst-spec and myst-eps to encode this process in those repositories as well.

[chrisjsewell]

Thanks @choldgraf this looks great, I'll have a proper look shortly 🙏

[rowanc1]

Another point to consider/document is the license of the MEPs, in @chrisjsewell's draft (I think based on PEPs?) I quite liked:

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

I am not sure if that needs to be in each MEP, or is included in the license of the repository, but I think it is important to document/include in some respect. (CC-BY-4.0 could also be used potentially, and I am not sure if it would have too many differences/downsides in this context).

[chrisjsewell]

So the team-compass sounds great, I'll leave that to you guys

On the extension-proposal side, my main questions are:

1. As opposed to the current draft where there is three types of MEP: Standards Track, Informational, and Process, this only mentions the "Standards Track" one. Is it still the intention to have the three? Obviously, this actual document, of the MEP process would be a process one, or is intended that this sits outside the actual myst-eps (in the team-compass?). Their probably won't be so many Informational ones, but what I had envisaged (e.g. in https://github.com/executablebooks/myst-eps/pull/5) was having at least one with "core design principles/guidelines" that could/would be referenced in Standard MEPs, e.g. "does this syntax introduce multiple ways of doing the same thing", "does this syntax break commonmark".

2. The current terminology seems quite focussed on changes to current syntax. What about additions of syntax or (very likely) additions of new directives and roles? I could even think of extending this to proposing e.g. changes/additions to the options of directives, is this within the scope of a MEP?

3. If the MEP is accepted, update its status metadata to Accepted and merge the PR. Once a PR is merged, it closes the issue and a decision has been made.

What does it mean that the decision is made, and how does it relate to actual implementations?

Most syntaxes are tied to reference implementations, e.g.:

• https://peps.python.org/pep-0001/#pep-review-resolution: Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and incorporated into the main source code repository, the status will be changed to “Final”.

• https://jupyter.org/enhancement-proposals/jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.html: If in the course of implementation, it is discovered that the implementation needs to be radically different from what was defined in the original JEP, then a pull request is submitted to modify the original JEP with the new necessary implementation, and a note citing the need for a modification to the JEP. This pull request should be re-approved by the original review team.

• https://github.com/commonmark/commonmark-spec: CommonMark is a rationalized version of Markdown syntax, with a spec and BSD-licensed reference implementations in C and JavaScript.

Should there be a defined reference implementation for myst-spec, and should it be mystjs? and/or should there be an extra step in the MEP process, to allow for the implementation(s) and feedback from it? What if a certain implementer, finds they cannot implement it fully or at all? (I could certainly envisage this with myst-parser, and its need to adhere with sphinx/docutils)

Here this makes me think of the web spec and https://caniuse.com, in that perhaps its ok that implementers don't implement all syntax (particularly all directives/roles), but then there is a way for them to provide a concise "manifest" of what they do support.

---

Perhaps this thought is a bit too technical for here:

Additionally, let's say we are targetting output formats HTML, PDF and Word. Does the MEP have to discuss how the syntax can be rendered in each format? Here I am thinking of thing like tabs, already implemented in both https://sphinx-design.readthedocs.io/en/furo-theme/tabs.html and https://js.myst.tools/guide/dropdowns-cards-and-tabs#tabs would we envisage something like this becoming part of the myst-spec and, if so, add it in a MEP with guidance on what the behaviour should be for different

[choldgraf]

Hey all thanks for comments - some quick responses here:

Another point to consider/document is the license of the MEPs

I'm happy with whatever license people think makes sense. My feeling is that we should use the most permissive license possible, with attribution. Let's decide this at the implementation stage?

As opposed to the current draft where there is three types of MEP...Is it still the intention to have the three?

How about for now we avoid formally categorizing each MEP, and leave that to the MEP authors to explain what the next steps would be in the "implications / migration" section of their document? If we feel that we are losing something important by not formally having multiple categories, we can add them in the future. My goal in this document is "the simplest possible system that might possibly function".

How about we add a section like this to the documentation:

## When should I open a MEP?

The goal of Enhancement Proposals are to align the team on major strategic decisions about MyST Markdown, and to formally record a decision. When deciding whether to propose a MEP, consider whether the importance or complexity of the topic is worth the extra overhead of the MEP process. Ultimately, the most important thing is that we follow principles of open and inclusive discussion, iteration and collaborative writing, and making decisions explicit.

As a guide, below are examples of topics that warrant a MEP:

- Changing or extending the syntax or major functionality of MyST.
- Defining high-level strategy and vision for the language
- Amending the MEP process itself

What does it mean that the decision is made, and how does it relate to actual implementations?

I think it means "the team has decided that the proposed change is a good idea, a pull request to the myst-spec repository (or any implementations) will be non-controversial, and PR change requests should be restricted to implementation and not design".

Should there be a defined reference implementation for myst-spec, and should it be mystjs?

Good question - I don't feel qualified to answer and feel like the Core Team should decide on what is the right "source of truth" for the specification. If people on the Core Team can quickly align on the right path forward, let's just go with that. If not, then let's start off by saying that myst-spec needs to be a descriptive source of truth, and we can leave it to a future decision if the lack of implementation there creates too much tension or sluggishness.

Does the MEP have to discuss how the syntax can be rendered in each format?

I'm not sure - but I think we should prefer under-designing rather than over-designing this process for now. Let's leave this question to a future iteration on the process once we have some experience and learn more where it is deficient?

[chrisjsewell]

a pull request to the myst-spec repository (or any implementations) will be non-controversial if the lack of implementation there creates too much tension or sluggishness

Yeh this is obviously the point I'm getting at, particularly from the view-point of myst-parser (or indeed stakeholders for any implementation): If I see an MEP where my feeling is "I like the syntax, but oof this could be a nightmare to implement (particularly within the sphinx framework)", but I won't be clear until an actual implementation has been tried, do I:

1. Say no, this MEP cannot be merged until there is a PR for an implementation in myst-parser (which could possibly take a while)
2. Say yes, this MEP can be merged (and the spec changed), with the caveat that it can be modified or even revoked after an implementation is tried in myst-parser. During this intermediate time, then, myst-parser and thus jupyter-book would be non-spec compliant.
3. Say yes, this MEP can be merged (and the spec changed), and if ultimately it cannot be implemented that is "ok" and myst-parser/jupyter-book are simply non-spec compliant in this area

I feel this situation could arise very quickly, and I'm very keen to have clarity here from the team from the outset

[choldgraf]

Why not use something similar to your proposal above:

Here this makes me think of the web spec and caniuse.com, in that perhaps its ok that implementers don't implement all syntax (particularly all directives/roles), but then there is a way for them to provide a concise "manifest" of what they do support.

I'd hope the Core Team could raise and resolve most concerns about implementation feasibility at the design level before the MEP proposal is accepted. But if we miss something that is only uncovered later at implementation, the person could:

1. Create a section of their documentation that lists specific ways in which their implementation deviates from the MyST spec, but otherwise adheres to spec version X.x.
   • This would let implementors have a little bit of wiggle room, provided they weren't deficient in major functionality.
   • We should also make it clear in the spec repository (via a design / strategy doc like the one y'all are working on) that MyST should make choices that are not overfit to a specific implementation.
2. If it's really bad, create a new MEP to propose revoking it or modifying it to be more feasible. If there's a delay between when MEPs are accepted and when a new "version" of the MyST spec is released, there might also be some time to modify as-needed once implementation has happened. Ideally when a new version of the MyST spec is released, many implementations have already tried making the necessary changes.

[rowanc1]

Should there be a defined reference implementation.

My thoughts are to under-specify this to start with, and either define it in the spec documents (supported by x-parsers) or specified in each parsers documentation (which might be easier). Right now myst-spec is very AST heavy, and not overly complete at specifying the actual markup syntax (there are examples, but not overly complete/comprehensive). I hope that will change/improve over the coming months, and I think this will put it on much more equal footing for myst-parser. As we have also developed out mystjs, the spec being defined at parse-stage only is also missing some obvious pieces that are necessary as transformations come into play (and could be optional type suggestions for the actual spec).

I'd hope the Core Team could raise and resolve most concerns about implementation feasibility at the design level

To @chrisjsewell your point, I think that folks on the core team can likely identify that there is uncertainty pretty early on, and that might put the MEP on hold while we gather more info/try an implementation/think through it etc. I do think this process has the most value when coupled to direct improvements in various parsers and end-user impacts. I am partial to the under-specify the process now, and improve the process if necessary when we come across those situations.

When should I open a MEP?

@choldgraf, +1 from me on adding more concrete examples to the process, I think this grounds things and sets some expectations!

core design principles/guidelines as a MEP

Regarding these documents, I think it may be difficult to (1) define the scope of these -- leading to ambiguity on "when is it done" and "can it be merged?" and (2) we may want to remove/edit/revise/add design guidelines and vision over time, and having more agility/flexibility than the MEP process is advantageous. In my opinion those documents and ideas are important to contribute to and define now, and I think the best place is to refine them in the team-compass at this stage. I don't think that space is necessarily the home forever, and perhaps after we have the first few MEPs under our belts, as well as a draft in team compass, we will be more confident that some of these design guidelines can be increasingly formalized and we incorporate them in a MEP. I think this is actually functionally the same, but kicks parts of the formal process down the road and allows for a simpler process out of the gate.

[fwkoch]

This looks great - I'm also in favor of a relatively light process around MEPs for now.

Especially thinking about the current state of myst-spec, there is quite a bit missing: basic features already used in the myst-cli and therefore present on the myst.tools doc pages (tab groups, citations, etc.), formalization around html/word/latex outputs (or explicitly deciding against including these in the spec). I suspect the first handful of MEPs addressing these shortcomings will be mostly non-controversial and will already have corresponding implementations. Contrast that to future proposals which may have less prior work, need more discussion, etc. Nice to not have to dictate a process that works with all of these start.

[chrisjsewell]

I think that folks on the core team can likely identify that there is uncertainty pretty early on, and that might put the MEP on hold I am partial to the under-specify the process now, and improve the process if necessary when we come across those situations.

Yep thanks @choldgraf @rowanc1, indeed I don't want to hold up the initial process merge here, it was principally to get a feel for how "blocking" should a lack of a particular implementation should be.

[choldgraf]

I believe that I've addressed the comments above, and have also left a few comments and empty areas in-line with the markdown to show where we need to fill in information in future iterations.

I split the document into three pages to make it easier to parse, but the content hasn't meaningfully changed beyond the suggestions people gave.

I request approval from @jstac and @gregcaporaso, and once given I'll start creating a team-compass repository. I've encoded those next steps in an issue here:

• https://github.com/executablebooks/meta/issues/846

[choldgraf]

OK I updated the text w/ the minor comments that remained. I see 2 approvals from the @executablebooks/steering-council (I just created that group by the way) so will merge in!

[gregcaporaso]

Thanks @choldgraf for leading this effort, and thanks everyone for the input!