Improvements to support `parts` for exports

[fwkoch]

Individual files allow parts to be defined in their frontmatter. These are respected for static exports of those files. However, parts cannot be defined at the project/export level. We should extend the frontmatter parts to be allowed there:

...
project:
  exports:
    - format: pdf
      abstract: My Custom Export Abstract

Additionally, it would be nice to allow these to be paths to files rather than inline text.

[rowanc1]

Very excited about this, especially around enabling parts to be defined by files in general.

[fwkoch]

We should make sure this works for site templates, including live reload if the part is a file path.

[fwkoch]

Ok - just hashing through things a bit to make sure I understand (1) the current state of things and (2) how to proceed...

How do parts currently work?

• parts may be defined in four ways:
  1. project frontmatter parts
  2. page frontmatter parts
  3. directly in a page, under a tagged block part
  4. directly in a page, under a section with the same name as the part (these are implicit parts)
• During processing, this is simplified:
  1. project frontmatter parts are simply ignored currently
  2. page frontmatter parts become tagged blocks and are prepended to the page content. These blocks are marked as visibility: remove - the transform to do this is basically the first transform during mdast processing: https://github.com/jupyter-book/mystmd/blob/main/packages/myst-cli/src/process/mdast.ts#L170
  3. existing tagged blocks are unchanged
  4. existing implicit part sections are unchanged
• For static exports:
  • the export template.yml defines parts
  • these are then extracted from the page using extractParts - this pulls out tagged blocks (which now include page frontmatter parts) as well as implicit parts.
  • extracted parts are then available to the template
  • page frontmatter parts that were not extracted do not show up, due to their visibility; tagged blocks and implicit parts defined in the page that were not extracted do show up, though.
  • for multi-page exports, parts are extracted across all pages, and if there are duplicates, they error (so, for example, abstract must be defined on one-page-and-only-one-page of the export)
  • an individual export cannot set it's own export-specific parts
• For site build:
  • the site theme template.yml could define parts, but it currently does not
  • instead, known parts are always extracted in myst-theme on a per-page basis. This uses the same extractParts function as the static exports, but only applies to the hard-coded known parts: https://github.com/jupyter-book/myst-theme/blob/50c625864f091f52dc563cd3bd486e68de3625d1/packages/site/src/utils.ts#L21-L28
  • there are no project-level parts.
  • behaviour for other parts is the same as static exports: project frontmatter parts are ignored, page frontmatter parts are removed, tagged blocks remain.

What are the next steps

• [ ] Allow frontmatter part to be a file
  • During processing, this is treated the same as other files
  • This file must be excluded from implicit toc generation
  • The file needs to be (1) added as a page dependency and (2) watched
  • The resulting part mdast should inserted into the frontmatter when serializing to JSON
• [ ] Modify behaviour of in-line frontmatter parts be processed as a "pseudo-file" rather than injected into the page tree
  • Question: do we also still inject this into the page tree? If not, and if the part is only available in the frontmatter, this becomes a breaking change to myst-theme, i.e. we will need to pass frontmatter into these: https://github.com/jupyter-book/myst-theme/blob/50c625864f091f52dc563cd3bd486e68de3625d1/packages/site/src/utils.ts#L22 - not a big deal I suppose!
• [ ] Allow site to define parts
  • These must also be parsed and added to config JSON
• [ ] Allow exports to define parts
  • These must also be parsed and added to frontmatter/config JSON
• [ ] Update static export templating logic and extractPart for new part definition locations
  • For single-page and multi-page exports, prioritize export parts and fall back to existing logic
• [ ] Update site templating logic to respect parts in template.yml
  • These are project-level parts only (i.e. this does not replace the hard-coded, per-page parts that the site theme always extracts)

What isn't covered in the above scope

• Actual support for project frontmatter parts
  • This should be very straightforward after implementing site parts, as they are both parts at the myst.yml/config.json level
  • This will also impact exports and site builds:
  • Multi-page exports should prioritize project frontmatter parts over the existing page frontmatter part logic.
  • Site builds and single-page exports should likely fall back to project frontmatter parts if site-level and page-level parts, respectively, are not defined.
• There's some extra complexity for site builds and multi-page exports where parts may either be (1) at the project level (so for each build there must only be one) or (2) at the page level (so for each page in the build there must be one - these are like the hard-coded parts supported in the site theme)
  • This might require template.yml#parts to specify kind: page / project or something
  • This also is probably tied into an overhaul of multi-page exports, as currently multi-page basically compile all the pages into a single "page" then perform the export (there is no place for per-page parts or options)