Support Markdown as an alternative to Sanity docs for Reference/Ideas

[microbit-matt-hillsdon]

Description

We'd like to explore an alternative to the (commercial) Sanity CMS suitable for forks of the editor and other usecases such as extension modules providing their own documentation.

This issue is a placeholder for the moment to track requirements gathering and interest.

CC @felixerdy after discussion on #1148.

Requirements/challenges

Assumptions

Input very welcome.

1. Developers are trying to either (1) replace the Reference section functionality with alternative content OR (2) augment it with additional content suitable for extra MicroPython C or Python modules.
   • Other possibilities not catered for here could include adding additional tabs of content with similar structure to Reference. Is this useful? The tabbed UI has space issues that might be problematic.
2. Markdown is a useful format for this

Content model outline and possible markdown expression

toolkit

• name (unused)
• slug (used only to identify the one reference toolkit that we do display)
• contents (used to define the ordering of the topics in Reference, this is curated and alphabetical wouldn't be useful to us)

The toolkit type is residual from a stage where we had more than one toolkit corresponding to separate tabs in the sidebar. It could be replaced with any way to capture the ordering of Reference topics, e.g. a markdown list of links or an auxiliary JSON structure.

toolkit topic

• name
• slug
• compatible products
• image
• subtitle
• introduction (rich text)
• contents (ordered list of topic entries)

Each topic has a fair amount of metadata. They could be individual markdown documents with front matter. The introduction is rich text and we use at least links and images in practice.

topic entry

• name
• slug
• compatible products
• content (rich text)
• label for a dropdown of alternative content (optional)
• alternative content sections, each with a name for the dropdown and rich text
• detailed content (rich text)

We display the main content field's content then the current alternative content (with a dropdown to choose defaulting to the first entry) then detailed content. Many sections don't have alternative or detailed content.

We use some progressive disclosure in the UI. Initially we truncate content before the code example to be one line and show the code example and no further content. Entries without a code example are always shown in full. Clicking "More" expands the content.

This structure might be challenging to model in markdown. As we're unlikely to convert the existing content to markdown it's worth discussing whether this feature set is needed for alternative editor content. We'd welcome input from folks considering using a markdown content feature.

The entries themselves could be individual markdown files referenced from the topic (with the ordering problem as per toolkit). Or perhaps they could be sections in a single markdown document that covers the entire topic. This would require us to be able to split the document for rendering and would need an approach to per section "front matter" which doesn't seem to have a conventional solution for markdown.

pythonIdea

• name
• slug
• compatible products
• card image
• content (rich text), code embed is treated specially in these files

These are easy compared to the above.

pythonIdeasConfig

This is a singleton document that stores the display order of the ideas.

This could use the same solution as for the toolkit ordering.

rich text support

• h3 heading
• normal text
• strong
• code (inline so like this in markdown),
• links to API terms
• internal links to other toolkit entries
• external links
• embedded python code
• embedded image with alt text

This is all easy to model in markdown except internal links to toolkit entries which would need some specially recognised link format, e.g. [whatever](reference/{topic-slug}/{entry-slug}).

python module

• name
• array of links to topics or topic entries from API names
  • links to topic entries can link to a specific piece of alternative content

We likely need to keep this as a separate mapping as we need this information in the API -> Reference direction so we can show at most one carefully chosen link to Reference for each API entry (these are used in autocomplete help).

We could use a JSON file with key-value pairs of api calls to reference documentation ids/slugs, prepopulated with each api entry pointing to null to be customised.

Translation

An additional complication is support for translation. At the Foundation we translate content via an integration between Sanity CMS and the Crowdin platform. For Markdown I think we'd just define a folder/path structure using language codes and the actual translation would be out of scope.

Search

The client-side search feature might need updating depending on how we modelled the data. Some changes will be needed to index the markdown instead of portable text.

Existing markdown use and possible new dependencies

We currently use marked for rendering markdown returned from Pyright.

We should experiment with using its tokenizer API to split markdown documents, read front matter and control link/image rendering. It's not clear without trying whether it offers enough control so we might need to switch to something else. We might need something react-aware so we can render our custom code blocks.

Example

A potential example of a document per topic with per entry metadata using a custom (and likely to change!) format. The other document types don't have examples for now.

If we can't find a markdown parser that lets us easily split the file then we'd likely be pushed towards a file per topic entry instead.

---
name: Variables
subtitle: Keep track of data that changes
image: ./variables-icon.svg
slug: variables
compatibility: microbitV1,microbitV2
language: en
---

Variables are used to store data that may change, such as the number of points you have in a game. Variables can have different [data types](/reference/data-types).

### Assigning number variables
@slug assigning-number-variables
@compatibility microbitV1,microbitV2

Set variables to their initial values at the start of a program using the = sign:
```python
count = 0

Assigning strings

@slug assigning-strings @compatibility microbitV1,microbitV2

A group of words or symbols is called a string. Put quotation marks around the contents of a string:

from microbit import *

name = 'micro:bit'
display.scroll(name)

You can use single ' or double quotation marks " around strings, but you should be consistent. We use single quotation marks in examples because they are easier to type and double quotation marks can be confused for two single quotation marks.

See String manipulation for examples of how you can manipulate strings.

# Conclusion

<!-- To be completed when the task is done. -->
<!-- A very short and concise summary of anything not obvious. -->

[microbit-matt-hillsdon]

@felixerdy we've added some notes on the challenges of a markdown version of the docs and would welcome any input if this is still something that interests you.