Support Mermaid diagrams at build-time

[dtuite]

Mermaid is a diagramming and visualization library similar to graphviz. It has an MkDocs plugin.

Some companies who are adopting TechDocs will already have large numbers of Mermaid diagrams in existing MkDocs documents, and would like them to work in TechDocs, without forking the mkdocs-techdocs-core and maintaining new TechDocs containers which use the modified core.

Would it be agreeable to add this plugin to TechDocs? We would, of course, do the work to implement this.

[dtuite]

Related (or perhaps even a dupe): https://github.com/backstage/backstage/issues/4123

[emmaindal]

Hi @dtuite thanks for opening this issue. Looking at https://github.com/backstage/backstage/issues/4123 it seems like the Mermaid diagram plugin requires some javascript to be executed on render? This is currently not supported in TechDocs due to it being a security threat for TechDocs. But I would like to know more about if its required and if so - if there is a specific type we can allow, similar to how it could be allowed for MathJax. https://github.com/backstage/backstage/issues/4123#issuecomment-854097995

[dtuite]

Thanks @emmaindal . I commented on the other issue to try and understand more.

[fgascon]

What if the client javascript was loaded by Backstage instead. That should remove the security issue.

It would have the downside of requiring the file to be loaded in every documentation site, even if that documentation doesn't use Mermaid. But that shouldn't be an issue if it's something the Backstage integrators can opt-in to.

[johanneswuerbach]

We are now using mermaid-cli to render graphs at build time (more details https://github.com/backstage/backstage/issues/4123#issuecomment-1010213038).

Sadly mermaid-cli requires puppeteer / chromium, so I'm not sure, whether this could be added to the container by default as it definitely increases the image size a lot.

[testn]

@johanneswuerbach Is there a way to make do without puppeteer / chromium?

[dmckernanacv]

Hello, my users are also interested in rendering mermaid diagrams in their tech-docs, especially-so since github now supports mermaid in it's own markdown.

[johanneswuerbach]

@testn not that I’m aware of :-(

[iamEAP]

Labeled this issue with help-wanted and also renamed to help distinguish it from #backstage/backstage#4123.

Although the core team does not plan on working on this ourselves, we're happy to review PRs that would add build-time support! A mermaid-cli based approach could be good. There's also a partial attempt to enable this using kroki here that someone could try and take across the finish line.

[johanneswuerbach]

I‘m happy to send an upstream PR, but I assumed it isn’t acceptable as mermaid-cli uses puppeteer, which requires chromium and overall adds ~600MB to the image :-(

For us that is acceptable as we heavily use mermaid, but I'm not sure everyone would think the same way.

The kroki local version btw. has the same problem https://github.com/yuzutech/kroki/blob/main/mermaid/Dockerfile#L24 and using the hosted version wasn't an option for us as it requires procurement etc.

[testn]

What do you think about something similar to https://mermaid.ink/?

[johanneswuerbach]

At least from my company perspective sending internal diagrams to a 3rd party, where you have no commercial relations with, is always undesirable.

[geertvanheusden]

On my local setup I installed the Kroki mkdocs plugin next to the mkdocks-techdocs-core. The Kroki public service or local setup (Docker containers in my case), allows you to render the following diagrams:

Having this plugin installed by default in the mkdocs-techdocs-core might resolve the fact that we need to install all the separate diagram renderers and their plugins...

---

This is my local setup as I wanted to prevent my diagrams being rendered in the public Kroki cloud service:

mkdocs.yml

site_name: Library A

nav:
  - Home: index.md

plugins:
  - techdocs-core
  - kroki:
     ServerURL: http://localhost/api/proxy/kroki

app-config.yaml

...
proxy:
  '/kroki':
    target: 'http://kroki-core'
    changeOrigin: true
...

docker-compose.yml (DB and env vars left out of the example)

 version: '3.1'

services:

  backstage:
    image: backstage
    ports:
      - 80:7007

  kroki-core:
    image: yuzutech/kroki
    environment:
      - KROKI_BLOCKDIAG_HOST=kroki-blockdiag
      - KROKI_MERMAID_HOST=kroki-mermaid
      - KROKI_BPMN_HOST=kroki-bpmn
      - KROKI_EXCALIDRAW_HOST=kroki-excalidraw
    ports:
      - "8000:8000"

  kroki-blockdiag:
    image: yuzutech/kroki-blockdiag
    ports:
      - "8001:8001"

  kroki-mermaid:
    image: yuzutech/kroki-mermaid
    ports:
      - "8002:8002"

  kroki-bpmn:
    image: yuzutech/kroki-bpmn
    ports:
      - "8003:8003"

  kroki-excalidraw:
    image: yuzutech/kroki-excalidraw
    ports:
      - "8004:8004"

[geertvanheusden]

Sorry, I should have read the full thread as I missed the open PR where Kroki was introduced... My bad 😉

[iamEAP]

Yes, the challenge with the above PR is integrating the kroki server within the TechDocs container (rather than the docker compose method you highlighted above)

All of that being said, the fact that the default configuration for the kroki plugin is to use the public / 3rd party service (which then explicitly has to be overridden to point locally or to a self-hosted server), makes me hesitant to include it by default in mkdocs-techdocs-core, given not everyone who uses this plugin necessarily uses the techdocs-container.

Naturally, TechDocs adopters can (and are able to) support this in a variety of ways (highlighted in this thread, documented elsewhere, etc). ...However, given mermaid is a client-side diagraming solution, it might be best to avoid build-time solutions and instead introduce a mermaid addon once we ship the render-time-oriented addon framework (which is actively under development and nearing completion).

[geertvanheusden]

All of that being said, the fact that the default configuration for the kroki plugin is to use the public / 3rd party service (which then explicitly has to be overridden to point locally or to a self-hosted server), makes me hesitant to include it by default in mkdocs-techdocs-core, given not everyone who uses this plugin necessarily uses the techdocs-container.

Would it make a difference if setting the Kroki server URL would be mandatory if you want to use the plugin? If it becomes mandatory, you can choose your own infrastructure or go with the publicly hosted Kroki server. But you have to choose rather than setting the public one by default. If you don't specify it, the plugin will not be enabled.

Seems to me like a low-risk solution where you have full control without any surprises... What do you think @iamEAP ?

[iamEAP]

That approach seems reasonably secure, @geertvanheusden; a couple of other thoughts:

• Container startup time: do you know how much overhead would be added if invoking the spotify/techdocs container also spun up one or more kroki servers before running mkdocs? A use-case of the container is to run things interactively, locally, so keeping startup time low would be great.
• Image size: Echoing Johannes' note on affecting spotify/techdocs' size; it's hard to estimate how much the additional kroki dependencies would add, just looking at the corresponding images on docker hub. Any idea on a ballpark? It'd be great to keep the image size low too.

[johanneswuerbach]

Bundling kroki mermaid rendering would likely have a similar ~600MB addition as kroki also needs chromium & puppeteer https://github.com/yuzutech/kroki/blob/main/mermaid/Dockerfile#L24

[geertvanheusden]

I am not sure if I follow your concern about the container startup time. The Kroki server is built to serve "live" image requests coming directly from the rendered web pages. It would not be called while running the mkdocs command itself. Although the mkdocs-kroki plugin provides an option (DownloadImages) to embed the SVG image into the page which could be used during the mkdocs run command.

If you stick with the first option, the Kroki container must stay available to be able to serve the image requests. But you just have to start it once.

Here is an overview of the "latest" image sizes:

• yuzutech/kroki: 510 MB
• yuzutech/kroki-mermaid:latest: 474 MB
• yuzutech/kroki-blockdiag: 120 MB
• yuzutech/kroki-bpmn: 351 MB
• yuzutech/kroki-excalidraw: 351 MB

So indeed it would start adding up in file size. To be honest, I would not be in favour to add all these dependencies in the spotify/techodcs container. It's hard to maintain and combine, that's one of the reasons Kroki splits it up into multiple containers.

[axdotl]

Out of curiosity - is this issue solved with the upgrade to mkdocs v9?

• 109

[SayakMukhopadhyay]

Out of curiosity - is this issue solved with the upgrade to mkdocs v9?

* [Bump mkdocs material to v9 #109](https://github.com/backstage/mkdocs-techdocs-core/pull/109)

This is something that I have been trying to test myself. Before this PR, mkdocs-material version getting used was 8.1.x which didn't have mermaid support. Version 8.2.0 brought in "native" support for mermaid from mkdocs-material. But I have not managed to get it working. Here's is what I have tried for reference:

My mkdocs.yaml looks like this now

site_name: Technical Documents
site_description: Technical documents

nav:
  - Home: index.md
  - <other items>

plugins:
  - techdocs-core
  - kroki:
      ServerURL: <personal kroki server>

markdown_extensions:
  - admonition
  - pymdownx.details
  - pymdownx.superfences:
      custom_fences:
        - name: mermaid
          class: mermaid
          format: !!python/name:pymdownx.superfences.fence_code_format

My markdown file has the following

```mermaid
flowchart LR
  A[Parent 1] --> B[Child 1]
  A --> C[Child 2]
  A --> D[Child 3 Parent 2]
  D --> E[Child 4]
  D --> F[Child 5]

I am building this using github actions and they have the following steps

```yaml
    - name: Install techdocs-cli
      run: |
        npm i -g @techdocs/cli
      shell: bash

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.9'

    - name: Install mkdocs and mkdocs plugins
      run: |
        pip install mkdocs-techdocs-core==1.*
        pip install mkdocs-kroki-plugin
      shell: bash

    - name: Generate docs
      run: |
        techdocs-cli generate --no-docker --verbose
      shell: bash

I am able to see mkdocs-material features like admonition but the mermaid diagram doesn't render and only shows the mermaid code in a code block like any other triple back ticked code.

I have been checking how the output should look like from mkdocs-material own website and it seems like the generated diagram should be an SVG and should be reactive to theme changes (admonition is actually reactive to theme changes in Backstage which is great!).

I don't think there is a necessity of using client side JS if mkdocs-material builds an SVG for us. But at the moment I have not been able to figure out why its not working. Maybe I am missing something.

Also this comment might be useful https://github.com/backstage/mkdocs-techdocs-core/pull/101#issuecomment-1339058853

[SayakMukhopadhyay]

I think I have found the source of the issue. It seems like if I remove https://github.com/backstage/mkdocs-techdocs-core/blob/b12f8e6f3ae315d24e3ca81db2c2dfca5558a0a4/src/core.py#L68 and https://github.com/backstage/mkdocs-techdocs-core/blob/b12f8e6f3ae315d24e3ca81db2c2dfca5558a0a4/src/core.py#L112 I am able to generate the SVG mermaid diagram. I have tested by running mkdocs from source, passing the same arguments that techdocs-cli does.

There was a conversation in the PR by @agentbellnorm which added line 68 https://github.com/backstage/mkdocs-techdocs-core/pull/109#discussion_r1147340159

Line 112 seems to have been added way back and I don't know why keeping it is also causing the SVG to not be created. But if I remove "pymdownx.extra" from core.py and add it in mkdocs.yaml, the SVG is getting created just fine. The same is not true for theme.palette. Removing it from core.py generates the SVG but adding this to the mkdocs.yaml again stops generating the SVG.

theme:
  name: material
  palette: ""

I am still debugging and will update here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SayakMukhopadhyay]

Could this be reopened? I think this is still a valid feature request.

[stephanschielke]

We are using the techdocs-cli to render and upload techdocs to S3 and ended up with the following docker wrapper image using the mermaid-cli and the markdown-inline-mermaid plugin.

Similar to this setup: https://github.com/TwistoPayments/actions-techdocs/pull/1/files

FROM node:18

ENV DEBIAN_FRONTEND=noninteractive

# Install python for mkdocs
# Install chromium for mermaid
RUN apt-get update && \
    apt-get -y install --no-install-recommends python3 python3-pip chromium && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install AWS cli for S3 upload
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
    unzip awscliv2.zip && \
    ./aws/install

# Install techdocs and mermaid cli
# techdocs-cli being used in CI
RUN npm install -g @techdocs/cli -g @mermaid-js/mermaid-cli

# Install all mkdocs plugins + markdown extensions
RUN pip3 install --break-system-packages --no-cache-dir \
    mkdocs-techdocs-core==1.* \
    markdown-inline-mermaid

# Switch to default user
USER node

# Set mermaid-cli envs to link to installed chromium
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium

ENTRYPOINT ["mkdocs"]

# Build a local image like this:
#   docker build . -t techdocs-markdown:local
# Start the container like this:
#   docker run --rm techdocs-markdown:local /bin/bash -c "techdocs-cli --version"
# Or interactively:
#   docker run -it --rm --entrypoint /bin/bash techdocs-markdown:local
# Or as dind through the techdocs-cli and render stuff in a standalone server
#   techdocs-cli serve:mkdocs --verbose -i techdocs-markdown:local
# Or add it to your local backstage configuration like this
#   techdocs:
#       generator:
#           runIn: 'docker'
#           dockerImage: 'techdocs-markdown:local'
#           pullImage: false
#       builder: local

mkdocs.yml must include this extension:

plugins:
  - techdocs-core
markdown_extensions:
  - markdown_inline_mermaid

It's far from perfect but works locally and in our CI. Hope this helps anybody