search

remcohaszing / remark-mermaidjs

A remark plugin to render mermaid diagrams using playwright
MIT License
70 stars 7 forks source link

Tests are unstable #5

Closed remcohaszing closed 1 year ago

remcohaszing commented 2 years ago

The tests are currently unstable due to the unstable output of mermaid. As a result I had to allow test failures in CI, which is really bad. I’m open to suggestions to make the tests more stable.

aloisklink commented 1 year ago

You can try setting deterministicIds to true in the Mermaid config: https://github.com/mermaid-js/mermaid/blob/03c5bc112947eebb428e1d540a665392b5eb800c/packages/mermaid/src/defaultConfig.ts#L146, which should help for most IDs (if that's the only thing that is changing).

Unfortunately, that still doesn't help everywhere: I just had to fix a bug in the entity relationship diagrams that weren't deterministic, see https://github.com/mermaid-js/mermaid/pull/3916.

There's also some discussion in https://github.com/mermaid-js/mermaid/issues/3491 about test snapshots in Mermaid, but it's mainly centered around rendered .png snapshots, not .svg snapshots.

.png snapshots do seem to be more stable (but much much slower) and it's what I use for https://github.com/aloisklink/remark-mermaid-dataurl (although I have much less tests than your project!)

remcohaszing commented 1 year ago

Thanks for the suggestions and cross-referencing related issues! It helps to have this information together.

The problem isn’t really deterministic IDs though. It’s mostly pixel offsets. The tests are relatively stable on my machine. In #17 Bruce says they are for him too. I’m inclined to believe they are in CI too. The problem is that they are unstable across different environments.

I like the idea of adding support for different output options, such as SVG or PNG data URLs. They would be a additional options though, not a replacement.

aloisklink commented 1 year ago

The problem is that they are unstable across different environments.

It might help to make sure all environments have the same fonts installed. Mermaid defaults to using trebuchet ms as the default font, which isn't great, since ms stands for Microsoft. You can install it pretty easily on CI using sudo apt-get install ttf-mscorefonts-installer, but it's not open-source, so you need to accept a EULA first.

(my wishlist is for Mermaid to default to using an open-source font like Atkinson Hyberlegible)

I doubt you'll get it to be pixel perfect on all environments though. I've heard that ARM-based CPUs and x86_64 CPUs both render CSS shadows slightly differently. My gut feeling is that ClearType/FreeType might also change how text is rendered, depending on what sub-pixel arrangement your screen has, so plugging an external monitor into your laptop might change the output!

I like the idea of adding support for different output options, such as SVG or PNG data URLs. They would be a additional options though, not a replacement.

If you do add SVG data URLs to your project, then I'll be able to archive my https://github.com/aloisklink/remark-mermaid-dataurl! Or convert it to a wrapper that uses this plugin.

remcohaszing commented 1 year ago

This issue has been solved by 2312642c34d97da498bd910a8159012d2f6988ef and 236c16c04777388d8160698919bf180ee9ba29c6.

@aloisklink

If you do add SVG data URLs to your project, then I'll be able to archive my https://github.com/aloisklink/remark-mermaid-dataurl! Or convert it to a wrapper that uses this plugin.

rehype-mermaidjs supports multiple rendering strategies, including img-png for base64 PNG data URLs and img-svg for SVG data URLs. It also supports better accessibility.