Cache generated files to speed up generation

[Karreg]

Diagram generation can be slow, depending on the diagrams complexity. Especially with the ELK layout engine.

For example, with a simple markdown documentation:

• generation will take 0.61 seconds with no d2 diagrams
• generation will take 7 seconds with the following diagram included twice (embedded and included):

  direction: right
  🧔‍♂️ -> 🌍: 👋

  (values depending on external factors, but magnitude orders are about the same)

When writing a documentation with mkdocs, the best option is to use mkdocs serve that will rebuild the documentation when the source is changed. In this case, mkdocs-d2-plugin will re-generate all diagrams, even though the source file is not updated.

A cache system, allowing to rebuild diagrams only when the source file has been changed would be great. drawio exporter plugin has such an feature for the same reason (especially since drawio requires chrome to generate images).

[landmaj]

generation will take 7 seconds with the following diagram included twice

It's way faster on my MacBook Air M1 - building documentation of this project, with 5 diagrams (one uses ELK) takes 1.61 second - but I agree, it's too slow. The main culprit is actually calling an external process, not generation itself, at least for small to medium diagrams.

I do intent to add a cache.

[Karreg]

M1 heh, lucky you :D I have to work in a linux virtual machine on a clogged Windows host...

I have tested, and you are totally right:

gitpod@6a7754c6b56c$ mkdocs build
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /workspaces/docker-mkdocs/public
True
True
INFO    -  Documentation built in 3.61 seconds
gitpod@6a7754c6b56c$ d2 docs/d2.d2 
success: successfully compiled docs/d2.d2 to docs/d2.svg in 143.375124ms

On another note, I think you still have a debug leftover with the following lines, looks like a line per diagram:

True
True

:)

Thanks for your great plugin!

[landmaj]

I just released version v1.1.0 with caching. Link to documentation

Here you can see two subsequent runs on my computer:

~/Documents/GitHub/mkdocs-d2-plugin/docs master
.venv ❯ mkdocs build
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (None)
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory:
           /Users/landmaj/Documents/GitHub/mkdocs-d2-plugin/docs/site
INFO    -  Documentation built in 1.22 seconds

~/Documents/GitHub/mkdocs-d2-plugin/docs master
.venv ❯ mkdocs build
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (dbm.ndbm)
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory:
           /Users/landmaj/Documents/GitHub/mkdocs-d2-plugin/docs/site
INFO    -  Documentation built in 0.22 seconds

[Karreg]

I confirm the speed up when using mkdocs build:

gitpod@6a7754c6b56c$ mkdocs build
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (None)
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /workspaces/docker-mkdocs/public
INFO    -  Documentation built in 3.89 seconds

gitpod@6a7754c6b56c$ mkdocs build
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (dbm.ndbm)
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /workspaces/docker-mkdocs/public
INFO    -  Documentation built in 1.06 seconds

But it has no effect with mkdocs serve, which would be more beneficial:

gitpod@6a7754c6b56c$ mkdocs serve
INFO    -  Building documentation...
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (None)
INFO    -  Cleaning site directory
INFO    -  Documentation built in 2.52 seconds
INFO    -  [08:19:53] Watching paths for changes: 'docs', 'mkdocs.yml'
INFO    -  [08:19:53] Serving on http://127.0.0.1:8000/

INFO    -  [08:20:01] Detected file changes
INFO    -  Building documentation...
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (dbm.ndbm)
INFO    -  Documentation built in 2.83 seconds
INFO    -  [08:20:04] Reloading browsers

I'll try to have a look at where the cache is failing.

[Karreg]

That's weird. I have modified the python file for the plugin, but no change in behavior. Just some logs. And now it's working fine even with mkdocs serve:

INFO    -  [08:37:04] Detected file changes
INFO    -  Building documentation...
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (dbm.ndbm)
INFO    -  mkdocs-d2-plugin: source not found in cache.
INFO    -  Documentation built in 2.37 seconds
INFO    -  [08:37:06] Reloading browsers

INFO    -  [08:37:18] Detected file changes
INFO    -  Building documentation...
INFO    -  mkdocs-d2-plugin: Using cache at .cache/plugin/d2/db (dbm.ndbm)
INFO    -  Documentation built in 0.72 seconds
INFO    -  [08:37:18] Reloading browsers

I can't explain that. If I ever reproduce the issue, I'll dig into it and find a fix...

Thanks!

[landmaj]

I tested mkdocs serve so I would be surprised if it didn't work. If you ever encounter this issue again, please open an issue and I will try to reproduce it.

I still have a few ideas for improvements but nothing major.

[Karreg]

Yes, as is the plugin is already great. It's only missing the multilayer IMHO, but it's more on d2 side. I have started a discussion on their Discord by the way. No answer yet, but if it happens I'll notify you: https://discord.com/channels/1039184639652265985/1039185315685998693/1171364484095426601