search

asciidoctor / asciidoctor-diagram

:left_right_arrow: Asciidoctor diagram extension, with support for AsciiToSVG, BlockDiag (BlockDiag, SeqDiag, ActDiag, NwDiag), Ditaa, Erd, GraphViz, Mermaid, Msc, PlantUML, Shaape, SvgBob, Syntrax, UMLet, Vega, Vega-Lite and WaveDrom.
http://asciidoctor.org
MIT License
439 stars 107 forks source link

Generate Diagrams in Parallel #432

Open KartikSoneji opened 1 year ago

KartikSoneji commented 1 year ago

At the moment, it looks like the extension generates diagrams as it comes across them, which can get quite slow. Is it possible to generate diagrams in parallel with a pool of n workers? That would greatly speed up generation for documents with many diagrams and a slow processor like mermaid.

Not sure if it would need changes to the asciidoctor extension system, but either way I'll be more than happy to help.

pepijnve commented 1 year ago

The extension system isn't designed for asynchronous operation as far as I know. The callback hooks are all synchronous. It might be possible to hack together a solution that achieves some degree of asynchrony, but I think this would be at the cost of reliability.

@mojavelinux wdyt?

mojavelinux commented 1 year ago

In theory, it would be possible to replace the diagram with a marker, then start working on those markers in the background. The caller would have to force a join when the document is done converting. But this is unlikely to help except for the specific circumstances when the document is extremely large.

I think a better approach would be to separate diagram generation from processing altogether if you want to be able to work in this mode. That would not be the behavior/responsibility of this extension (as I would view it to be out of scope), but rather a different one that makes use of this extension for that purpose/workflow. In other words, you would need to think about separating diagram generation from AsciiDoc conversion entirely.

KartikSoneji commented 1 year ago

Except for inline images, the actual generation is independent of the markup, right? Can't the extension return the markup in those cases at least?

pepijnve commented 1 year ago

Perhaps we could make a companion post processor extension to handle this. When it's loaded, the diagram extension goes through the usual motions, but instead of actually producing the images it collects all the "production instructions". Then in the post processor you execute those instructions in parallel. The post processor could still be synchronous by waiting for the work queue to be completed entirely.

PRs welcome ;)