rikhuijzer / PlutoStaticHTML.jl

Convert Pluto notebooks to HTML in automated workflows
https://PlutoStaticHTML.huijzer.xyz
MIT License
81 stars 7 forks source link

Limit concurrency #98

Closed rikhuijzer closed 2 years ago

rikhuijzer commented 2 years ago

Closes #36.

Slowly but surely, my blog is containing more and more notebooks. When doing a full run over all the 12 notebooks, my PC ended up being unusable because PlutoStaticHTML.jl was spawning too many threads causing all time to be spent switching between tasks instead of doing productive work. I thought that Pluto.jl took care of this, but that is apparently not the case.

This PR fixes the problem by starting multiple async Pluto.jl evaluations via Threads.@threads. In each loop, PlutoStaticHTML.jl will wait for the async task to finish before starting more notebooks.

I've tested this functionality on my blog posts. Indeed, it nicely starts 6 notebooks and waits for one to finish before starting new notebooks. I'm typing this while the processes keep my PC busy at around 100%. Still, the PC isn't grinding to a halt like I had yesterday, so all appears to work as expected :rocket: :balloon: