Is it possible for django-distill to utilize all CPUs to perform multithreaded export

[kirmola]

Basically same as title. I want to know if there is or will be an option to generate static site to keep build time minimal. For a non-fair comparision, we can compare it to Hugo which is the fastest static site generator available.

It seems important to me because my static site is generating a 100K pages and I wish to keep the build time minimal so that most of static site hosting provider doesn't throttle build.

PS: I just checked and my generated public folder is around 3 GBs.👀

[meeb]

The rendering process is just a loop over the URLs so there's no reason I can immediately think of why you couldn't relatively easily concurrently generate pages. No-one has reported generating a 100k page static site so it's not been an issue previously. I'll pop it on the wishlist!

[meeb]

The above commit adds --parallel-render [number of threads] to both ./manage.py distill-local and ./manage.py distill-publish commands. You can set this to any positive integer you like, such as one per CPU core you have available. It'll be bundled into the next release. Some steps will still be single threaded (preparing to render, collecting static files etc.) but this should improve performance for sites where most of the CPU time is spent spinning on template rendering.

[kirmola]

The above commit adds --parallel-render [number of threads] to both ./manage.py distill-local and ./manage.py distill-publish commands. You can set this to any positive integer you like, such as one per CPU core you have available. It'll be bundled into the next release. Some steps will still be single threaded (preparing to render, collecting static files etc.) but this should improve performance for sites where most of the CPU time is spent spinning on template rendering.

Thanks for this enhancement, most of my projects involve rendering static sites at large scale, this sure would be a huge help. Thanks again.