Need faster/smarter compilation

[MicahElliott]

It appears that the whole world gets rebuilt every time any file is changed instead of just the single file. I'm sure you're aware of this problem. When I had only a couple pages/posts, I was compiling in ~1 second. But now with ~15, it’s taking 4+s, and sometimes 9s.

This doesn’t feel like a showstopper yet, but I expect it to get pretty painful for those who write a lot and like to frequently look at a browser view of their posts.

Note that these are Asciidoc files, and I don’t know if that’s just slower than compiling Markdown.

I haven’t tried profiling the compilation yet, but it seems that the watcher should only recompile a single .asc/.md file. Hopefully that’s where most of the time is going. It looks like the majority of time is spent in the stage “compiling assets...”.

I also just noticed that lein is growing to use ~3 GB of memory!! after a lot of compilations. That’s not what’s causing the slow compiles, though.

(Sorry for the flurry of issues. I just want to make sure the findings are being documented as I hit them.)

[lacarmen]

It seems like Markdown compilation is faster than Asciidoc. I think @yogthos mentioned he can do ~40 posts in 4-5 seconds.

Either way, you're mostly right about recompiling only a single page. If you consider all the different pages that could be affected by changing one post, at least 5 files would have to be recompiled - the post that changed, the previous/next post, the archives and the index page.

If you have any suggestions I'd love to hear them :)

[Tankanow]

I have a really simple fix to speed up the asciidoc generation. The asciidoc render-fn uses the (Asciidoctor$Factory/create) factory method to create an Asciidoctor converter instance. This instance creation is what takes a little bit of time.

Unfortunately the render-fn creates a new instance every time it converts a file. The fix is to simply assign the result of (Asciidoctor$Factory/create) to a private or local variable. I'm working on a branch for this now ... I should have a merge request this week.

Cheers,

Adam

[lacarmen]

Awesome, thanks for investigating!

[lacarmen]

cryogen-asciidoc 0.1.2 containing the above fix has been pushed to Clojars. However I'll leave this issue open in order to address the idea of "smart" compilation.

[harlanji]

Any movement or new thoughts on this? I'm building a stack based on RasPi and the newly generated site takes nearly a minute to build using cryogen-core 0.1.60 (Dockerfiles: 1 2, base openjdk:8 armhf).

It's not a blocker for me yet but it's hurting my case when I show it off, so I'd appreciate any suggestions for improvements. I've deeply invalidated "Clojure is slow" on a huge+fast x86 system so I'm confident a solution is forthcoming, maybe it's just too-constrained memory or slow SD access; I'll test that before I do any code interrogations. I may disappear for a while as it's kinda low priority, but thanks in advance for any pointers.

[coreagile]

Boy would I love this. Just started using the project and ported a previous blog over. 2 seconds per save is excruciating for just 33 files. And if I increase my post frequency, which I hope to do, I imagine it will get worse fast.

[holyjak]

I guess we could add a "quick and dirty" mode where only the changed file is recompiled (and not the pages that point to it - index, tag pages, ...). For proper, complete incremental rendering we would need a dependency graph. Perhaps we could steal a solution from somewhere...

[markokocic]

One simple solution to compile the changed page first, and everything else afterwards. That way at least one would not need to wait too long to see his changes on the page they are editing.

[holyjak]

Closing this as there is now in 0.4.0 http://cryogenweb.org/docs/fast-compilation.html It is now perfect but perhaps good enough.