Closed mbutterick closed 2 years ago
I spent some time testing this last night.
I didn't happen to see any improvement when doing a full from-scratch rebuild on either 7.6 BC or 7.7 BC. In the case where I do a full build and then did touch pollen.rkt
and I did see an improvement of between 6–9 seconds. (By the way, my raco make pollen.rkt
step still pays pretty hefty performance dividends)
But. Each of my pages is doing one or more INSERT
s on a SQLite file during the template rendering phase. I now wonder if for that reason my projects are not a fair comparison to use for this improvement? I want to delve into what happens behind the scenes when multiple Racket threads attempt to do this. I always assumed that SQLite's file locking would have a minimal effect, but now I am not so sure.
I think there is no denying a 3x improvement in any one project!
I think the fact that there's more than one more INSERT
is due to the reuse of namespace, as I detailed in #43 :(
Yes, I made the change you suggested in #43: reuse a namespace (looking back I see that you also got nearly 3x performance from that change). I realized I could introduce an extra parameter to signal when it was safe to reuse the namespace. Good idea — thanks!
Each of my pages is doing one or more INSERTs on a SQLite file during the template rendering phase
In my admittedly minimal experiments with SQLite in Pollen, I found that the per-transaction costs of a SQL query were very high compared to the average Pollen computation. Unless each INSERT
creates state that affects later renders, you might consider sending all the INSERT
s to be handled in a separate thread, so they don’t block the main render threads (or better — batch them into one large query at the end). Again, this is the opinion of a SQLite idiot.
I think the fact that there's more than one more INSERT is due to the reuse of namespace
Actually, the multiple INSERT
s are something I’m doing on purpose. Each page does at least one, but if certain content is present that will be saved to the database as well, currently in a separate query.
I will do some experimenting with threads and batching for sure. Will start another thread about bespoke caching in general when I have any results to share.
Could you please point to a specific commit and/or summarize the changes that allowed for faster renders?
I’ve been burned before by overconfidence (see: parallel processing). But today I had (what I hope will turn out to be) a genuinely good idea: to create a fast path for
raco pollen setup
andraco pollen render
that skips certain expensive luxuries that one really needs only during interactive development sessions with the project server (which will be as sluggish as always). For instance, rebuilding Beautiful Racket is now 3x faster. Try it and let me know if it improves your life (or the opposite).