Serve mode performance with many pages

[marvinhagemeister]

Enter your suggestions in details:

I've been helping out with content work on https://docs.deno.com/ which is built on top of lume in the past week. Whilst working with other members of the team, we noticed that build times varied greatly between machines. For our docs we render about ~9000 pages with lume.

You can try out our setup by running these steps:

1. Clone https://github.com/denoland/docs
2. Run cd docs/reference_gen && deno task types && deno task doc to prebuilt the reference docs (where most of our pages come from)
3. Run cd .. to go back to the repo root folder
4. Run deno task serve to start lume

In particular, we're interested in areas which make the serve mode faster, as this is the one we interact with the most frequently when working on our docs. On my fully spec'ed Macbook M2 Pro it takes around 8s when saving a markdown file until it's fully processed and the browser is notified. For another member with a slower machine this takes upwards of 30s or even more.

Poking around some CPU profiles it seems like the tailwind plugin in particular is quite expensive:

Of the 8s of work caused by the change to markdown file ~2.6s of that are spent in the tailwind plugin. Looking at the implementation of this plugin it seems like it loops over all pages and feeds the content of all pages into tailwindcss for further processing. Since the number of pages in our scenario is quite large, this takes a lot of time.

https://github.com/lumeland/lume/blob/203e22e133f233e32bd3933cc6728d6c14a45e0c/plugins/tailwindcss.ts#L42-L53

Another plugin where a similar thing seems to be happening is the prism plugin. The callstacks in the red box are caused by it.

What seems to take the most time there is not the actual highlighting itself, but rather the time it takes to parse every page into a full HTML AST to grab out the relevant bits for prism to do its thing on. Similar to the tailwind plugin, this goes happens for all pages, whenever a single page changed.

https://github.com/lumeland/lume/blob/203e22e133f233e32bd3933cc6728d6c14a45e0c/plugins/prism.ts#L58-L61

Looking at these traces, I wonder if there is a way we can make lume only operate on the pages that changed, rather than all of them in these two plugins. That alone should be a nice performance improvement for lume.

[oscarotero]

Hey @marvinhagemeister Thanks for the useful data!

I'm aware about Tailwind performance. In fact I don't recomend to use Tailwind (specially big sites) because it's not scalable, reusable nor easy to maintain. If I were you, I'd focus on building a good CSS design system for Deno (but it's only my opinion).

Anyway, I'm open for ideas to improve Tailwind performance.

• We have to define what "a page doesn't change" means. Because, the output html of a page can change for several reasons: the markdown has edited, the layout, a template, a single component, a processor, a single variable, etc...
• Tailwind needs to pass all HTML code (even those pages that didn't change) to the plugin because the CSS code of all pages is stored into a single file. If only the changed pages where passed to the plugin, it would remove the CSS code related to the unchanged pages. Maybe UnoCSS plugin could help here, because it has a mode to output the styles of every page separately (creating a <style> to set the styles inline). In this case it would be possible to skip the pages that didn't change (it's not already implemented but it wouldn't be hard).
• Once the html code of a page is parsed (accessing to the page.document property), the document is cached. You can see here the implementation of page.content and page.document. This avoid to reparse the document if there are consecutive plugins using the DOM API.
• We have been experimenting linkedom to replace deno-dom as the DOM implementation and we found it much more faster and consume 50% of memory. We didn't change it yet because there is a couple of plugins that can be affected (because the different implementation of the <template> element). The good news is it's easy to make the change for you using import maps, so we can test this for your Deno docs site.

[oscarotero]

Hi @marvinhagemeister

I see you have created a plugin to run tailwind directly scaning the source files instead of the output html pages. That's a good solution for your use case. Any reason you didn't use the Lume postcss plugin with tailwind? I mean:

site.use(postcss({
  plugins: [tailwind()]
}));

[marvinhagemeister]

No particular reason. Mostly copied this over, but could've used the official postcss plugin as well.

[oscarotero]

Okay. I asked it just in case you found a bug or limitation with the postcss plugin. But if it's not the case, no problem.

I think I can close this issue. Feel free to reopen it or create a new one if you have more problems.

Thanks!