Improve the performance of the module navigation tree on the package documentation pages

[sabine]

Problem

Currently, the package documentation pages are very slow for some packages that contain a lot of symbols. For example, see https://ocaml.org/p/core/latest/doc/index.html.

The cause of that is most likely that the entire HTML structure of the module navigation tree is rendered every single time the page is loaded.

Task

Look into improving the performance of package documentation pages with a lot of entries in the module tree navigation on the left.

Here's something you could try:

1. Use AlpineJS on the frontend to render only the open sections of the module tree.
2. This requires passing the module tree to the frontend via the generated HTML (most likely as JSON data).
3. Now, since we do the highlighting of open and active pages in AlpineJS, we no longer need to swap out the sidebar content with HTMX - the UX will improve because there will be no jumping around of items in the sidebar anymore.

Rendering the JSON data should be much faster and takes up less space than rendering the entire module tree to HTML. (For example, see the package.json data for the core library here: https://docs-data.ocaml.org/live/p/core/v0.16.2/package.json)

Important: The data format in https://docs-data.ocaml.org/live/p/core/v0.16.2/package.json is not stable, it could be subject to change. That is why the voodoo_serialize library exists (found in tool/voodoo_serialize, and why there is a module src/ocamlorg_package/lib/package_info.ml that translates the data into a format that the frontend could rely on. If needed, you could change the types in src/ocamlorg_package/lib/package_info.ml to be more amenable for use by AlpineJS and serialize this to JSON via the yojson library.

[sabine]

Ok, so the other, simpler thing that could be tried here:

Cut the module tree: don't give people the option to expand branches in the first place, instead navigate to the page in question.

This will greatly reduce the data that needs to be sent to the client and we can still send it as HTML.

[huwaireb]

Would love to tackle this,

will update this comment after experimenting with my ideas 😁.

edit

There's more than one clog that bottleneck the performance of package and subsequently documentation pages. This issue is scoped to the documentation page so we'll stick to that for now.

Two words, virtual scrolling. That's generally all what's necessary to speed things up drastically. Chrome conveniently offers a CSS property accessible on all chromium-based browsers called content-visiblity.

On non-chromium browsers, as a "polyfill", one could instead rely on alpine to offer such through perhaps the Intersect plugin.

One more optimization that could be done when discussing with @sabine is reducing initial server response times by deferring calls done to docs-data endpoints to the client-side. It would've been a lot better if docs-data was available directly to ocaml.org through perhaps a database, but that's too much work to do although it would've offered significant speed ups across all packages.

In regards to the module tree. Only few elements are initially rendered yet we're shipping the entire module tree which greatly increase the size of the document, storing that data as JSON instead and adding/removing them to the dom as needed may be ideal here as @sabine suggested above. If the goal is to reduce the document size, I don't believe, given they have display: none set that they have other costs e.g rendering.

I see so much potential for htmx to avoid unnecessary full page rendering when traversing module tree 😭

[The-Amoghavarsha]

@sabine should we halt this for the time being?