Slow / Exponential Build Time Caused by Component.Explorer

[RyanGreenup]

Problem Statement

The Exponent Component (see also Explorer in the docs)) of Quartz causes a non-linear build time, For example with the default sort function:

n	Explorer Component	No Explorer
1	3.17	4.87
10	3.13	3.35
50	3.47	3.45
100	3.33	3.44
500		3.85
1000	6.74	4.28
2000	18.57	5.14
3000	37.15	5.95
		6.64
5000	92.31	9.81

Proposed Solution

Either one of the following would solve this issue

1. The default layout should not incur exponential build times
2. Amend the documentation to
   1. Warn of long build times with the default sort function
   2. Include an example of such a function

Methodology

1. Description

   1. Clone a fresh repository

   2. Setup with relative links and default content directory

   3. Copy the `index.md` into separate files

   4. Build the site and time the duration

   5. Comment out the component in quartz.layout.ts

      export const defaultContentPageLayout: PageLayout = {
         ...
         left: [
             ...
             // Component.DesktopOnly(Component.Explorer({}),
             ...
         ],
         ...
         }
      

   Here's the basic idea in python

   import system
   import os
   n = 1000
   times = {
            'n':    [],
            'time': []
            }
   
   for i in range(n):
       os.system(f'cp content/intro.md content/{i}_intro.md')
       start = time.time()
       os.system('npx quartz build --concurrency 4')
       finish = time.time()
       times['n'].append(finish)
   

2. System

   Key	Value
   OS	Gentoo (Glibc / OpenRC)
   npm --version	10.8.1
   Quartz	4.3.1 (c365513)
   node --version	22.3.0
   Browser	N/A

Additional Notes

This doesn't seem to occur for defaultListPageLayout, however, this may be due to a limited number of directories in my experiments.

[saberzero1]

Thanks for the detailed report!

My first thought is that the explorer component gets calculated for every page. (despite being the same on every page)

This means that as the number of pages increases, both the number of explorer components that are parsed increase AND the number of elements inside the explorer component.

Basically, every page added adds an extra cost to every other page as well.

We should probably look into a way to calculate the entire explorer tree only once per site build, instead of per build per page.

@RyanGreenup do you have an example repo to reproduce your tests? I want to do some experiments, especially regarding the sort function, as that was introduced fairly recently.

[jackyzha0]

hm Quartz memoizes the explorer file tree, this shouldnt happen, a repro would be helpful here

[jackyzha0]

can you see if https://github.com/jackyzha0/quartz/commit/16a9caa555a2d63b7ff8af0731fbfd3231d6225c fixes it?

[RyanGreenup]

can you see if 16a9caa fixes it?

I put together a python script in this exemplar repo.

I'll check this when I get back from work.

Thanks for the quick turn around!

[RyanGreenup]

I'll double check my work and run it over more docs when I get home, but it doesn't look like it made a difference:

[jackyzha0]

Ah can reproduce on my end, thanks for the script, I'll take another look tomorrow when I have some spare time