eventcatalog build goes OOM

[otbe]

With small fixes mentioned in this post I was now finally able to run npm run build aka eventcatalog build

My build gets stuck at this point:

18:54:11 [types] Added src/env.d.ts type declarations.
18:54:14 [types] Added src/env.d.ts type declarations.
18:54:14 Types generated 2.40s
18:54:14 [check] Getting diagnostics for Astro files in eventcatalog/.eventcatalog-core...
Result (90 files): 
- 0 errors
18:54:37 [types] Added src/env.d.ts type declarations.
18:54:37 [vite] Re-optimizing dependencies because vite config has changed
18:54:38 [types] Added src/env.d.ts type declarations.
18:54:38 [build] output: "static"
18:54:38 [build] directory: eventcatalog/.eventcatalog-core/dist/
18:54:38 [build] Collecting build info...
18:54:38 [build] ✓ Completed in 1.75s.
18:54:38 [build] Building static entrypoints...
18:54:38 [types] Added src/env.d.ts type declarations.

After like 5min I get the OOM

<--- Last few GCs --->

[86902:0x128008000]    99589 ms: Scavenge 4021.5 (4122.3) -> 4019.9 (4136.5) MB, 52.17 / 0.00 ms  (average mu = 0.557, current mu = 0.343) allocation failure; 
[86902:0x128008000]   102070 ms: Mark-Compact 4037.4 (4137.7) -> 4034.8 (4151.6) MB, 2431.83 / 0.00 ms  (average mu = 0.312, current mu = 0.145) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

This run was executed with the standard memory settings. I was playing with --max-old-space-size up to 25GB of RAM but it always fails, maybe a bit later.

I tested this with node v20 and v22.

My catalog contains 30 domains (no versions just index.md), 24 services (no versions just index.md) and ~950 events (including versions, schema file and markdown).

[otbe]

Update:

I was now finally able to build it at least once with --max-old-space-size set to 50GB! and the TS related fixes from https://github.com/event-catalog/eventcatalog/compare/main...otbe:eventcatalog:fix-ts-issues

Observations:

• building static endpoints (where it got stuck always) took 28 minutes with a peak of 48GB used, until this point the node process itself read almost 500GB!! from disk (my overall disk size is 500GB :D)
• building client afterwards took another 18min and kept RAM at the previous level
• as soon as it reached the generiating static routes part RAM usage dropped to 1,5GB and it finally started to write stuff to disk (pages I assume)
• generiating static routes took around also a lot of time and the majority was spent to write disocvers/events/index.html

Overall the whole process took ~1h20m

It produces also quite some huge files btw

All in all it seems to work (apart from broken links due to trailingSlash) and also pagefind seems to work quite nice.

[otbe]

Some more findings.

When setting optimize to true in astros mdx config the building static endpoints goes down from 28min to 1m30s and only uses at max 4GB of RAM. Overall build time is 32min.

The majority of time then is spent while rendering discover/events/index.html. I feel like this is done ~950 times (number of events) however the file does not change anymore after the first iteration.

[otbe]

Indeed. The getStaticPaths() function from discover/[type]/index.astro is broken. It basically creates a type page per collection entry (so 950 times the same one).

Replacing it by something like this fixes it:

export async function getStaticPaths() {
  const events = await getEvents();
  const commands = await getCommands();
  const services = await getServices();
  const domains = await getDomains();

  const buildPage = (type: string, collection: CollectionEntry<CollectionTypes>[]) => {
    return {
      params: {
        type,
      },
      props: {
        type,
        data: collection,
      },
    };
  };

  return [
    buildPage('events', events),
    buildPage('commands', commands),
    buildPage('services', services),
    buildPage('domains', domains),
  ];
}

So bottom line: With optimize=true and the fix for getStaticPaths() overall build time is down from 1h20m to 7m with max 4GB of RAM 👍

[boyney123]

Amazing, love it. I can add this getStaticPaths fix into the project, unless you wanna make a PR?

Also I will add some docs around optimize, @otbe thoughts on if we enable this by default? Or you think it should opt in like they have it?

[otbe]

Opt in would be fine for me. The related option of ignoreElementNames somehow indicates this might be an issue in certain edges only. Sadly the docs do not give us much about side effects.

Feel free to add it, Im about to leave soon :)

Unrleated: there are some console.log in the components that are quite annoying while looking at the build output :D

[boyney123]

Thanks the logs have now been removed in the components