Closed otbe closed 1 month ago
Update:
I was now finally able to build it at least once with --max-old-space-size
set to 50GB! and the TS related fixes from https://github.com/event-catalog/eventcatalog/compare/main...otbe:eventcatalog:fix-ts-issues
Observations:
building static endpoints
(where it got stuck always) took 28 minutes with a peak of 48GB used, until this point the node process itself read almost 500GB!! from disk (my overall disk size is 500GB :D)building client
afterwards took another 18min and kept RAM at the previous levelgeneriating static routes
part RAM usage dropped to 1,5GB and it finally started to write stuff to disk (pages I assume)generiating static routes
took around also a lot of time and the majority was spent to write disocvers/events/index.html
Overall the whole process took ~1h20m
It produces also quite some huge files btw
All in all it seems to work (apart from broken links due to trailingSlash) and also pagefind seems to work quite nice.
Some more findings.
When setting optimize to true in astros mdx config the building static endpoints
goes down from 28min to 1m30s and only uses at max 4GB of RAM. Overall build time is 32min.
The majority of time then is spent while rendering discover/events/index.html
. I feel like this is done ~950 times (number of events) however the file does not change anymore after the first iteration.
Indeed. The getStaticPaths() function from discover/[type]/index.astro
is broken. It basically creates a type page per collection entry (so 950 times the same one).
Replacing it by something like this fixes it:
export async function getStaticPaths() {
const events = await getEvents();
const commands = await getCommands();
const services = await getServices();
const domains = await getDomains();
const buildPage = (type: string, collection: CollectionEntry<CollectionTypes>[]) => {
return {
params: {
type,
},
props: {
type,
data: collection,
},
};
};
return [
buildPage('events', events),
buildPage('commands', commands),
buildPage('services', services),
buildPage('domains', domains),
];
}
So bottom line:
With optimize=true and the fix for getStaticPaths()
overall build time is down from 1h20m to 7m with max 4GB of RAM 👍
Amazing, love it. I can add this getStaticPaths fix into the project, unless you wanna make a PR?
Also I will add some docs around optimize, @otbe thoughts on if we enable this by default? Or you think it should opt in like they have it?
Opt in would be fine for me. The related option of ignoreElementNames
somehow indicates this might be an issue in certain edges only. Sadly the docs do not give us much about side effects.
Feel free to add it, Im about to leave soon :)
Unrleated: there are some console.log in the components that are quite annoying while looking at the build output :D
Thanks the logs have now been removed in the components
With small fixes mentioned in this post I was now finally able to run
npm run build
akaeventcatalog build
My build gets stuck at this point:
After like 5min I get the OOM
This run was executed with the standard memory settings. I was playing with
--max-old-space-size
up to 25GB of RAM but it always fails, maybe a bit later.I tested this with node v20 and v22.
My catalog contains 30 domains (no versions just index.md), 24 services (no versions just index.md) and ~950 events (including versions, schema file and markdown).