search

Tirraa / dashboard_rtm

dashboard_rtm
MIT License
5 stars 1 forks source link

[Prebuilder][Perf] The prebuilder is too greedy #78

Open gustaveWPM opened 9 months ago

gustaveWPM commented 9 months ago

I knew it was going to be a problem. To be honest, in terms of parallelization and speed, there aren't that many bad surprises. But in terms of RAM consumption, we have serious problems.

I found myself benchmarking RAM consumption of 500Mb in stress tests. This is not acceptable, especially as I know that this consumption will increase as the MDX folders get heavier. I guess we need to refactor some parts of the code and use generators/continuations (yield).

The current implementation is enough for a PoC, as it was expected, but does not scale properly (as it was expected too).

Nevertheless, it might require some refactoring work that I'm too lazy to do right now. It also raises questions about how to generate metadatas and then pass them to codegens functions. I think the real RAM consumption problem is here.

I'm removing this issue from any milestone for the time being and will think about it later.

These optimizations issues don't seem to me to be a priority. I will just offer me the luxury of being lazy and telling myself that, for the moment, Vercel's hobby plan provides 8 GB of RAM. I think that until then, there's plenty of time to really design optimizations to deal with real-world issues where this excessive RAM consumption for completely huge websites would become really problematic.

EDIT: The more I think about it, the more I wonder how it could be smartly optimized. The problem of peak RAM consumption can be everywhere: folders traversals, metadatas builders, type generations via ts-morph... I didn't think I'd be opening such a can of worms when I created this issue...

I could be considering that a website as huge as the ones where this consumption could be a problem would inevitably have the budget for an extra 8 GB of RAM on their server, and powerful dev machines, but as a developer, I really don't like this ready-made answer...

NOTE: It's important to keep this issue opened.

gustaveWPM commented 9 months ago

Okay, I guess I should refactor, using yield everywhere.

gustaveWPM commented 8 months ago

https://ts-morph.com/details/generators

(Just a side note)

gustaveWPM commented 8 months ago

https://github.com/piscinajs/piscina

(Just a side note)

gustaveWPM commented 8 months ago

Hmmm... It would also be the opportunity to implement some caching... Gonna think about it!