search

jkomoros / card-web

The web app behind thecompendium.cards
Apache License 2.0
52 stars 8 forks source link

OOMs are starting to happen in production more often #522

Open jkomoros opened 3 years ago

jkomoros commented 3 years ago

Just due to the size of production card set.

jkomoros commented 3 years ago

Find dialogs seem to cause slowdowns, as does opening up the card collection tag cloud (especially for very large card collections)

jkomoros commented 3 years ago

Fingerprint generator seems to be the largest

makeSortInfo for rendering find dialog seems to be expensive too

jkomoros commented 3 years ago

There don't appear to be any detached DOM retained

The regexp cache for words is large (but not so large as to cause a problem). Specifically wordBoundaryRegExp, which is run for... every ngram?. That's inside of ngramWithinOther, which is extremely hot. I wonder if the answer is not so much to save the regexp, but to save the result of that combination. (How many combinations are there?)

jkomoros commented 3 years ago

Things are starting to OOM every time I edit in production, creating a concept card, editing it. etc

jkomoros commented 3 years ago

On main chrome and canary. Maybe something to do with redux tools?

jkomoros commented 3 years ago

Disabling redux/reselect tools seems to help a teensy bit not but enough, it still OOMs.

I guess the problem happens when there are two fingerprint generators live at the same time, two sets of cards, etc. Everything is (temporarily) doubled. (Which happens when a card is created or edited)

jkomoros commented 3 years ago

If fingerprintgenerator is where a lot of memory is spent (it holds an entire collection of cards!), in memoizedFingerprintGenerator, we keep 3 entries, but we only really need one (maybe two). Maybe drop that down?

Nope, still not enough.

jkomoros commented 3 years ago

The problem only happens for admins, who can see all of the thousands of working notes cards as well. A workaround is to temprorarly set a higher V8 memory space? https://stackoverflow.com/questions/17491022/max-memory-usage-of-a-chrome-process-tab-how-do-i-increase-it/32671934#32671934

jkomoros commented 3 years ago

A huge portion of the memory is in the wordBoundaryRegExp cache. ngramWithinOther is being called not with every ngram that is a concept string (which would be ~1000 but every single ngram in any card (!!) which is closer to 6000. I'm still not sure why it's taking so much memory compared to everything else? 60000 regular expressions really shouldn't be that bad...