Throne3d / 2epub-various-ruby

A scraper of various glowfic collections
1 stars 0 forks source link

Reduce RAM consumption #12

Open Throne3d opened 7 years ago

Throne3d commented 7 years ago

Don't keep serialized data in memory. Export it to a .json file, have individual ones for each chapter, get it on demand, provide a pseudo-array for chapters or something (keep authors in memory, they're small).

Yeah, and dirtying is memory-intensive, ew.

This was attempted in the add-model-upgrade branch but for some reason caused an infinite JSONifying loop? Fix merge conflicts and check it out, since it could improve things quite neatly.

(Actually seems like that branch had other issues such as being slower and not actually clearing stuff appropriately?)