search

rnordstrom / LobitaDownloader

Media downloader for use with LobitaBot.
0 stars 0 forks source link

Improve backup memory-management #64

Closed rnordstrom closed 3 years ago

rnordstrom commented 3 years ago

Since day one, the application has had issues with high memory usage. This is largely an effect of keeping the entirety of the backup files in memory, alongside the entirety of the indices. Perhaps a better solution would be to keep the barebones, pre-download index-files in memory, while reading and writing related data to another document in a simple, forward-only manner? A batched approach would be most effective, though it may cause some issues when writing to the database.

rnordstrom commented 3 years ago

This was a nice idea, but it appears that the writing-portion of the requisite set of operations would be cumbersome at best and impossible at worst. For now, I'll just have to live with the high memory usage.

Perhaps in the future, I can consider spitting the content up into several files.

rnordstrom commented 3 years ago

Splitting things up is indeed what I did. Let's give this one a final go.