search

shermp / Kobo-UNCaGED

UNCaGED, for Kobo devices
GNU Affero General Public License v3.0
95 stars 7 forks source link

Memory usage improvements #41

Closed shermp closed 3 years ago

shermp commented 3 years ago

This PR makes a few changes to how metadata is handled to hopefully reduce needed data structures, and to allocate as much memory upfront as practical without completely overhauling how I do things.

I've also reworked how I do image resizing, I now wait for the thumbnail generation/resizing to complete before getting the next book from Calibre. The old code had no limits on the number of goroutines that could be created, which could make memory usage explode when sending a large number of books (as sending books is almost always quicker than generating thumbnails), and probably introduced CPU usage contention as well.

Also made debug log statements a bit nicer.

@pgaskin if/when you have a moment, would you mind doing a quick code review please?

shermp commented 3 years ago

Thanks for the review @pgaskin

I can't seem to make that slow SQL query any quicker so not going to worry about it for now. Only other change is I've over allocated the initial metadata map by 10% to potentially avoid a later exponential allocation when adding new books (of course, if lots of new books are added, this could still happen).