Improve Performance by adding lazy loading

[Handyfon]

Hi DSPaul,

Hope you're doing well! I wanted to start by saying how much I love the app. However, I have some suggestions to improve the loading times for larger collections:

• Split the collection into smaller "chunks" of 100-200 books and display each chunk before loading the next batch
• Add a page system (my least favourite idea, but probably easy to implement).
• Load a group of books after scrolling down, but only on user input

Thanks for taking the time to consider my ideas. Let me know if you have any questions or need further clarification. ~Handyfon

[Handyfon]

To add to that, my main issue is the loading process itself (the loading of a specific collection) as it freezes the app for a couple of seconds. The performance is actually quite good once they are loaded, it only gets slightly worse after a loooot of books :D

[DSPaul]

I went back and forth on this quite a bit in development and its tough because you kinda have to pick your poison: either you wait for a few seconds for everything to load which is annoying, especially because the entire app freezes but once it loads it's really smooth. The second option is to lazy load which I have tried in the past but that results in a lot of small stutters when scrolling which personally annoys me a lot more so I chose the former. GOG Galaxy 2.0 has these stutters as an example if you want to see what it would be like.

I agree that there is definitely still room for improvement and some layouts are better than others. The card layout for example is awfully slow with collections above a 100 files because each card contains a lot of elements that dynamically resize which slows down rendering considerably so there is also some room for optimisation in that regard.

I think that if I could find a way to move the rendering of the layout to a separate thread from the main UI thread so it stops freezing the entire app, and maybe show a loading animation while its busy, it would greatly improve the feel of the app to the point where it's not a problem anymore, even if it takes equally as long to load on paper.

Maybe I could also make the resolution of the thumbnails configurable so you can lower it for bigger collections which also helps loading times a bit. I think a hybrid system where it lazy loads but in big 100 file blocks as you suggested could be a good option but I wouldn't really know how to implement that so I would have to do some more research on how the UI rendering pipeline works and all that stuff.

I also have to confess that I was not prepared for the massive size of some people's libraries. I have around 150 pdfs in my DnD Collection and I thought that was a lot and then day one I get a github issue from a guy trying to import 2000 files and I already saw a guy comment on the reddit post saying he has close to 10.000 files which I assume includes a lot of creature token and such but still. I only optimized to a degree where it could handle the size of my library but for these massive collections, it's just plain not good enough.

[fantom87]

I confess, I know little about coding in general, but for the loading issue with large library sizes, would it help to have a cache in a different drive than the files? What I mean by that is for me, I have my collection on a standard SATA drive, and contains hundreds of thousands of books. Would the program be able to have the cache or whatever it is in say the AppData directory? I know for me it would certainly load the cards much faster since that's on my main M.2 drive.

[DSPaul]

Not really, COMPASS only stores the path to the files along with some other metadata and not the files themselves, those are only accessed/loaded when you try to open them. The only files that do need to be loaded are the images for the covers but these already get stored in the AppData directory.