search

echodot-llc / Red

3 stars 0 forks source link

Hangs if db has too many keys #16

Open blueberry6401 opened 2 years ago

blueberry6401 commented 2 years ago

My test Redis db has over 7 millions keys. When open, app takes too much time to load and while it's loading, it hangs. Is it running SCAN? If so, can I set keys limit to load when open app?

scuml commented 2 years ago

Confirming the problem on my end. I test up to 2.5M keys, and while it should take 10-20 seconds to load that on an M1, it's beachballing right now. I'll check into what's causing the hang up. Thanks for alerting me to the problem.

Handling large databases has been one of the biggest challenges with development. The application runs multiple scan calls and then uses threads to asynchronously create the objects used in the sidebar. After all that is done, a sort operation is run - and this is usually what hangs the system. Another problem can occur after the sort, as Mac OS then has to handle rendering all 7M items in the sidebar.

One optimization the app implements is that is skips sort in subdirectories - so if you organize your data using separated parent:child:key format, the app won't bother sorting those items. Are you using any sort of key format to organize your data? Do you use slashes? Or is it a long flat list?

blueberry6401 commented 2 years ago

Yep I have those keys in parent:child:key format. I think the basic optimization is limit the key to scan. If keys count exceeds like 100,000 keys, it should stop and show load more. Because most of the time I'll use filter instead of browsing through the sideview.

scuml commented 2 years ago

This should perform much better with version 1.7.6 - available now on echodot.com/red and tomorrow on the Mac App Store. Still will take a while to load initially, but the hanging behavior should be resolved.