search

mopidy / mopidy-local-sqlite

DEPRECATED (Mopidy SQLite local library extension)
https://mopidy.com
Apache License 2.0
30 stars 10 forks source link

Incremental commit of library updates #77

Closed palfrey closed 8 years ago

palfrey commented 8 years ago

So, I've got a large library (~80k tracks) on a disk attached to a Raspberry Pi. When trying to do local scan with mopidy-local-sqlite it gets to the end of the scan and then crashes with OOM. I guessed this was due to a large number of updates in one go, and so I made a patch to make it commit every so many changes and this fixed the problem.

tkem commented 8 years ago

Did you try setting local/scan_flush_threshold = 100 or an even lower value? This should have a similar effect.

palfrey commented 8 years ago

local/scan_flush_threshold has been 100 from the start with this device, and no such luck. Haven't tried lower values yet.

tkem commented 8 years ago

Strange, this should have the same effect as your patch: When running a local scan, every local/scan_flush_threshold tracks flush() is called, which does the commit. So setting this to a lower value would commit smaller batches more often. What value did you set changes_per_commit to to get it working? Try the same (or maybe a slightly lower) value for local/scan_flush_threshold.

tkem commented 8 years ago

@palfrey: Did you check whether setting local/scan_flush_threshold has some effect?

palfrey commented 8 years ago

I'd set to changes_per_commit to 100 as well (randomly chosen pretty much). I've now been running without my changes from here, and all appears fine, so I'm going to close this for the moment and retry with lower scan_flush_threshold if I hit this again. Thanks for your time!

tkem commented 8 years ago

Just reopen this if the problem appears again and playing with scan_flush_threshold doesn't show the desired effect. Thanks for your efforts! If you're interested in hacking mopidy-local-sqlite, there's also a bunch of open issues, and I'm afraid I won't have much time to spend on this any more... ;-)

palfrey commented 8 years ago

Thanks. I probably won't directly have time, these were just the items affecting me while getting Mopidy setup locally, but if I ever get bored I'll have a look :)

tkem commented 8 years ago

OK. I'll just wait until you get annoyed by m-l-s only returning tracks and no albums or artists when searching your library;-)