80% of cpu - Githubissues

zoic21 commented 12 years ago

I have two problems with headphones :

sometimes headphones use 80% of cpu (2 cores at 2.4ghz) and lot of memory
and sometimes headphones auto shutdown I don't know why there is nothing in log

What can I do to help you to solve this.

rembo10 commented 12 years ago

On develop branch?? On Aug 7, 2012 9:18 PM, "cptjhmiller" notifications@github.com wrote:

Just to report that cpu usage did return to normal after the scan completed.

2012-08-07 11:52:15 INFO Scanning music directory: /pool/NAS/Audio/My Music 2012-08-07 14:17:39 INFO Completed scanning directory: /pool/NAS/Audio/My Music

2hrs 25 mins for 40,321 files. Thats around 5 seconds per file.

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-7557551.

cptjhmiller commented 12 years ago

yep, dual core running at 3Ghz and 4Gb ram.

rembo10 commented 12 years ago

That's pretty slow. It took mine 3 minutes for 4000 songs. On an atom netbook. Something is definitely wrong - that seems normal for the master branch though On Aug 7, 2012 9:24 PM, "cptjhmiller" notifications@github.com wrote:

yep, dual core running at 3Ghz and 4Gb ram.

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-7557750.

cptjhmiller commented 12 years ago

Version: 7bbdeeac619c33dd94cb2d871843941dc88915e5

Taken from bottom of page, matches latest git version.

Begall commented 12 years ago

Could the slow part not be scraping from Musicbrainz? It's not exactly the fastest service...

Enverex commented 12 years ago

Headphones is currently using between 70 and 180% on my quad core Ivybridge (i5 3570) whilst doing a scan. I assume that's normal for the scanning? (I couldn't tell from this bug title whether the issue was with idle CPU usage or scanning CPU usage).

The scan has also been running for ~24 hours now which seems a little extreme (using your VIP server) which also seems a little mad (~550 artists, ~17000 tracks, all FLAC, all correctly tagged and filed via Musicbrainz Picard). Still going now. No errors from what I can see.

cptjhmiller commented 12 years ago

Enverex, when you say its still scanning do you mean "Scanning music directory /folder" or "Now adding/updating Artist Name" as for me with around the same number of tracks the latter does take awhile, where as the fisrt with the latest version of headphones takes around 25 mins.

I was getting high cpu while scanning untill i did a clean install of the latest version and let Headphones build a brand new database. Normaly i just used a backed up database files so as to avoid a total re scan. Does any of this match with you?

Now with "Scanning music directory /folder" i get a max of 3% cpu while doing this scan and with "Now adding/updating Artist Name" i get a max of around 40% but this goes up and down as you would expect due to the nature of scanning/updating.

Enverex commented 12 years ago

The last entries in the log were:

2012-08-15 11:05:39 INFO Scanning music directory: /mnt/store/Music 2012-08-15 11:05:28 INFO Updating artist information from Last.fm 2012-08-15 11:05:28 INFO Rodrigo y Gabriela is already in the database. Updating 'have tracks', but not artist information 2012-08-15 11:05:28 INFO Пётр Ильич Чайковский is already in the database. Updating 'have tracks', but not artist information 2012-08-15 11:05:28 INFO Cannot determine the best match from an artist/album search. Using top match instead 2012-08-15 11:05:27 INFO Found an artist with a disambiguation: Peter Ilyich Tchaikovsky - doing an album based search

It's now 11:19 so those entries are nearly 15 minutes old.

It's currently doing a forced scan of my music folder since I switched to the Headphones mirror, as when I first set it up with the Musicbrainz official server it ended up with only about half the artists added and almost no albums (literally only around 10).

At no point have I seen it drop below 50% CPU usage, it's normally around 70 and has peaked around 180% (so nearly maxing out 2 cores).

Think it's worth me starting with a clean DB?

cptjhmiller commented 12 years ago

Its worth a try. You could backup your current DB files anyway then (i would) delete base folder then reinstall, add config settings then just do the scan and watch top in terminal to check cpu. This seems to have solved cpu issues i was getting and like you were very high and made using headphones scary.

rembo10 commented 12 years ago

I would do that do. From your log it looks like it finished one scan and started another one right after (probably because the scan took longer than your library_scan_interval set in the config - I'd also increase that to run every other day or something)

On Aug 15, 2012, at 6:34 PM, cptjhmiller notifications@github.com wrote:

Its worth a try. You could backup your current DB files anyway then (i would) delete base folder then reinstall, add config settings then just do the scan and watch top in terminal to check cpu. This seems to have solved cpu issues i was getting and like you were very high and made using headphones scary.

— Reply to this email directly or view it on GitHub.

Enverex commented 12 years ago

I've cleared the DB and I've started again. CPU usage was around 10% for some time but is now up to 35% or so.

My libraryscan_interval is set to 0 which I assumed disabled it. Please correct me if I'm wrong.

Enverex commented 12 years ago

I've set the libraryscan_interval to a day and now the scan is using anywhere between 0% and 50% (most often around 12% when it shows up).

Alfiegerner commented 12 years ago

Following the suggestions here I just tried moving my old DB and have it start again: good news - it worked :)

Not sure if it was some sort of DB corruption causing the lag, but I'm running off master and CPU was very tolerable. Will let you know if it plays up again.

Headphones rockkkkkks. Thanks rembo.

zoic21 commented 12 years ago

Maybe I found something : When I'am on the home page I got this : 3465 headphon 20 0 1094m 148m 3628 S 175 7.5 8:58.63 python (175% of CPU is in use!!!!!) But if I change page the cpu charge decrease (60-80% but headphones is scanning), I thing the over charge it's du to generation of miniature.

For information I'am on dev branch

rasmuseeg commented 12 years ago

Just a quick note: I have the "same" problem on my mediecenter, with default theme. I found that the loading icon, stops and everything is very slow. Then i change to another theme, and everything runs fine.

zoic21 commented 12 years ago

I run the last version of devellopement branch since 3 days and the probleme seem solved.

Enverex commented 11 years ago

Rembo: I've just nuked my library again and I'm watching what's going on (this machine has an i5-3570K).

During the scanning of files to add to the library, CPU usage was around 4%.

It's now getting the artist information (Retrieving artist information, Now adding/updating, Seeing if we need album art for, etc) and CPU usage is between 60-100%.

Should it be that high? I'm not sure what it could be doing that would cause CPU usage that high. Here's a bit from the log whilst it's using 100%:

adrianmoisey commented 11 years ago

My headphones is currently using 100% CPU. Last item in the logs were: 23-Oct-2012 14:22:05 - INFO :: Thread-385 : Found 3229 tracks in: '~/Music/Sorted/'. Matching tracks to the appropriate releases....

That was 20 minutes ago and the CPU is high.

Alfiegerner commented 11 years ago

Try backing up the db and starting again with a new one, worked for me.

fullbright commented 11 years ago

Same thing here. I'll try to backup the db and start over.

rembo10 commented 11 years ago

Guys - i'm going to start looking into the performance issues more fully. I think it happens with massive libraries, since every db commit checks the whole table (which can get quite big). Going to try to cut down the commits by ~90%

ghost commented 11 years ago

Although it would be fine, if you can speed up the search.

For example: Search only for Artist and parse the results for the needed albums. It reduce the API calls and traffic extreme. (for Backlog)

theguardian commented 11 years ago

Just wanted to chime in to say that this is happening to me as well.

Headphones Master Branch (15b9d98d7c44027449ded40491e11a96f6f4a1f7) Library Size: 19,000 tracks Server: VIP Hardware: QNAP TS-509, 3GHz Dual-Core CPU / 4GB RAM Python 2.7 installed, but Headphones seems to prefer using Python 2.6

Software Actions: "Update Active Artists" burns 100% CPU of both cores for 14 hours "Scan Music Library" takes ~12% CPU of a single core "Matching Tracks" burns 100% CPU for 10 hours

Trashing the database definitely sped up the process after upgrading to a new commit about a month ago, but CPU load has been this way for the 4-5 months I've been using Headphones.

rembo10 commented 11 years ago

Yeah its happening to me too. Its all the database commits. Working on a fix now

On Nov 13, 2012, at 4:57 PM, theguardian notifications@github.com wrote:

Just wanted to chime in to say that this is happening to me as well.

Headphones Master Branch (15b9d98) Library Size: 19,000 tracks Server: VIP Hardware: QNAP TS-509, 3GHz Dual-Core CPU / 4GB RAM Python 2.7 installed, but Headphones seems to prefer using Python 2.6

Software Actions: "Update Active Artists" burns 100% CPU of both cores for 14 hours "Scan Music Library" takes ~12% CPU of a single core "Matching Tracks" burns 100% CPU for 10 hours

Trashing the database definitely sped up the process after upgrading to a new commit about a month ago, but CPU load has been this way for the 4-5 months I've been using Headphones.

— Reply to this email directly or view it on GitHub.

prophetizer commented 11 years ago

i take it this is still being worked on? terminal windows stops at Preparing to write metadata to tracks...... and then CPU spike that never ends

update - after while it posted this - WARNING:lib.apscheduler.scheduler:Execution of job "headphones.postprocessor.checkFolder (trigger: interval[0:05:00], next run at: 2012-11-29 09:35:00.302645)" skipped: maximum number of running instances reached (1)

and still high CPU

fullbright commented 11 years ago

Hi

Just to say that I'm still having issues with my install, CPU still at 100%.

Thank for your work, you are doing great.

rembo10 commented 11 years ago

Yeah - this is my #1 priority right now. It's taking a bit longer than I had anticipated though, with a bunch of kinks still to work out

On 11/30/12, fullbright notifications@github.com wrote:

Hi

Just to say that I'm still having issues with my install, CPU still at 100%.

Thank for your work, you are doing great.

Reply to this email directly or view it on GitHub: https://github.com/rembo10/headphones/issues/645#issuecomment-10885899

prophetizer commented 11 years ago

it's caused by the busted post processing

prophetizer commented 11 years ago

i assume none of the recent updates have fixed this issue?

ljunkie commented 11 years ago

Issue is not resolved, just came here to find out if anyone else had noticed this issue and for a quick/easy fix. Looks like I will try and take a look.

danepowell commented 11 years ago

I think there are a lot of related/duplicate issues on this, here's the most recent: https://github.com/rembo10/headphones/issues/990

danepowell commented 11 years ago

I deleted the database and allowed Headphones to recreate it, now database updates seem to be working fine.

prophetizer commented 11 years ago

still using 90% cpu when scanning library, i don't even have that large of a library, i mean i do about 100GB, but only have 26GB in itunes of music, not sure why this is still a big issue when it's been reported for close to a year. anything i can do to help get this resolved?

ogg1e commented 11 years ago

Mine still does it as well. Every time it scans the library.

FleischKarussel commented 11 years ago

Hello there, I got that problem too. headphones uses 90-100% cpu and 278M memory. my music library is about 146G.

VIRT RES CPU% MEM% PID USER NI S TIME+ IO_R/s IO_W/s NAME 1.5G 278M 91.0 15.8 22207 headphone 0 S 27:49.34 0 7K python2

kierr commented 11 years ago

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 30660 rtorrent 20 0 1163M 184M 3464 R 104. 10.6 2h33:34 /usr/bin/python Headphones.py --quiet --daemon --nolaunch --config /home/rto 1340 rtorrent 20 0 1163M 184M 3464 S 104. 10.6 2h37:17 /usr/bin/python Headphones.py --quiet --daemon --nolaunch --config /home/rto

I too am struggling with ridiculous CPU usage, my library has been scanning for days now (700 artists updating) with 100% CPU on all cores.

pi_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Freddie+Gibbs 29-Mar-2013 11:19:24 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=The+Miseducation+of+Freddie+Gibbs&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Freddie+Gibbs 29-Mar-2013 11:19:24 - INFO :: Thread-169 : Seeing if we need album art for: Freddie Gibbs 29-Mar-2013 11:19:24 - INFO :: Thread-169 : Updating complete for: Freddie Gibbs 29-Mar-2013 11:19:25 - INFO :: Thread-169 : Now adding/updating: Angus & Julia Stone 29-Mar-2013 11:19:25 - INFO :: Thread-169 : Now adding/updating: Down the Way 29-Mar-2013 11:20:07 - INFO :: Thread-169 : Seeing if we need album art for Down the Way 29-Mar-2013 11:20:07 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=Down+the+Way&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Angus+%26+Julia+Stone 29-Mar-2013 11:20:09 - INFO :: Thread-169 : Now adding/updating: A Book Like This 29-Mar-2013 11:20:19 - INFO :: Thread-169 : Seeing if we need album art for A Book Like This 29-Mar-2013 11:20:19 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=A+Book+Like+This&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Angus+%26+Julia+Stone 29-Mar-2013 11:20:19 - DEBUG :: Thread-169 : No album summary found from: http://ws.audioscrobbler.com/2.0/?album=A+Book+Like+This&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Angus+%26+Julia+Stone 29-Mar-2013 11:20:19 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=A+Book+Like+This&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Angus+%26+Julia+Stone 29-Mar-2013 11:20:19 - INFO :: Thread-169 : Seeing if we need album art for: Angus & Julia Stone 29-Mar-2013 11:20:19 - INFO :: Thread-169 : Updating complete for: Angus & Julia Stone 29-Mar-2013 11:20:21 - INFO :: Thread-169 : Now adding/updating: Camo & Krooked 29-Mar-2013 11:20:21 - INFO :: Thread-169 : Now adding/updating: Between the Lines 29-Mar-2013 11:20:33 - INFO :: Thread-169 : Seeing if we need album art for Between the Lines 29-Mar-2013 11:20:33 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=Between+the+Lines&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:33 - DEBUG :: Thread-169 : No album summary found from: http://ws.audioscrobbler.com/2.0/?album=Between+the+Lines&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:33 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=Between+the+Lines&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:33 - INFO :: Thread-169 : Now adding/updating: Cross the Line 29-Mar-2013 11:20:45 - INFO :: Thread-169 : Seeing if we need album art for Cross the Line 29-Mar-2013 11:20:45 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=Cross+the+Line&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:45 - DEBUG :: Thread-169 : No album summary found from: http://ws.audioscrobbler.com/2.0/?album=Cross+the+Line&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:45 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=Cross+the+Line&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:20:46 - INFO :: Thread-169 : Now adding/updating: Above & Beyond 29-Mar-2013 11:21:09 - INFO :: Thread-169 : Seeing if we need album art for Above & Beyond 29-Mar-2013 11:21:09 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=Above+%26+Beyond&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:21:09 - DEBUG :: Thread-169 : No album summary found from: http://ws.audioscrobbler.com/2.0/?album=Above+%26+Beyond&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:21:09 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=Above+%26+Beyond&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Camo+%26+Krooked 29-Mar-2013 11:21:10 - INFO :: Thread-169 : Seeing if we need album art for: Camo & Krooked 29-Mar-2013 11:21:10 - INFO :: Thread-169 : Updating complete for: Camo & Krooked 29-Mar-2013 11:21:11 - INFO :: Thread-169 : Now adding/updating: Band of Skulls 29-Mar-2013 11:21:11 - INFO :: Thread-169 : Now adding/updating: Sweet Sour 29-Mar-2013 11:21:26 - INFO :: Thread-169 : Seeing if we need album art for Sweet Sour 29-Mar-2013 11:21:26 - DEBUG :: Thread-169 : Retrieving artist information from: http://ws.audioscrobbler.com/2.0/?album=Sweet+Sour&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Band+of+Skulls 29-Mar-2013 11:21:27 - DEBUG :: Thread-169 : No album summary found from: http://ws.audioscrobbler.com/2.0/?album=Sweet+Sour&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Band+of+Skulls 29-Mar-2013 11:21:27 - DEBUG :: Thread-169 : No album infomation found from: http://ws.audioscrobbler.com/2.0/?album=Sweet+Sour&format=json&api_key=690e1ed3bc00bc91804cd8f7fe5ed6d4&method=album.getInfo&artist=Band+of+Skulls 29-Mar-2013 11:21:27 - INFO :: Thread-169 : Now adding/updating: Baby Darling Doll Face Honey

I'm probably going to have to uninstall headphones which is a big shame. My system is really quite powerful, clearly there is some kind of bug somewhere. I can be watching 1080p on XBMC whilst updating my very large library, doing a full sickbeard/couchpotato update, whist verifying 2000 rtorrent torrents and headphones still needs more resources than all of them combined for something as simple as updating artist/album information.

I hope a fix comes around soon.

rembo10 commented 11 years ago

It's something to do with the sqlite queries - that much I know. Everytime it does a commit, it basically has to reverify everything in the database, so with a large library, it uses a lot of resources. So far I've come up with a couple solutions:

Give the option of using mysql if you have a big library, but then that will require running a mysql server
Do a commit every 100 or so changes during the library update. The problem I'm running into with this is having multiple threads trying to update the database at the same time, and geting "database is locked" error.

I think the best solution might be to build a queue for database entries, so rather than any given thread trying to hit the db directly, it goes to the queue first and then entered, and then committed every 100 change or so, or upon shutdown. Hopefully this will prevent the system from getting bogged down.

sgdesmet commented 11 years ago

Personally I'd go for giving the option of using mysql. It's not that difficult to install and configure, and I think a fair share of users may already have one running anyway. It's also more robust than SQLite.

Also, as developer you'll have less headaches compared to implementing a queue yourself, which comes with issues such as concurrency, deadlocks, possible unexpected database state after a crash and so on, while you'll pretty much be reimplementing something already available in MySQL.

Also thanks for the great work on HeadPhones ;)

rembo10 commented 11 years ago

yeah that might be the better option. there's already a branch that has mysql support so i might start testing that. it's a pretty obnoxious bug though - and is my top priority (for realz this time)

On Mon, May 13, 2013 at 5:18 PM, Stein Desmet notifications@github.comwrote:

Personally I'd go for giving the option of using mysql. It's not that difficult to install and configure, and I think a fair share of users may already have one running anyway. It's also more robust than SQLite.

Also, as developer you'll have less headaches compared to implementing a queue yourself, which comes with issues such as concurrency, deadlocks, possible unexpected database state after a crash and so on, while you'll pretty much be reimplementing something already available in MySQL.

Also thanks for the great work on HeadPhones ;)

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-17803401 .

cptjhmiller commented 11 years ago

Even tho I have a MySQL server running I still think it should be kept as SQLite. Using MySQL might reduce the number of users who could run headphones as not all devices can install/run a MySQL server, things like routers, Rpi and other low power devices.

For me personally it does not matter but for some it could be a big hurdle.

rembo10 commented 11 years ago

It would be optional for those with large libraries

On Mon, May 13, 2013 at 5:32 PM, Jamie notifications@github.com wrote:

Even tho I have a MySQL server running I still think it should be kept as SQLite. Using MySQL might reduce the number of users who could run headphones as not all devices can install/run a MySQL server, things like routers, Rpi and other low power devices.

For me personally it does not matter but for some it could be a big hurdle.

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-17804008 .

FleischKarussel commented 11 years ago

I really appreciate the idea of using MySQL/MariaDB as base for headphones. How long do you estimate does the implementation take? Is there a working branch we may be able to test? btw: Thank you for your effort, I already donated to headphones mb server. :)

DynBob commented 11 years ago

Yeah I've had issues when getting over 300~ artists, moving the .db over to an SSD helped a bit, but it still is noticeably slow to load an artist's page etc.

Will love it if it becomes an option for those with larger libraries, will most likely donate once more since this is really useful software, and hopefully will keep on improving. I'm even considering trying to get into python so I can try and help out over the summer but I probably won't be well skilled and my helping will only hurt instead!

tazmad commented 11 years ago

Hi All,

First off many thanks for all your hard work with this program - it is awesome! Just wanted to know if there was a branch that was already setup to test / use MySQL? I already have a MySQL server set up for various other bits of software I am running and also have some basic db monitoring setup so would be interested to see if moving to that would help this CPU issue (which is causing me to leave headphones off more than it is on).

Thanks

nozn commented 11 years ago

It may seem counter intuitive, but turning on WAL resolved this issue for me. The cpu usage comes from individual threads spinning while trying to gain access to the db, but using WAL is a simple way to gate access at a lower level.

Change headphones/db.py line 53 to: self.connection.execute("PRAGMA journal_mode = wal")

DynBob commented 11 years ago

Will try that out tonight, thanks! Haven't been using Headphones for a while due to this issue. Also by any chance do you know what the Headphones Indexer is under Search Providers?

nozn commented 11 years ago

BTW -- this doesn't keep the program from doing the cpu and io intensive tasks associated with the full library and artist scans. Reading the id3 tags from [tens of] thousands of files and updating that info in sqlite still takes a while. What I'm no longer seeing is the cpu usage without any corresponding io.

I'm guessing it's a newznab compatible indexer, probably optimized for headphones users as a perk for being a VIP.

rembo10 commented 11 years ago

This might be exactly what I've been looking for. Have no idea how i missed it. It seems like the only case it wouldn't work would be if you're running headphones with the data directory on a network share?

Will start playing around with this. Thank you On Jul 23, 2013 1:18 AM, "normzbig" notifications@github.com wrote:

BTW -- this doesn't keep the program from doing the cpu and io intensive tasks associated with the full library and artist scans. Reading the id3 tags from [tens of] thousands of files and updating that info in sqlite still takes a while. What I'm no longer seeing is the cpu usage without any corresponding io.

I'm guessing it's a newznab compatible indexer, probably optimized for headphones users as a perk for being a VIP.

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-21370473 .

nozn commented 11 years ago

Accessing a sqlite db over a network share is a bad idea on so many levels... but yeah WAL doesn't work without shared memory.

Now if you can change the logic so that a full scan doesn't update every single row in the db... the "upserts" in librarysync never have the opportunity to not run a query because the controlValueDict isn't populated from the db data that would be overwritten. So even when nothing changes, an update is always executed that's a good chunk of io that can be eliminated.

Another way to reduce load would be to store a signature of the file along with the track in the db -- location + size + mtime is probably enough -- and don't even bother reading the id3 if the file has been scanned before.

rembo10 commented 11 years ago

Nice one. Will look into it. Cleaning up the code to be more efficient makes the most sense to me. The reason I was scanning the files every time was to see if any of the metadata changed, which wouldn't necessarily change the file size, which is basically the only way I thought to compare. Haven't looked at this in a while. Fastest would probably to see if I can grab a last modified date or something.

Anyways - do you have any interest in helping me out on this more actively? There's a lot of stuff I'm just totally unaware of and it would be a great help. I want to get this problem sorted once and for all

Thanks! On Jul 23, 2013 10:19 AM, "normzbig" notifications@github.com wrote:

Accessing a sqlite db over a network share is a bad idea on so many levels... but yeah WAL doesn't work without shared memory.

Now if you can change the logic so that a full scan doesn't update every single row in the db... the "upserts" in librarysync never have the opportunity to not run a query because the controlValueDict isn't populated from the db data that would be overwritten. So even when nothing changes, an update is always executed that's a good chunk of io that can be eliminated.

Another way to reduce load would be to store a signature of the file along with the track in the db -- location + size + mtime is probably enough -- and don't even bother reading the id3 if the file has been scanned before.

— Reply to this email directly or view it on GitHubhttps://github.com/rembo10/headphones/issues/645#issuecomment-21393033 .

rembo10 / headphones

80% of cpu #645