Python uses 100+% CPU while matching songs by artist

[nlsko]

The scan takes multiple minutes/artist sometimes, I am using a Musicbrainz mirror on a VM, is it because of database size?

top output: 3065 xbmc 20 0 1372m 333m 4188 S 101 4.4 516:07.91 python

it always happens during this phase, after a scan of the music directory: 2014-04-10 05:17:13 INFO Now matching songs by Enthroned 2014-04-10 05:17:11 INFO Now matching songs by Enter Shikari 2014-04-10 05:16:10 INFO Now matching songs by Enslaved 2014-04-10 05:16:05 INFO Now matching songs by Ensemble Ambrosius 2014-04-10 05:13:41 INFO Now matching songs by Ennio Morricone 2014-04-10 05:11:50 INFO Now matching songs by Enigma 2014-04-10 05:10:45 INFO Now matching songs by Endstille 2014-04-10 05:09:35 INFO Now matching songs by En Vouge 2014-04-10 05:04:50 INFO Now matching songs by Emperor

Details: Ubuntu 12.04 Python 2.7.3 latest HP via git my collection: +-1500 Artists/5000 Albums/76000 songs

The database grew quite large since yesterday: -rwxr--r-- 1 xbmc nico 679M Apr 10 05:19 headphones.db -rwx------ 1 xbmc xbmc 1.1M Apr 10 05:24 headphones.db-wal -rwx------ 1 xbmc xbmc 32K Apr 10 05:24 headphones.db-shm

if you need more info, please let me know, keep up the awesome work :)

[basilfx]

Did you try running with the verbose/debug option turned on? It should provide more information about the things happening during a scan (so between the log lines).

The log page does not display the debug information, so it's better to copy it from the log file itself, or the console.

[nlsko]

How do I turn on this option?

[basilfx]

Pass the --verbose option when starting Headphones. Make sure you dont combine it with --daemon, so it will log tot console.

[nlsko]

ok thnx, it's scanning now..

[nlsko]

There is nothing to see even with --verbose.. see attached screenshot, as soon as the phase "Now matching songs by artist" starts the CPU goes up till 100+% and stays there, even whn the scan is finished.

[nlsko]

10-Apr-2014 16:54:25 - INFO :: MainThread : Checking to see if the database has all tables.... 10-Apr-2014 16:54:25 - DEBUG :: MainThread : Trying to execute: "git rev-parse HEAD" with shell in /home/xbmc/headphones 10-Apr-2014 16:54:25 - DEBUG :: MainThread : Git output: d3727d1fef25b57df972cbf38566062ecf2a6682 10-Apr-2014 16:54:25 - DEBUG :: MainThread : Trying to execute: "git rev-parse --abbrev-ref HEAD" with shell in /home/xbmc/headphones 10-Apr-2014 16:54:25 - DEBUG :: MainThread : Git output: master 10-Apr-2014 16:54:25 - INFO :: MainThread : Retrieving latest version information from GitHub 10-Apr-2014 16:54:25 - DEBUG :: MainThread : Requesting URL via GET method: https://api.github.com/repos/rembo10/headphones/commits/master 10-Apr-2014 16:54:26 - DEBUG :: MainThread : Latest version is 432354dcd1af3ae02100b1100730d4db29522c07 10-Apr-2014 16:54:26 - INFO :: MainThread : Comparing currently installed version with latest GitHub version 10-Apr-2014 16:54:26 - DEBUG :: MainThread : Requesting URL via GET method: https://api.github.com/repos/rembo10/headphones/compare/d3727d1fef25b57df972cbf38566062ecf2a6682...432354dcd1af3ae02100b1100730d4db29522c07 10-Apr-2014 16:54:26 - DEBUG :: MainThread : Response status code 200 is not white listed, raising exception 10-Apr-2014 16:54:26 - DEBUG :: MainThread : In total, 0 commits behind 10-Apr-2014 16:54:26 - INFO :: MainThread : Headphones is up to date 10-Apr-2014 16:54:26 - DEBUG :: MainThread : Using the following server values: MBHost: 192.168.66.75, MBPort: 5000, Sleep Interval: 0 10-Apr-2014 16:54:26 - INFO :: MainThread : Starting Headphones on http://0.0.0.0:8181/

[nlsko]

just found out: When it downloads artists from my Last.fm user and updates their albums it also practically stays at 100% cpu, is that normal?

update: it's already on Tread-100 and CPU stays between 160-175% during artists updates, looks like treads are not reused, is that correct?

top - 19:32:24 up 1 day, 18:19, 4 users, load average: 6.33, 5.95, 4.87 Tasks: 347 total, 1 running, 346 sleeping, 0 stopped, 0 zombie Cpu(s): 79.8%us, 18.9%sy, 0.2%ni, 1.0%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 7770648k total, 7571836k used, 198812k free, 45660k buffers Swap: 7974428k total, 54664k used, 7919764k free, 1037672k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16475 root 20 0 1384m 564m 6292 S 177 7.4 168:40.87 python
2975 xbmc 20 0 1723m 157m 30m S 18 2.1 517:33.33 xbmc.bin

[nlsko]

just wondering, is this the same as this issue? https://github.com/rembo10/headphones/issues/645

all the symptoms look alike in my opinion, but this one should have been solved right?

in the meantime it looks my db is growing out of proportion for only 80.000 songs..

-rwxr--r-- 1 root root 672K Apr 10 19:58 headphones.db-shm -rwxr--r-- 1 root root 83M Apr 10 19:58 headphones.db-wal -rwxr--r-- 1 xbmc root 763M Apr 10 19:58 headphones.db

[rembo10]

Yeah, that seems pretty huge

On Apr 10, 2014, at 11:04 AM, nlsko notifications@github.com wrote:

just wondering, is this the same as this issue? #645

all the symptoms look alike in my opinion, but this one should have been solved right?

in the meantime it looks my db is growing out of proportion for only 80.000 songs..

-rwxr--r-- 1 root root 672K Apr 10 19:58 headphones.db-shm -rwxr--r-- 1 root root 83M Apr 10 19:58 headphones.db-wal -rwxr--r-- 1 xbmc root 763M Apr 10 19:58 headphones.db

— Reply to this email directly or view it on GitHub.

[nlsko]

rembo10: is it possible to shrink the database by using the VACUUM command? https://sqlite.org/lang_vacuum.html

update:

I tried VACUUM on a backup of the DB, it works but didn't help much, the size went from 806MB to 753MB and python still uses mostly 100% CPU when updating artists.

-rwxr--r-- 1 root root 806M Apr 11 12:38 headphones-backup.db -rwxr--r-- 1 xbmc root 753M Apr 11 12:42 headphones.db drwxr-xr-x 9 xbmc root 4.0K Apr 11 12:42 .

[piejanssens]

@nlsko Any luck with getting this issue resolved?

[Steggl]

@nlsko Any luck with getting this issue resolved?

bump

[saschamt]

Would be interested in this as well.

[basilfx]

Today I tried to attach GDB to a running Python process to debug a similar issue with Headphones, and it works very well. I managed to find the thread and code that caused so much CPU.

For the details, see https://wiki.python.org/moin/DebuggingWithGdb. Make sure you run it as the same user Headphones runs as, in the same directory.

[rembo10]

Going to fix this while i'm looking at #1278 & #1979