search

theotherp / nzbhydra2

Usenet meta search
Other
1.26k stars 75 forks source link

high cpu usage when idle #96

Closed phairplay closed 6 years ago

phairplay commented 6 years ago

hi i keep seeing java (nzbhydra2) using 25-30% of my cpu yet it doesnt seem to be doing anything. could you please help with this.

im running windows 10 nzbhydra2 version 1.2.0

here is the latest logs nzbhydra-debuginfos-2018-01-28-18-08.zip

theotherp commented 6 years ago

Please enable debug log level and the following log markers: "Performance" and "Download status updating" as seen here: https://i.imgur.com/QcORofu.png

phairplay commented 6 years ago

nzbhydra-debuginfos-2018-01-28-22-05.zip ive turn them on here is the logs currently using zero cpu ill post another when it happens again. if required

phairplay commented 6 years ago

hi its happening again consuming 25-30% cpu nzbhydra-debuginfos-2018-01-30-08-51.zip

theotherp commented 6 years ago

Hm, I don't see anything in there. Only that you recently queried the stats. Perhaps that caused it. Please try increasing the XMX value (in the main config at the bottom) to perhaps 256 and restart. Generate the stats again and see if that helps.

I'll need to think about how to get better debugging information for performance problems.

phairplay commented 6 years ago

Okay I've made the change and will report back,

Nzbhydra while idle consumes around 200mb currently is this normal?

theotherp commented 6 years ago

Yes.

fryfrog commented 6 years ago

I've seen this a handful of times on my server too, nzbhydra2 using > 100% cpu for long periods of time. I've turned my logging up to debug and turned on "PERFORMANCE" for log markers. I'll post a debug and logs when it happens again.

fryfrog commented 6 years ago

It happened this evening, I watched it in htop using 100-350% CPU while doing totally normal looking things like handling periodic searches from radarr / sonarr, finishing them and then doing nothing. While the issue was occurring, I tried to get a debug zip file, but it OOM'd. So I doubled the memory and restarted, the usage is back to normal, but it made the debug zip. Let me know if this is useful, or what I can do to make it more useful.

nzbhydra-debuginfos-2018-02-20-21-46.zip

theotherp commented 6 years ago

The next time you observe this please follow these instructions (only step 1 and first part of step 2) and post the thread dump.

phairplay commented 6 years ago

hi im seeing java again using a high level of my cpu i can crate a windows dmp file i dont really understand the link you provided

phairplay commented 6 years ago

hi @theotherp could you please help with how to get the correct java dump, i dont really understand your link

java is using high usage again

im on windows 10

theotherp commented 6 years ago
  1. Download ZIP from https://visualvm.github.io/download.html and extract somewhere. Start exe from bin folder.
  2. https://i.imgur.com/fi0juqV.png
  3. https://i.imgur.com/sVgpvt6.png

Save file and post here. You might need to zip it first.

Please also go to the "Threads" and "Sampler" tabs and post screenshots of their content when CPU usage is high while no search is active. Post the timestamp and your latest debug infos zip.

phairplay commented 6 years ago

hi thank you for the assistants i downloaded the visualvm and it asked my to also download java development (don't know if it matters)

here is an image on the visualvm https://imgur.com/a/tps35 here is a snapshot of the task manager https://imgur.com/a/6QKTB

and the zip file as requested.

i really hope you can help me fix this :) 15-03-18 19.15pm.zip

theotherp commented 6 years ago

Thanks, will hopefully have time to take a look at the weekend.

theotherp commented 6 years ago

Alright, unfortunately the dump and screenshot you made are of the visualvm process, not of Hydra. I think this would work better if we had a chat, otherwise this will take forever.

You seem to be in the US (and I'm in germany) so it might be a bit complicated to find a time. Find me on discord via "TheOtherP#1176" and we'll try to find a solution.

phairplay commented 6 years ago

Dude that super good of you.

Looking at this again I noticed two things

1, I’m a muppet for some reason i didn’t notice I was providing details of visualvm. (I so sorry about that). 2, Visualvm doesn’t see nzbhydra If it’s run as a windows service.

So I’ve done the following: Uninstalled and reinstalled java Loaded nzbhydra2.exe Confirmed visualvm can see nzbhydra running. I’ll give it 24hrs and report back

Oh and I’m in the UK 👍

phairplay commented 6 years ago

hi im having the issue again yet when i look into visualvm there is no tab for a thread dump

https://imgur.com/a/F4bqb

theotherp commented 6 years ago

Hmm, apparently you need JDK instead of JRE. Sorry about that, it's always hard to know what the environment on other machines look like (i.e. I have a development environment installed and made the mistake of assuming this would work with every java installation).

Install the windows JDK (64Bit) from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and install it. You may need to uninstall Java before to make sure that the JDK will be used to run Hydra. In its installation folder you will find a subfolder called "bin" which contains a "jvisualvm.exe". Running that should allow you to properly monitor the hydra process and create dumps and get a reading of its threads and CPU usage.

Sorry for all the inconveniences.

phairplay commented 6 years ago

Dude don't apologise I appericate the help. I'll get it done know ans report back

phairplay commented 6 years ago

morning, its happening again and i have hopefully captured everything you need.

task manager https://imgur.com/a/zFNVs

shot of the treads on visualvm (check and it is looking at nzbhydra) https://imgur.com/a/Zd9SD

Cpu samples https://imgur.com/a/Ry0as

thread cpu time https://imgur.com/a/YMmXJ

memory sample https://imgur.com/a/dnrbM

debug log nzbhydra-debuginfos-2018-03-23-05-43.zip

thread dump taken at 23-03-18 5.50am threaddump-23-03-18 05.50am.zip

theotherp commented 6 years ago

I haven't forgotten you, life's just very busy at the moment ;-)

phairplay commented 6 years ago

Haha No worries :)

theotherp commented 6 years ago

Well, to be honest, I have no idea what's going on. I wonder if the fact that nobody else has been complaining about this means that it only affects you or that just nobody else has noticed... I kinda hope the former but that would make it even harder to help you ;-) I think fryfrog's problem was caused by not allocating enough memory.

I'm not very skilled in performance analysis so it's possible there are some indicators which I just don't see.

I have one last set of instructions.

  1. Update your wrapper manually. Shutdown Hydra, replace the exe files from [the latest release]https://github.com/theotherp/nzbhydra2/releases/download/v1.4.8/nzbhydra2-1.4.8-windows.zip) and restart Hydra.
  2. The next time you notice high CPU usage and you're sure Hydra is idle make a screenshot of the "Monitor" tab in VisualVM. Make sure the CPU usage in that tab is actually roughly the same as the one in the task manager. If not it's the wrapper which is causing the high CPU usage (which I doubt but who knows).
  3. In VisualVM go to the "Sampler" tab and click the "CPU" button. Wait a couple of seconds (10 or so) and then click "Snapshot" (a little to the lower right). Click the "Export" button (below "Profiler Snapshot"). Save the file, zip and post it along with the latest debug infos. Sampler 1 Sampler 2 Sampler 3

If the result doesn't show me anything I'm at a loss.

phairplay commented 6 years ago

many thanks fingers crossed :)

phairplay commented 6 years ago

hi, I've just finished moving house and I have my system back up and running, i'll keep you posted :)

phairplay commented 6 years ago

Hi, i'm back had isp issues so not been able to run hydra,

it's been up for 5days and I've notice java is stealing cpu again

here is the cpu view

https://imgur.com/a/j96uyWM

here is the sampler file you asked for sampler snapshot.zip

I hope this helps

alron commented 6 years ago

@theotherp

I just wanted to chime in on this. I've noticed this issue on two different hydra2 installs as well. Just, with it being in docker, I've no real way of easily doing the dumps.

Docker image linuxserver/hydra2 installed on Ubuntu 16.04 with Atom C2758 and Ubuntu 18.04 with Xeon D-2146NT.

If you don't restart hydra2 every day or so (few hours in some cases), it'll leak a thread (or spinlock/deadlock) or two and go to 70-100% CPU use on each of them. Additionally, this isn't a constant cpu use. It fluctuates noticeably, with rhythmic pulses to 100% from 70% or so every few moments.

It doesn't seem to matter how much or little memory is assigned via config. It does it with 512M just as often as it does it with 4G. I've replaced configs, rebuilt from scratch, installed on new hardware platform, reworked network, replaced filesystems, updated docker. No luck.

I do have a fairly active hydra2, with 7 indexers being used by Medusa and Radarr, with nzbget as the download agent. It tends to trigger quickly when I've been using it hard. Is it possible it is something to do with the download status monitoring? There is no apparent way to disable said functionality.

theotherp commented 6 years ago

@alron I've always had a theory that this was caused by download status monitoring but never found prove of it. Please set your logging level to debug and enable the log marker for "Download status updating". Restart, give it a couple of hours and post the debug infos ZIP.

phairplay commented 6 years ago

Could this also be a course for me I'm finding not having it as a Windows service it happens more quickly

theotherp commented 6 years ago

No idea. I'm currently testing a minor change that might help. If you want to test it shutdown your instance, replace the JAR file in the lib folder with this one: https://ufile.io/vlpni and see what happens. It's honestly a shot in the dark but who knows. If I'm right the CPU usage should be high for a bit but then at least go down for 10 seconds or so. But the log you posted some time ago doesn't indicate that the status update tasks are running which is what @alron suspects is the cause of the issue.

theotherp commented 6 years ago

I just released an update which allows you to disable the download status updated. See if it helps. If it does I'll need to find some proper fix.

phairplay commented 6 years ago

Sorry Do I update and update jar?

theotherp commented 6 years ago

No, just update.

phairplay commented 6 years ago

okay its happening again. also it happens quicker if not run as a windows server

theotherp commented 6 years ago

Have you tried disabling the download status updating?

fryfrog commented 6 years ago

Config -> Downloading -> Update statuses, right?

phairplay commented 6 years ago

its off

theotherp commented 6 years ago

Hm, too bad.

fryfrog commented 6 years ago

I actually ended up limiting mine with systemd :(

phairplay commented 6 years ago

Does this mean there is no fix?

theotherp commented 6 years ago

If I knew what causes the CPU usage I could probably fix it. So far I haven't found out. I may have an idea or just time to do some in-depth debugging at some point but not for now. So either you use limit the impact (e.g. like @fryfrog describes) or live with it or go back to v1. Sorry, I don't have a better answer for now.

theotherp commented 6 years ago

I have an idea. In the main folder create a folder "config" (next to "lib") and create a file application.properties in it. Open it and add the line spring.datasource.url=jdbc:h2:file:${nzbhydra.dataFolder:.}/database/nzbhydra;DEFRAG_ALWAYS=TRUE;WRITE_DELAY=20000

Restart Hydra and see if it helps. Give it a couple of days. If and when CPU usage gets high again shut it down, remove the JAR file in the lib folder and replace it with this one https://ufile.io/eynpz (Note that you may need to do the next update manually when a new version is released)

theotherp commented 6 years ago

As it turns out I had this issue on my server but my latest changes (and / or the config change) seems to have fixed it.

fryfrog commented 6 years ago

I haven't seen it go bananas either after making that config change and updating to your latest.

phairplay commented 6 years ago

Cool beans I'll make the change too and report back

svms1 commented 6 years ago

I had this issue as well, but a day after applying that configuration the bug hasn't reappeared so far. I'll let you know if it appears again.

What was the bug?

phairplay commented 6 years ago

Day 6 and all is good whoop!

Thank you for all your help

fryfrog commented 6 years ago

Same, haven't seen cpu go crazy in ages. Yay! :)

alron commented 6 years ago

Yeah. Looks like that has fixed my issue as well. Much thanks. Xeon happy not burning as much power. :)

whispy commented 6 years ago

Updating to v1.4.14 and adding in the application.properties file with the tweak in it seems to have fixed any memory/CPU leak issues I was having. Previously, Hydra2 would become unresponsive after ~6 days of uptime, requiring a restart. I've now had it running for 6 days, and it seems as responsive today as it was when I restarted it 6 days ago.