Closed phairplay closed 6 years ago
Please enable debug log level and the following log markers: "Performance" and "Download status updating" as seen here: https://i.imgur.com/QcORofu.png
nzbhydra-debuginfos-2018-01-28-22-05.zip ive turn them on here is the logs currently using zero cpu ill post another when it happens again. if required
hi its happening again consuming 25-30% cpu nzbhydra-debuginfos-2018-01-30-08-51.zip
Hm, I don't see anything in there. Only that you recently queried the stats. Perhaps that caused it. Please try increasing the XMX value (in the main config at the bottom) to perhaps 256 and restart. Generate the stats again and see if that helps.
I'll need to think about how to get better debugging information for performance problems.
Okay I've made the change and will report back,
Nzbhydra while idle consumes around 200mb currently is this normal?
Yes.
I've seen this a handful of times on my server too, nzbhydra2 using > 100% cpu for long periods of time. I've turned my logging up to debug and turned on "PERFORMANCE" for log markers. I'll post a debug and logs when it happens again.
It happened this evening, I watched it in htop using 100-350% CPU while doing totally normal looking things like handling periodic searches from radarr / sonarr, finishing them and then doing nothing. While the issue was occurring, I tried to get a debug zip file, but it OOM'd. So I doubled the memory and restarted, the usage is back to normal, but it made the debug zip. Let me know if this is useful, or what I can do to make it more useful.
The next time you observe this please follow these instructions (only step 1 and first part of step 2) and post the thread dump.
hi im seeing java again using a high level of my cpu i can crate a windows dmp file i dont really understand the link you provided
hi @theotherp could you please help with how to get the correct java dump, i dont really understand your link
java is using high usage again
im on windows 10
Save file and post here. You might need to zip it first.
Please also go to the "Threads" and "Sampler" tabs and post screenshots of their content when CPU usage is high while no search is active. Post the timestamp and your latest debug infos zip.
hi thank you for the assistants i downloaded the visualvm and it asked my to also download java development (don't know if it matters)
here is an image on the visualvm https://imgur.com/a/tps35 here is a snapshot of the task manager https://imgur.com/a/6QKTB
and the zip file as requested.
i really hope you can help me fix this :) 15-03-18 19.15pm.zip
Thanks, will hopefully have time to take a look at the weekend.
Alright, unfortunately the dump and screenshot you made are of the visualvm process, not of Hydra. I think this would work better if we had a chat, otherwise this will take forever.
You seem to be in the US (and I'm in germany) so it might be a bit complicated to find a time. Find me on discord via "TheOtherP#1176" and we'll try to find a solution.
Dude that super good of you.
Looking at this again I noticed two things
1, I’m a muppet for some reason i didn’t notice I was providing details of visualvm. (I so sorry about that). 2, Visualvm doesn’t see nzbhydra If it’s run as a windows service.
So I’ve done the following: Uninstalled and reinstalled java Loaded nzbhydra2.exe Confirmed visualvm can see nzbhydra running. I’ll give it 24hrs and report back
Oh and I’m in the UK 👍
hi im having the issue again yet when i look into visualvm there is no tab for a thread dump
Hmm, apparently you need JDK instead of JRE. Sorry about that, it's always hard to know what the environment on other machines look like (i.e. I have a development environment installed and made the mistake of assuming this would work with every java installation).
Install the windows JDK (64Bit) from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and install it. You may need to uninstall Java before to make sure that the JDK will be used to run Hydra. In its installation folder you will find a subfolder called "bin" which contains a "jvisualvm.exe". Running that should allow you to properly monitor the hydra process and create dumps and get a reading of its threads and CPU usage.
Sorry for all the inconveniences.
Dude don't apologise I appericate the help. I'll get it done know ans report back
morning, its happening again and i have hopefully captured everything you need.
task manager https://imgur.com/a/zFNVs
shot of the treads on visualvm (check and it is looking at nzbhydra) https://imgur.com/a/Zd9SD
Cpu samples https://imgur.com/a/Ry0as
thread cpu time https://imgur.com/a/YMmXJ
memory sample https://imgur.com/a/dnrbM
debug log nzbhydra-debuginfos-2018-03-23-05-43.zip
thread dump taken at 23-03-18 5.50am threaddump-23-03-18 05.50am.zip
I haven't forgotten you, life's just very busy at the moment ;-)
Haha No worries :)
Well, to be honest, I have no idea what's going on. I wonder if the fact that nobody else has been complaining about this means that it only affects you or that just nobody else has noticed... I kinda hope the former but that would make it even harder to help you ;-) I think fryfrog's problem was caused by not allocating enough memory.
I'm not very skilled in performance analysis so it's possible there are some indicators which I just don't see.
I have one last set of instructions.
If the result doesn't show me anything I'm at a loss.
many thanks fingers crossed :)
hi, I've just finished moving house and I have my system back up and running, i'll keep you posted :)
Hi, i'm back had isp issues so not been able to run hydra,
it's been up for 5days and I've notice java is stealing cpu again
here is the cpu view
here is the sampler file you asked for sampler snapshot.zip
I hope this helps
@theotherp
I just wanted to chime in on this. I've noticed this issue on two different hydra2 installs as well. Just, with it being in docker, I've no real way of easily doing the dumps.
Docker image linuxserver/hydra2 installed on Ubuntu 16.04 with Atom C2758 and Ubuntu 18.04 with Xeon D-2146NT.
If you don't restart hydra2 every day or so (few hours in some cases), it'll leak a thread (or spinlock/deadlock) or two and go to 70-100% CPU use on each of them. Additionally, this isn't a constant cpu use. It fluctuates noticeably, with rhythmic pulses to 100% from 70% or so every few moments.
It doesn't seem to matter how much or little memory is assigned via config. It does it with 512M just as often as it does it with 4G. I've replaced configs, rebuilt from scratch, installed on new hardware platform, reworked network, replaced filesystems, updated docker. No luck.
I do have a fairly active hydra2, with 7 indexers being used by Medusa and Radarr, with nzbget as the download agent. It tends to trigger quickly when I've been using it hard. Is it possible it is something to do with the download status monitoring? There is no apparent way to disable said functionality.
@alron I've always had a theory that this was caused by download status monitoring but never found prove of it. Please set your logging level to debug and enable the log marker for "Download status updating". Restart, give it a couple of hours and post the debug infos ZIP.
Could this also be a course for me I'm finding not having it as a Windows service it happens more quickly
No idea. I'm currently testing a minor change that might help. If you want to test it shutdown your instance, replace the JAR file in the lib folder with this one: https://ufile.io/vlpni and see what happens. It's honestly a shot in the dark but who knows. If I'm right the CPU usage should be high for a bit but then at least go down for 10 seconds or so. But the log you posted some time ago doesn't indicate that the status update tasks are running which is what @alron suspects is the cause of the issue.
I just released an update which allows you to disable the download status updated. See if it helps. If it does I'll need to find some proper fix.
Sorry Do I update and update jar?
No, just update.
okay its happening again. also it happens quicker if not run as a windows server
Have you tried disabling the download status updating?
Config -> Downloading -> Update statuses, right?
its off
Hm, too bad.
I actually ended up limiting mine with systemd :(
Does this mean there is no fix?
If I knew what causes the CPU usage I could probably fix it. So far I haven't found out. I may have an idea or just time to do some in-depth debugging at some point but not for now. So either you use limit the impact (e.g. like @fryfrog describes) or live with it or go back to v1. Sorry, I don't have a better answer for now.
I have an idea. In the main folder create a folder "config" (next to "lib") and create a file application.properties in it. Open it and add the line
spring.datasource.url=jdbc:h2:file:${nzbhydra.dataFolder:.}/database/nzbhydra;DEFRAG_ALWAYS=TRUE;WRITE_DELAY=20000
Restart Hydra and see if it helps. Give it a couple of days. If and when CPU usage gets high again shut it down, remove the JAR file in the lib folder and replace it with this one https://ufile.io/eynpz (Note that you may need to do the next update manually when a new version is released)
As it turns out I had this issue on my server but my latest changes (and / or the config change) seems to have fixed it.
I haven't seen it go bananas either after making that config change and updating to your latest.
Cool beans I'll make the change too and report back
I had this issue as well, but a day after applying that configuration the bug hasn't reappeared so far. I'll let you know if it appears again.
What was the bug?
Day 6 and all is good whoop!
Thank you for all your help
Same, haven't seen cpu go crazy in ages. Yay! :)
Yeah. Looks like that has fixed my issue as well. Much thanks. Xeon happy not burning as much power. :)
Updating to v1.4.14 and adding in the application.properties
file with the tweak in it seems to have fixed any memory/CPU leak issues I was having. Previously, Hydra2 would become unresponsive after ~6 days of uptime, requiring a restart. I've now had it running for 6 days, and it seems as responsive today as it was when I restarted it 6 days ago.
hi i keep seeing java (nzbhydra2) using 25-30% of my cpu yet it doesnt seem to be doing anything. could you please help with this.
im running windows 10 nzbhydra2 version 1.2.0
here is the latest logs nzbhydra-debuginfos-2018-01-28-18-08.zip