google / sagetv

SageTV is a cross-platform networked DVR and media management system
http://forums.sagetv.com/
Apache License 2.0
267 stars 173 forks source link

coredump when viewing Parallel Schedule #311

Closed jwittkoski closed 6 years ago

jwittkoski commented 7 years ago

Recently I had several core dumps when viewing the Parallel Schedule page. Initially I believed it might be related to me switching to use SD on one of my tuners, but @enternoescape suggested this was unlikely and it might be some other part of the core system. (See thread starting at https://forums.sagetv.com/forums/showthread.php?p=606550&postcount=332). I had upgraded to v9 several weeks ago but hadn't noticed the problem until yesterday, so it may be related to the v9 upgrade (from v7).

Currently I am running 9.1.4 on linux x86_64 with Java 1.8.0_131. In all three cases where this occurred, I was scrolling around the parallel recording schedule listing while something was being recorded. Then the UI hung and shortly after the process crashed. This was on FiOS recording from HDHomeRun Prime tuner using my prime_encoder script (https://github.com/jwittkoski/prime_encoder) which has been pretty stable for several years.

I have a core file from the crash that occurred shortly after the initial hang. It's 1.5G in size so is there some information can I include from the core dump to help debug? Traceback? jstack dump?

Narflex commented 7 years ago

This looks like a crash in the image loader from the log you posted on the forums. The core dump won't be useful because it's for the whole JVM and I've never been able to get anything good out of those. The best way to resolve this is to see if it can be reproducible, I'd expect to see the same image being referenced in the log file when this crash happens. So if you can figure out what that is, then it should be easy to reproduce and fix.

On Mon, Jun 19, 2017 at 7:59 PM, John Wittkoski notifications@github.com wrote:

Recently I had several core dumps when viewing the Parallel Schedule page. Initially I believed it might be related to me switching to use SD on one of my tuners, but @enternoescape https://github.com/enternoescape suggested this was unlikely and it might be some other part of the core system. (See thread starting at https://forums.sagetv.com/ forums/showthread.php?p=606550&postcount=332). I had upgraded to v9 several weeks ago but hadn't noticed the problem until yesterday, so it may be related to the v9 upgrade (from v7).

Currently I am running 9.1.4 on linux x86_64 with Java 1.8.0_131. In all three cases where this occurred, I was scrolling around the parallel recording schedule listing while something was being recorded. Then the UI hung and shortly after the process crashed. This was on FiOS recording from HDHomeRun Prime tuner using my prime_encoder script (https://github.com/ jwittkoski/prime_encoder) which has been pretty stable for several years.

I have a core file from the crash that occurred shortly after the initial hang. It's 1.5G in size so is there some information can I include from the core dump to help debug? Traceback? jstack dump?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/311, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDCLmAweXINsDk-xvrwqdOMmtOCeoks5sFzWbgaJpZM4N_BxM .

-- Jeffrey Kardatzke jkardatzke@google.com Google, Inc.

jwittkoski commented 7 years ago

I'll look through the logs to try to identify the image.

jwittkoski commented 6 years ago

Was never able to find what image was triggering this. Nothing related in the logs that I could find.

Closing this issue.

jwittkoski commented 6 years ago

This happened again tonight and the process core dumped. The last few log lines are:

Sun 7/15 21:28:58.566 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:28:58.564
Sun 7/15 21:28:58.958 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:28:58.956
Sun 7/15 21:28:59.479 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:28:59.476
Sun 7/15 21:28:59.811 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:28:59.809
Sun 7/15 21:29:00.113 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:29:00.111
Sun 7/15 21:29:00.414 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:29:00.412
Sun 7/15 21:29:00.429 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:29:00.457 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=95 ptr=4758520
Sun 7/15 21:29:00.457 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideCellBorder.png
Sun 7/15 21:29:00.467 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideCellBorder.png width=203 height=95 ptr=4758576
Sun 7/15 21:29:00.467 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/ButtonHighlightNoBorder.png
Sun 7/15 21:29:00.477 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/ButtonHighlightNoBorder.png width=203 height=3 ptr=4758632
Sun 7/15 21:29:00.477 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:29:00.479 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=3 ptr=4758688
Sun 7/15 21:29:00.479 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideCellBorder.png
Sun 7/15 21:29:00.480 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideCellBorder.png width=203 height=3 ptr=4758744
Sun 7/15 21:29:00.480 [FinalRender-0023a5000330@c1545aa] Loading RawImage of size 3x3 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/RecordFavAll.png fileSize=1869
Sun 7/15 21:29:00.481 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/RecordFavAll.png width=3 height=3 ptr=4758800
Sun 7/15 21:29:00.826 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_down] evtTime=Sun 7/15 21:29:00.824
Sun 7/15 21:29:01.255 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_up] evtTime=Sun 7/15 21:29:01.253
Sun 7/15 21:29:01.582 [EventRouter-0023a5000330@46a6937d] processUserEvent-UserEvent[ch_up] evtTime=Sun 7/15 21:29:01.580
Sun 7/15 21:29:01.595 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png

In the logs, normally when that log line shows up, it is immediately followed by an "Allocated image-5..." line (example below), which doesn't appear when the crash happens.

Sun 7/15 21:29:00.477 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:29:00.479 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=3 ptr=4758688

I have four tuners, so there are four columns in the parallel display, and some of my shows extend before and after by between 1 and 5 minutes. Could something about the layout be causing bad values for the clipping of "GuideAiringBGDefault.png"? I noticed in the logs it appears to be clipped at different heights (see logs below).

Is there a way to log what it intends to clip? Could a math error in sizing the clippings result in a height of 0 or a negative that might cause a coredump? Any easy way to test this theory?

Sun 7/15 21:28:16.775 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=55 ptr=4750488
Sun 7/15 21:28:17.014 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:17.042 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=104 ptr=4756248
Sun 7/15 21:28:17.042 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:17.053 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=102 ptr=4752672
Sun 7/15 21:28:17.281 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:17.309 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=55 ptr=4755080
Sun 7/15 21:28:18.515 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:18.543 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=7 ptr=4755136
Sun 7/15 21:28:18.849 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:18.877 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=7 ptr=4755192
Sun 7/15 21:28:56.038 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:56.039 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=4 ptr=4755472
Sun 7/15 21:28:56.374 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:56.391 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=168 ptr=4755696
Sun 7/15 21:28:57.496 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:28:57.520 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=96 ptr=4758464
Sun 7/15 21:29:00.429 [FinalRender-0023a5000330@c1545aa] Creating RawImage with scaled insets from raw copy for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png
Sun 7/15 21:29:00.457 [FinalRender-0023a5000330@c1545aa] Allocated image-5 for /opt/sagetv/server/STVs/SageTV7/Themes/Standard/GuideAiringBGDefault.png width=203 height=95 ptr=4758520
Narflex commented 6 years ago

What type of client device are you using here?

jwittkoski commented 6 years ago

HD300

Narflex commented 6 years ago

Do you have this set in Sage.properties?

ui/enable_hardware_scaling_cache=true

If so, set it to false. That's a non-standard config which has undergone very little testing and is likely hitting some boundary condition.

Let me know if that's it.

jwittkoski commented 6 years ago

Yes, it's set to true. Will update it and re-test.

jwittkoski commented 6 years ago

Ok, I finally got around to testing this - I set ui/enable_hardware_scaling_cache to false and I can't reproduce the problem any longer, so I'll consider this closed.

Should this option be removed from the server going forward? Or is it simply passed to the clients?