gstreamill crash - Githubissues

i4tv / gstreamill

encoder with hls output based on gstreamer.

GNU General Public License v3.0

247 stars 84 forks source link

gstreamill crash #53

Open zhangping opened 9 years ago

zhangping commented 9 years ago

gdb:

Core was generated by `/usr/local/bin/gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  free_priv_data (priv_data=0x0) at httpmgmt.c:1209
1209            if (priv_data->fd == -1) {
Missing separate debuginfos, use: debuginfo-install augeas-libs-1.3.0-1.fc21.x86_64 glib2-2.42.1-1.fc21.x86_64 glibc-2.20-7.fc21.x86_64 gstreamer1-1.4.5-1.fc21.x86_64 libffi-3.1-7.fc21.x86_64 libselinux-2.3-5.fc21.x86_64 libxml2-2.9.1-6.fc21.x86_64 orc-0.4.22-3.fc21.x86_64 pcre-8.35-8.fc21.x86_64 xz-libs-5.1.2-14alpha.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) bt
#0  0x0000000000417c37 in free_priv_data (priv_data=0x0) at httpmgmt.c:1209
#1  0x0000000000417e14 in httpmgmt_dispatcher (data=0x7f7ada673010, user_data=<optimized out>) at httpmgmt.c:1302
#2  0x000000000040ce60 in thread_pool_func (data=0x1edd008, user_data=0x1edcf20) at httpserver.c:942
#3  0x00007f7af5123148 in g_thread_pool_thread_proxy () at /lib64/libglib-2.0.so.0
#4  0x00007f7af51227b5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#5  0x00007f7af4c9552a in start_thread () at /lib64/libpthread.so.0
#6  0x00007f7af478579d in clone () at /lib64/libc.so.6

log:

Apr 25 11:04:44.20585 WARN   gstreamill gstreamill.c:271: Remove live job: KANAL_A access: 0.
Apr 25 11:04:44.22707 WARN   gstreamill gstreamill.c:271: Remove live job: SLO_1 access: 0.
Apr 25 11:04:44.25334 WARN   gstreamill gstreamill.c:271: Remove live job: SLO_2 access: 0.
Apr 25 11:04:44.27335 WARN   gstreamill gstreamill.c:271: Remove live job: SLO_3 access: 0.

*** Apr 25 11:04:44 : gstreamill stoped ***

*** Apr 25 11:04:44 : gstreamill started ***

Apr 25 11:04:57.635342 WARN   gstreamill gstreamill.c:369: 239_10_1_12.encoder.0.audio heartbeat error 7298418611, restart
Apr 25 11:04:57.635472 WARN   gstreamill job.c:789: Restart job 239_10_1_12, pid 13402.
Apr 25 11:04:57.635843 WARN   gstreamill gstreamill.c:369: KANAL_A.encoder.0.audio heartbeat error 7287802690, restart
Apr 25 11:04:57.635900 WARN   gstreamill job.c:789: Restart job KANAL_A, pid 13425.
Apr 25 11:04:59.638972 WARN   gstreamill gstreamill.c:369: SLO_1.encoder.0.audio heartbeat error 8272132776, restart
Apr 25 11:04:59.639098 WARN   gstreamill job.c:789: Restart job SLO_1, pid 13462.
Apr 25 11:05:11.654686 WARN   gstreamill gstreamill.c:369: SLO_1.encoder.0.audio heartbeat error 8627727413, restart
Apr 25 11:05:11.654783 WARN   gstreamill job.c:789: Restart job SLO_1, pid 13585.
Apr 25 11:05:12.69555 WARN   gstreamill gstreamill.c:658: 239_10_1_12.encoder.0: record segment, but segment not found!

*** Apr 25 11:05:52 : gstreamill started ***

Apr 25 11:05:53.104836 WARN   gstreamill httpserver.c:903: warning!!! unprocessed event, sock 13 status 2 events 0
Apr 25 11:05:55.386237 WARN   gstreamill httpserver.c:903: warning!!! unprocessed event, sock 13 status 2 events 0
Apr 25 11:06:05.108326 WARN   gstreamill gstreamill.c:369: 239_10_1_12.encoder.0.audio heartbeat error 7360374191, restart
Apr 25 11:06:05.108412 WARN   gstreamill job.c:789: Restart job 239_10_1_12, pid 13719.
Apr 25 11:06:05.108726 WARN   gstreamill gstreamill.c:369: KANAL_A.encoder.0.audio heartbeat error 7344502766, restart
Apr 25 11:06:05.108786 WARN   gstreamill job.c:789: Restart job KANAL_A, pid 13742.
Apr 25 11:06:07.111930 WARN   gstreamill gstreamill.c:369: SLO_1.encoder.0.audio heartbeat error 8378156841, restart
Apr 25 11:06:07.112041 WARN   gstreamill job.c:789: Restart job SLO_1, pid 13778.

* Apr 25 11:05:52 : gstreamill started * crash here

zhangping commented 9 years ago

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/bin/gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000409570 in child_watch_cb (pid=773, status=2304, job=0x1fd9620) at gstreamill.c:952
952             if (*(job->output->state) == JOB_STATE_STOPING) {
Missing separate debuginfos, use: debuginfo-install augeas-libs-1.3.0-1.fc21.x86_64 glib2-2.42.1-1.fc21.x86_64 glibc-2.20-7.fc21.x86_64 gstreamer1-1.4.5-1.fc21.x86_64 libffi-3.1-7.fc21.x86_64 libselinux-2.3-5.fc21.x86_64 libxml2-2.9.1-6.fc21.x86_64 orc-0.4.22-3.fc21.x86_64 pcre-8.35-8.fc21.x86_64 xz-libs-5.1.2-14alpha.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) bt
#0  0x0000000000409570 in child_watch_cb (pid=773, status=2304, job=0x1fd9620) at gstreamill.c:952
#1  0x00007f6135043604 in g_child_watch_dispatch () at /lib64/libglib-2.0.so.0
#2  0x00007f6135046aeb in g_main_context_dispatch () at /lib64/libglib-2.0.so.0
#3  0x00007f6135046e88 in g_main_context_iterate.isra () at /lib64/libglib-2.0.so.0
#4  0x00007f61350471b2 in g_main_loop_run () at /lib64/libglib-2.0.so.0
#5  0x00000000004078eb in main (argc=1, argv=0x7fff6945d728) at main.c:542

zhangping commented 9 years ago

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/bin/gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fa7f44e2bdd in free () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install augeas-libs-1.3.0-1.fc21.x86_64 glib2-2.42.1-1.fc21.x86_64 glibc-2.20-7.fc21.x86_64 gstreamer1-1.4.5-1.fc21.x86_64 libffi-3.1-7.fc21.x86_64 libselinux-2.3-5.fc21.x86_64 libxml2-2.9.1-6.fc21.x86_64 orc-0.4.22-3.fc21.x86_64 pcre-8.35-8.fc21.x86_64 xz-libs-5.1.2-14alpha.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) bt
#0  0x00007fa7f44e2bdd in free () at /lib64/libc.so.6
#1  0x00007fa7f4edb7ff in g_free () at /lib64/libglib-2.0.so.0
#2  0x00000000004128b8 in job_dispose (obj=0x7fa7c40061b0) at job.c:163
#3  0x00007fa7f51d9cbc in g_object_unref () at /lib64/libgobject-2.0.so.0
#4  0x0000000000409db8 in gstreamill_monitor (gstreamill=0x17865e0) at gstreamill.c:281
#5  0x0000000000409db8 in gstreamill_monitor (clock=<optimized out>, time=<optimized out>, id=<optimized out>, user_data=0x17865e0)
    at gstreamill.c:587
#6  0x00007fa7f5a674c2 in gst_system_clock_async_thread () at /lib64/libgstreamer-1.0.so.0
#7  0x00007fa7f4efc7b5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#8  0x00007fa7f4a6f52a in start_thread () at /lib64/libpthread.so.0
#9  0x00007fa7f455f79d in clone () at /lib64/libc.so.6

zhangping commented 9 years ago

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/bin/gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fecf0766231 in g_queue_foreach () from /lib64/libglib-2.0.so.0
Missing separate debuginfos, use: debuginfo-install augeas-libs-1.3.0-1.fc21.x86_64 glib2-2.42.1-1.fc21.x86_64 glibc-2.20-7.fc21.x86_64 gstreamer1-1.4.5-1.fc21.x86_64 libffi-3.1-7.fc21.x86_64 libselinux-2.3-5.fc21.x86_64 libxml2-2.9.1-6.fc21.x86_64 orc-0.4.22-3.fc21.x86_64 pcre-8.35-8.fc21.x86_64 xz-libs-5.1.2-14alpha.fc21.x86_64 zlib-1.2.8-7.fc21.x86_64
(gdb) bt
#0  0x00007fecf0766231 in g_queue_foreach () at /lib64/libglib-2.0.so.0
#1  0x000000000041db24 in m3u8playlist_free (playlist=0x7fecbc0bc6f0) at m3u8playlist.c:44
#2  0x00000000004128d2 in job_dispose (obj=0x7fecbc0122d0) at job.c:165
#3  0x00007fecf0a59cbc in g_object_unref () at /lib64/libgobject-2.0.so.0
#4  0x0000000000409db8 in gstreamill_monitor (gstreamill=0x1b745e0) at gstreamill.c:281
#5  0x0000000000409db8 in gstreamill_monitor (clock=<optimized out>, time=<optimized out>, id=<optimized out>, user_data=0x1b745e0)
    at gstreamill.c:587
#6  0x00007fecf12e74c2 in gst_system_clock_async_thread () at /lib64/libgstreamer-1.0.so.0
#7  0x00007fecf077c7b5 in g_thread_proxy () at /lib64/libglib-2.0.so.0
#8  0x00007fecf02ef52a in start_thread () at /lib64/libpthread.so.0
#9  0x00007fecefddf79d in clone () at /lib64/libc.so.6

zhangping commented 9 years ago

3c5a504b96266d9ca9f367162e7221e104e6b7b7

Program terminated with signal SIGBUS, Bus error.
#0  0x000000000040c3ed in msg_thread (data=0xaca5e0) at gstreamill.c:840
840                             encoder_output->last_timestamp = encoder_output_rap_timestamp (encoder_output, *(encoder_output->last_rap_addr));

zhangping commented 9 years ago

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gstreamill'.
Program terminated with signal SIGBUS, Bus error.
#0  0x000000000040c4dd in msg_thread (data=0x210f5e0) at gstreamill.c:840
840                 encoder_output->last_timestamp = encoder_output_rap_timestamp (encoder_output, *(encoder_output->last_rap_addr));
(gdb) print encoder_output
$1 = (EncoderOutput *) 0x7f3d8006a220
(gdb) print encoder_output->last_rap_addr
$2 = (guint64 *) 0x7f3d43ffb81c
(gdb) print *(encoder_output->last_rap_addr)
Cannot access memory at address 0x7f3d43ffb81c

zhangping commented 9 years ago

warning: .dynamic section for "/usr/lib/libgstvideo-1.0.so.0" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:153
153     ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
(gdb) bt
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:153
#1  0x000000000040cb33 in memcpy (__len=18446744073709551612, __src=<optimized out>, __dest=0x7fd515a5c4f0)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#2  dvr_record_segment (
    seg_path=0x7fd51416f620 "\300\270\262\033\310 \206\243\276\237D\353\022\364}\320\065-\321|\247\370\200\027\063>;\372\255\060E%\221\230@\237\203\062\377̄\272\064W!y\200\tU3\266\063Ѫp\371\251\016\070=kŒ\362\240\225\070\275\202H\372\362V\237b\246B\204\202\016\315E\244\003\r\345\061\370\321\b\177\235\210\265\307:\204\370\237\071\071\020\327%\342h\312\b(}\265{(A\177\247G", duration=3000000000,
    encoder_output=0x7fd4e80219d0) at gstreamill.c:694
#3  msg_thread (data=0x1d1a5e0) at gstreamill.c:833
#4  0x00007fd51cc51f05 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007fd51c7c6182 in start_thread (arg=0x7fd519985700) at pthread_create.c:312
#6  0x00007fd51be1f47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

zhangping commented 8 years ago

eee0b854f

zhangping commented 8 years ago

962307f674

warning: Error reading shared library list entry at 0x1

warning: Corrupted shared library list: 0x0 != 0x21dece0
Core was generated by `gstreamill'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000415f09 in job_stat_update (job=0x0) at job.c:637
637         job->cpu_average = ((utime + stime) * 10000) / (ctime - job->start_ctime);
(gdb) bt
#0  0x0000000000415f09 in job_stat_update (job=0x0) at job.c:637
#1  0x0000000002060c70 in ?? ()
#2  0x0000000000000000 in ?? ()

zhangping commented 8 years ago

启动job时子进程挂掉，仍然进入watch过程，仍然被重启，造成重启失败，以后应该grep日志看看是否出现如下内容

May 13 10:22:58.450521 WARN   gstreamill httpmgmt.c:1435: Request from 10.10.0.106, uri is /admin/start
May 13 10:22:58.450606 WARN   gstreamill httpmgmt.c:842: Start job request from 10.10.0.106 with description:
 {"encoders": [{"elements": {"appsrc": {"property": {"is-live": "true", "format": 3}}, "tssegment": {"property": {"bitrate": 1500}}, "appsink": {"property": {"sync": "false"}}}, "bins": ["appsrc name=mpegts ! queue ! tssegment ! queue ! appsink"]}], "name": "00000110240180_1", "dvr_duration": 604800, "is-live": "true", "m3u8streaming": {"version": 3, "segment-duration": 5, "window-size": 10}, "source": {"elements": {"appsink": {"property": {"drop": "true", "sync": "false"}}, "udpsrc": {"property": {"uri": "udp://224.168.44.112:6000"}}}, "bins": ["udpsrc ! queue ! appsink name = mpegts"]}, "debug": "3"}

May 13 10:22:58.512946 WARN   gstreamill gstreamill.c:1001: Read job 00000110240180_1's stat failure.
May 13 10:22:58.513033 WARN   gstreamill gstreamill.c:1200: Start job 00000110240180_1 failure, return stat: JOB_STATE_START_FAILURE
May 13 10:27:46.261624 WARN   gstreamill main.c:538:
*** May 13 10:27:46 : gstreamill started ***

用gdb调试core文件如下

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gstreamill -d -m 10.10.124.133:20118 -a 10.10.124.133:20119'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000040aa97 in create_job_process (job=job@entry=0x7f0a0c009a30) at gstreamill.c:996
996 gstreamill.c: No such file or directory.
(gdb) bt
#0  0x000000000040aa97 in create_job_process (job=job@entry=0x7f0a0c009a30) at gstreamill.c:996
#1  0x000000000040ac1b in child_watch_cb (pid=28833, status=13, job=0x7f0a0c009a30) at gstreamill.c:1075
#2  0x00007f0a3e8adb94 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007f0a3e8b0ce5 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007f0a3e8b1048 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007f0a3e8b130a in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x0000000000408a0b in main (argc=1, argv=0x7fff381175c8) at main.c:601

03f21df63101

zhangping commented 7 years ago

0.8.7版本的挂掉状况

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gstreamill -d -m 10.10.2.61:20118 -a 10.10.2.61:20119'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fcb80db0bc4 in sem_post@@GLIBC_2.2.5 () from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7fcb7ba6d700 (LWP 47))]
(gdb) bt
#0  0x00007fcb80db0bc4 in sem_post@@GLIBC_2.2.5 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x000000000040d619 in msg_thread (data=0x799830) at gstreamill.c:949
#2  0x00007fcb81236bb5 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00007fcb80da86ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007fcb803e682d in clone () from /lib/x86_64-linux-gnu/libc.so.6