Kurento 6.1.1 some filters are leaking memory when not connected downstream

[tsonghong]

Hi, I've noticed my Kurento server 6.1.1 would segfault with an error regarding rtpjitterbuffer and would like to check with you guys for any ideas. (I believe I had this issue since when I was using Kurento first time when it was 6.0.1 but decided to mentioned it now and don't suspect it's something introduced in 6.1.1 in particular)

Everytime there's a crash, I notice this rtpjitterbuffer line added in dmesg syslog:

[2383818.318980] rtpjitterbuffer[2022]: segfault at 7f68bd8d8022 ip 00007f6ed98b2005 sp 00007f69837d4eb8 error 6 in libc-2.19.so[7f6ed981a000+1bb000] [2384993.694412] nvidia 0000:83:00.0: irq 242 for MSI/MSI-X [2384996.247091] nvidia 0000:84:00.0: irq 243 for MSI/MSI-X [2392555.749510] rtpjitterbuffer[2775]: segfault at 7f12115bc000 ip 00007f17c586e0b6 sp 00007f14a57f8eb8 error 6 in libc-2.19.so[7f17c57d6000+1bb000] [2437114.484534] nvidia 0000:83:00.0: irq 242 for MSI/MSI-X [2437117.035556] nvidia 0000:84:00.0: irq 243 for MSI/MSI-X [2452113.745150] rtpjitterbuffer[6211]: segfault at 7f33d3bc3c32 ip 00007f3834156005 sp 00007f34337fceb8 error 6 in libc-2.19.so[7f38340be000+1bb000]

Right now, I don't know what exactly triggers this. We have multiple users trying to view webrtc connections and involve the WebRtcEndpoint and CV Filters and we are using H.264 only (no VP8). It doesn't crash constantly, just seemingly random perhaps a few times a day.

Here's the version of software we are running for gstreamer, kurento:

tsonghong@topper:/tmp/cores$ dpkg --list | grep kurento ii kurento-media-server-6.0 6.1.1~20150917092655.5.g9fdeccd.trusty amd64 Kurento Media Server ii kurento-media-server-6.0-dev 6.1.1~20150917092655.5.g9fdeccd.trusty amd64 Kurento media server, development files and tools ii kurento-module-creator-4.0 4.0.1.trusty all Kurento module creator tsonghong@topper:/tmp/cores$ dpkg --list | grep kms ii bbswitch-dkms 0.7-2ubuntu1 amd64 Interface for toggling the power on nVidia Optimus video cards ii dkms 2.2.0.3-1.1ubuntu5.14.04.1 all Dynamic Kernel Module Support Framework ii kms-chroma-6.0 6.1.1~20150911095807.1.gfcd2964.trusty amd64 Kurento Chroma Filter ii kms-cmake-utils 1.2.2~20150714125501.1.g3acfd2b.trusty all Kurento cmake utils ii kms-core-6.0 6.1.2~20150911093636.55.gc48a96a.trusty amd64 Kurento core module ii kms-core-6.0-dev 6.1.2~20150911093636.55.gc48a96a.trusty amd64 Kurento core module ii kms-crowddetector-6.0 6.1.1~20150911095836.1.gacc19a5.trusty amd64 Kurento Crowd Detector Filter ii kms-elements-6.0 6.1.1~20150911095143.52.g9bf6042.trusty amd64 Kurento elements module ii kms-elements-6.0-dev 6.1.1~20150911095143.52.g9bf6042.trusty amd64 Kurento elements module ii kms-filters-6.0 6.1.1~20150911095453.1.g7dcf61b.trusty amd64 Kurento filters module ii kms-filters-6.0-dev 6.1.1~20150911095453.1.g7dcf61b.trusty amd64 Kurento filters module ii kms-jsonrpc-1.0 1.0.1~20150714130721.1.g78881e4.trusty amd64 Kurento jsonrpc library ii kms-jsonrpc-1.0-dev 1.0.1~20150714130721.1.g78881e4.trusty amd64 Kurento jsonrpc library ii kms-platedetector-6.0 6.1.1~20150911095820.1.g437567f.trusty amd64 Kurento Plate Detector Filter ii kms-pointerdetector-6.0 6.1.1~20150911095927.1.gc6c3092.trusty amd64 Kurento Pointer Detector Filter ii kmsjsoncpp 1.6.3~20150901123209.38.g263929e.trusty amd64 Kurento jsoncpp library ii kmsjsoncpp-dev 1.6.3~20150901123209.38.g263929e.trusty amd64 Kurento jsonrpc library tsonghong@topper:/tmp/cores$

tsonghong@topper:/tmp/cores$ dpkg --list | grep gst ii gir1.2-gst-plugins-base-1.5 1.5.90~20150904162934.164.g70e0c36.trusty amd64 Description: GObject introspection data for the GStreamer Plugins Base library ii gir1.2-gstreamer-1.0 1.2.4-0ubuntu1 amd64 Description: GObject introspection data for the GStreamer library ii gir1.2-gstreamer-1.5 1.5.90~20150904161713.208.g2893eef.trusty amd64 Description: GObject introspection data for the GStreamer library ii gstreamer0.10-pulseaudio:amd64 0.10.31-3+nmu1ubuntu5 amd64 GStreamer plugin for PulseAudio ii gstreamer1.0-clutter 2.0.8-1build1 amd64 Clutter PLugin for GStreamer 1.0 ii gstreamer1.0-plugins-base:amd64 1.2.4-1~ubuntu1 amd64 GStreamer plugins from the "base" set ii gstreamer1.0-plugins-good:amd64 1.2.4-1~ubuntu1 amd64 GStreamer plugins from the "good" set ii gstreamer1.0-x:amd64 1.2.4-1~ubuntu1 amd64 GStreamer plugins for X11 and Pango ii gstreamer1.5-alsa:amd64 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer plugin for ALSA ii gstreamer1.5-doc 1.5.2.1~20150901123757.112.ga536485.trusty all GStreamer core documentation and manuals ii gstreamer1.5-libav:amd64 1.5.2.1~20150901123759.34.g07a7b16.trusty amd64 libav plugin for GStreamer ii gstreamer1.5-libav-dbg:amd64 1.5.2.1~20150901123759.34.g07a7b16.trusty amd64 libav plugin for GStreamer (debug symbols) ii gstreamer1.5-nice:amd64 0.1.7.1~20150901123618.10.ge060eb5.trusty amd64 ICE library (GStreamer plugin) ii gstreamer1.5-plugins-bad:amd64 1.5.90~20150908080720.82.g02d110d.trusty amd64 GStreamer plugins from the "bad" set ii gstreamer1.5-plugins-bad-dbg:amd64 1.5.90~20150908080720.82.g02d110d.trusty amd64 GStreamer plugins from the "bad" set (debug symbols) ii gstreamer1.5-plugins-bad-doc 1.5.90~20150908080720.82.g02d110d.trusty all GStreamer documentation for plugins from the "bad" set ii gstreamer1.5-plugins-base:amd64 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer plugins from the "base" set ii gstreamer1.5-plugins-base-apps 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer helper programs from the "base" set ii gstreamer1.5-plugins-base-dbg:amd64 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer plugins from the "base" set ii gstreamer1.5-plugins-base-doc 1.5.90~20150904162934.164.g70e0c36.trusty all GStreamer documentation for plugins from the "base" set ii gstreamer1.5-plugins-good:amd64 1.5.90~20150911130940.37.g4b11097.trusty amd64 GStreamer plugins from the "good" set ii gstreamer1.5-plugins-good-dbg:amd64 1.5.90~20150911130940.37.g4b11097.trusty amd64 GStreamer plugins from the "good" set ii gstreamer1.5-plugins-good-doc 1.5.90~20150911130940.37.g4b11097.trusty all GStreamer documentation for plugins from the "good" set ii gstreamer1.5-plugins-ugly:amd64 1.5.90~20150904171312.32.g7b02ce4.trusty amd64 GStreamer plugins from the "ugly" set ii gstreamer1.5-plugins-ugly-dbg:amd64 1.5.90~20150904171312.32.g7b02ce4.trusty amd64 GStreamer plugins from the "ugly" set (debug symbols) ii gstreamer1.5-plugins-ugly-doc 1.5.90~20150904171312.32.g7b02ce4.trusty all GStreamer documentation for plugins from the "ugly" set ii gstreamer1.5-pulseaudio:amd64 1.5.90~20150911130940.37.g4b11097.trusty amd64 GStreamer plugin for PulseAudio ii gstreamer1.5-tools 1.5.2.1~20150901123757.112.ga536485.trusty amd64 Tools for use with GStreamer ii gstreamer1.5-x:amd64 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer plugins for X11 and Pango ii libclutter-gst-2.0-0:amd64 2.0.8-1build1 amd64 Open GL based interactive canvas library GStreamer elements ii libgstreamer-plugins-bad1.5-0:amd64 1.5.90~20150908080720.82.g02d110d.trusty amd64 GStreamer development files for libraries from the "bad" set ii libgstreamer-plugins-base0.10-0:amd64 0.10.36-1.1ubuntu2 amd64 GStreamer libraries from the "base" set ii libgstreamer-plugins-base1.0-0:amd64 1.2.4-1~ubuntu1 amd64 GStreamer libraries from the "base" set ii libgstreamer-plugins-base1.5-0:amd64 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer libraries from the "base" set ii libgstreamer-plugins-base1.5-dev 1.5.90~20150904162934.164.g70e0c36.trusty amd64 GStreamer development files for libraries from the "base" set ii libgstreamer-plugins-good1.0-0:amd64 1.2.4-1~ubuntu1 amd64 GStreamer development files for libraries from the "good" set ii libgstreamer0.10-0:amd64 0.10.36-1.2ubuntu3 amd64 Core GStreamer libraries and elements ii libgstreamer1.0-0:amd64 1.2.4-0ubuntu1 amd64 Core GStreamer libraries and elements ii libgstreamer1.0-dev 1.2.4-0ubuntu1 amd64 GStreamer core development files ii libgstreamer1.5-0:amd64 1.5.90~20150904161713.208.g2893eef.trusty amd64 Core GStreamer libraries and elements ii libgstreamer1.5-0-dbg:amd64 1.5.90~20150904161713.208.g2893eef.trusty amd64 Core GStreamer libraries and elements ii libgstreamer1.5-dev 1.5.90~20150904161713.208.g2893eef.trusty amd64 GStreamer core development files ii openh264-gst-plugins-bad-1.5:amd64 1.5.90~20150908080720.82.g02d110d.trusty amd64 GStreamer plugins from openh264 ii openwebrtc-gst-plugins 0.10.0~20150901123621.96.gf9d5cbe.trusty amd64 OpenWebRTC specific GStreamer plugins tsonghong@topper:/tmp/cores$

ANd a dump of one of the gdb core print:

tsonghong@topper:/tmp/cores$ sudo gdb /usr/bin/kurento-media-server core.rtpjitterbuffer.2173.topper.1444919632
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kurento-media-server'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:79
79  ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:79
#1  0x00007f17306f18eb in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=0x7f11b038714a)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#2  gst_rtp_h264_set_src_caps (rtph264depay=rtph264depay@entry=0x7f14d4061570) at gstrtph264depay.c:367
#3  0x00007f17306f2432 in gst_rtp_h264_depay_handle_nal (rtph264depay=rtph264depay@entry=0x7f14d4061570, 
    nal=nal@entry=0x7f1317e723e0, in_timestamp=in_timestamp@entry=308597263135, marker=marker@entry=1)
    at gstrtph264depay.c:826
#4  0x00007f17306f302b in gst_rtp_h264_depay_process (depayload=depayload@entry=0x7f14d4061570, 
    rtp=rtp@entry=0x7f14a57f9380) at gstrtph264depay.c:1213
#5  0x00007f17c3d0db23 in gst_rtp_base_depayload_handle_buffer (filter=filter@entry=0x7f14d4061570, 
    in=in@entry=0x7f1317e72a40, bclass=<optimized out>, bclass=<optimized out>) at gstrtpbasedepayload.c:444
#6  0x00007f17c3d0e325 in gst_rtp_base_depayload_chain (pad=pad@entry=0x7f146400d020, 
    parent=parent@entry=0x7f14d4061570, in=in@entry=0x7f1317e72a40) at gstrtpbasedepayload.c:508
#7  0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400d020)
    at gstpad.c:4088
#8  gst_pad_push_data (pad=pad@entry=0x7f172c3e2c80, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#9  0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
#10 0x00007f17c6faf9fb in gst_proxy_pad_chain_default (pad=pad@entry=0x7f172c3e1b20, 
    parent=parent@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40) at gstghostpad.c:126
#11 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f172c3e1b20)
    at gstpad.c:4088
#12 gst_pad_push_data (pad=pad@entry=0x7f146400c960, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#13 0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f146400c960, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
#14 0x00007f176c3375ab in gst_rtp_pt_demux_chain (pad=pad@entry=0x7f146400c4e0, parent=parent@entry=0x7f16e41eb900, 
    buf=buf@entry=0x7f1317e72a40) at gstrtpptdemux.c:442
#15 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400c4e0)
    at gstpad.c:4088
#16 gst_pad_push_data (pad=pad@entry=0x7f146400c060, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#17 0x00007f17c6fc5e5e in gst_pad_push (pad=0x7f146400c060, buffer=buffer@entry=0x7f1317e72a40) at gstpad.c:4459
#18 0x00007f176c331fbe in pop_and_push_next (jitterbuffer=jitterbuffer@entry=0x7f156802b610, seqnum=39076)
    at gstrtpjitterbuffer.c:2979
#19 0x00007f176c332c69 in handle_next_buffer (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3078
#20 gst_rtp_jitter_buffer_loop (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3513
#21 0x00007f17c6feeaae in gst_task_func (task=0x7f16e41eba70) at gsttask.c:343
#22 0x00007f17c6a637cc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007f17c6a62e45 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#24 0x00007f17c80ab182 in start_thread (arg=0x7f14a57fa700) at pthread_create.c:312
#25 0x00007f17c58d047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) where full
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:79
No locals.
#1  0x00007f17306f18eb in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=0x7f11b038714a)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:51
No locals.
#2  gst_rtp_h264_set_src_caps (rtph264depay=rtph264depay@entry=0x7f14d4061570) at gstrtph264depay.c:367
        data = 0x7f11b038714a "gB\300\036\032\062\065\001@z@<\"\021\250"
        len = <optimized out>
        new_size = <optimized out>
        i = 31
        profile_compat = <optimized out>
        codec_data = 0x7f1317e724f0
        map = {memory = 0x7f11124ae010, flags = (GST_MAP_READ | GST_MAP_WRITE), 
          data = 0x7f11124ae0a0 "\001B\300\036\377\340\216\242gB\300\036\034\062\065\001@z@<\"\021\250", 
          size = 2641755421, maxsize = 2641755428, user_data = {0x7f17c6fb31f7 <gst_debug_log+135>, 0x7f17307213fd, 
            0x7f14a57f8f48, 0x1}, _gst_reserved = {0x3000000030, 0x7f14a57f9028, 0x7f14a57f8f60, 0x4}}
        nalmap = {memory = 0x7f13e44594c0, flags = GST_MAP_READ, 
          data = 0x7f0db19f20a4 "gB\300\036\032\062\065\001@z@<\"\021\250", size = 4287130303, maxsize = 4287130310, 
          user_data = {0x7f17c6f8d316 <gst_buffer_map_range+534>, 0x1198830, 0x7f17c6f7fb21 <gst_object_ref+33>, 0xc}, 
          _gst_reserved = {0x7f13d33fbd70, 0xc, 0x1, 0x7f13c5a88ad0}}
        level = <optimized out>
        res = <optimized out>
        srccaps = 0x7f1484123de0
        __FUNCTION__ = "gst_rtp_h264_set_src_caps"
#3  0x00007f17306f2432 in gst_rtp_h264_depay_handle_nal (rtph264depay=rtph264depay@entry=0x7f14d4061570, 
    nal=nal@entry=0x7f1317e723e0, in_timestamp=in_timestamp@entry=308597263135, marker=marker@entry=1)
    at gstrtph264depay.c:826
        depayload = 0x7f14d4061570
        nal_type = 1
        map = {memory = 0x7f13c5a88ad0, flags = GST_MAP_READ, data = 0x7f13c5a88b60 "", size = 16, maxsize = 23, 
          user_data = {0x0, 0x0, 0x7f17c6a2d330 <g_hash_table_lookup+176>, 0xd4061570}, _gst_reserved = {
            0x7f1317e723e0, 0x7f146403275c, 0xc, 0x1}}
        outbuf = 0x0
        out_timestamp = 308597263135
        keyframe = 0
        out_keyframe = 0
        __FUNCTION__ = "gst_rtp_h264_depay_handle_nal"
#4  0x00007f17306f302b in gst_rtp_h264_depay_process (depayload=depayload@entry=0x7f14d4061570, 
    rtp=rtp@entry=0x7f14a57f9380) at gstrtph264depay.c:1213
        outsize = 16
        marker = 1
        payload_len = 12
        payload = 0x7f146403275c "a\301\020"
        header_len = 1
        nal_ref_idc = <optimized out>
        map = {memory = 0x7f13c5a88ad0, flags = GST_MAP_WRITE, data = 0x7f13c5a88b60 "", size = 16, maxsize = 23, 
---Type <return> to continue, or q <return> to quit--- 
          user_data = {0x7f1317e721c0, 0x7f1317e72600, 0x7f14a57f93d0, 0x18}, _gst_reserved = {0xca57f9380, 0x18, 0xc, 
            0x7f14a57f9380}}
        nalu_size = 12
        timestamp = 308597263135
        rtph264depay = 0x7f14d4061570
        buf = 0x7f1317e5e7f0
        outbuf = 0x7f1317e723e0
        nal_unit_type = 1 '\001'
        __FUNCTION__ = "gst_rtp_h264_depay_process"
#5  0x00007f17c3d0db23 in gst_rtp_base_depayload_handle_buffer (filter=filter@entry=0x7f14d4061570, 
    in=in@entry=0x7f1317e72a40, bclass=<optimized out>, bclass=<optimized out>) at gstrtpbasedepayload.c:444
        process_rtp_packet_func = <optimized out>
        process_func = <optimized out>
        priv = 0x7f14d4061510
        ret = GST_FLOW_OK
        out_buf = <optimized out>
        seqnum = <optimized out>
        rtptime = <optimized out>
        discont = <optimized out>
        buf_discont = <optimized out>
        gap = <optimized out>
        rtp = {buffer = 0x7f1317e72a40, state = 0, data = {0x7f1464032750, 0x0, 0x7f146403275c, 0x0}, size = {12, 0, 
            12, 0}, map = {{memory = 0x7f14640326c0, flags = GST_MAP_READ, 
              data = 0x7f1464032750 "\200\376\230\244\376+\356\316\003\363\001;a\301\020", size = 24, maxsize = 41, 
              user_data = {0x0, 0x0, 0x0, 0x0}, _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {memory = 0x0, 
              flags = (unknown: 0), data = 0x0, size = 0, maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, 
              _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {memory = 0x0, flags = (unknown: 0), data = 0x0, size = 0, 
              maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {memory = 0x0, 
              flags = (unknown: 0), data = 0x0, size = 0, maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, 
              _gst_reserved = {0x0, 0x0, 0x0, 0x0}}}}
#6  0x00007f17c3d0e325 in gst_rtp_base_depayload_chain (pad=pad@entry=0x7f146400d020, 
    parent=parent@entry=0x7f14d4061570, in=in@entry=0x7f1317e72a40) at gstrtpbasedepayload.c:508
        bclass = <optimized out>
        basedepay = 0x7f14d4061570
        flow_ret = <optimized out>
#7  0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400d020)
    at gstpad.c:4088
        chainfunc = 0x7f17c3d0e300 <gst_rtp_base_depayload_chain>
        parent = 0x7f14d4061570
        ret = <optimized out>
        handled = 0
#8  gst_pad_push_data (pad=pad@entry=0x7f172c3e2c80, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
        peer = 0x7f146400d020
        ret = <optimized out>
        handled = 0
---Type <return> to continue, or q <return> to quit---
        __PRETTY_FUNCTION__ = "gst_pad_push_data"
#9  0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
        res = <optimized out>
        __func__ = "gst_pad_push"
#10 0x00007f17c6faf9fb in gst_proxy_pad_chain_default (pad=pad@entry=0x7f172c3e1b20, 
    parent=parent@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40) at gstghostpad.c:126
        res = <optimized out>
        internal = 0x7f172c3e2c80
        __func__ = "gst_proxy_pad_chain_default"
#11 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f172c3e1b20)
    at gstpad.c:4088
        chainfunc = 0x7f17c6faf940 <gst_proxy_pad_chain_default>
        parent = 0x7f172c3e2c80
        ret = <optimized out>
        handled = 0
#12 gst_pad_push_data (pad=pad@entry=0x7f146400c960, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
        peer = 0x7f172c3e1b20
        ret = <optimized out>
        handled = 0
        __PRETTY_FUNCTION__ = "gst_pad_push_data"
#13 0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f146400c960, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
        res = <optimized out>
        __func__ = "gst_pad_push"
#14 0x00007f176c3375ab in gst_rtp_pt_demux_chain (pad=pad@entry=0x7f146400c4e0, parent=parent@entry=0x7f16e41eb900, 
    buf=buf@entry=0x7f1317e72a40) at gstrtpptdemux.c:442
        ret = GST_FLOW_OK
        rtpdemux = 0x7f16e41eb900
        pt = <optimized out>
        srcpad = 0x7f146400c960
        caps = <optimized out>
        rtp = {buffer = 0x0, state = 0, data = {0x0, 0x0, 0x0, 0x0}, size = {0, 0, 0, 0}, map = {{memory = 0x0, 
              flags = GST_MAP_READ, data = 0x7f1464032750 "\200\376\230\244\376+\356\316\003\363\001;a\301\020", 
              size = 24, maxsize = 41, user_data = {0x0, 0x0, 0x0, 0x0}, _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {
              memory = 0x0, flags = (unknown: 0), data = 0x0, size = 0, maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, 
              _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {memory = 0x0, flags = (unknown: 0), data = 0x0, size = 0, 
              maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, _gst_reserved = {0x0, 0x0, 0x0, 0x0}}, {memory = 0x0, 
              flags = (unknown: 0), data = 0x0, size = 0, maxsize = 0, user_data = {0x0, 0x0, 0x0, 0x0}, 
              _gst_reserved = {0x0, 0x0, 0x0, 0x0}}}}
        __FUNCTION__ = "gst_rtp_pt_demux_chain"
#15 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400c4e0)
    at gstpad.c:4088
        chainfunc = 0x7f176c3372e0 <gst_rtp_pt_demux_chain>
        parent = 0x7f16e41eb900
---Type <return> to continue, or q <return> to quit---
        ret = <optimized out>
        handled = 0
#16 gst_pad_push_data (pad=pad@entry=0x7f146400c060, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
        peer = 0x7f146400c4e0
        ret = <optimized out>
        handled = 0
        __PRETTY_FUNCTION__ = "gst_pad_push_data"
#17 0x00007f17c6fc5e5e in gst_pad_push (pad=0x7f146400c060, buffer=buffer@entry=0x7f1317e72a40) at gstpad.c:4459
        res = <optimized out>
        __func__ = "gst_pad_push"
#18 0x00007f176c331fbe in pop_and_push_next (jitterbuffer=jitterbuffer@entry=0x7f156802b610, seqnum=39076)
    at gstrtpjitterbuffer.c:2979
        priv = 0x7f156802b350
        result = GST_FLOW_OK
        item = <optimized out>
        outbuf = 0x7f1317e72a40
        outevent = 0x0
        outquery = 0x0
        dts = <optimized out>
        pts = <optimized out>
        percent = -1
        do_push = 1
        type = <optimized out>
        msg = 0x0
        __FUNCTION__ = "pop_and_push_next"
#19 0x00007f176c332c69 in handle_next_buffer (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3078
        gap = <optimized out>
        next_seqnum = <optimized out>
        priv = 0x7f156802b350
        item = <optimized out>
        seqnum = <optimized out>
#20 gst_rtp_jitter_buffer_loop (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3513
        priv = 0x7f156802b350
        result = <optimized out>
        __FUNCTION__ = "gst_rtp_jitter_buffer_loop"
#21 0x00007f17c6feeaae in gst_task_func (task=0x7f16e41eba70) at gsttask.c:343
        lock = 0x7f146400c0d0
        tself = 0x7f1464007190
        priv = 0x7f16e41eba20
        __PRETTY_FUNCTION__ = "gst_task_func"
#22 0x00007f17c6a637cc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#23 0x00007f17c6a62e45 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#24 0x00007f17c80ab182 in start_thread (arg=0x7f14a57fa700) at pthread_create.c:312
---Type <return> to continue, or q <return> to quit---
        __res = <optimized out>
        pd = 0x7f14a57fa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139726652679936, -608099250359815056, 0, 0, 139726652680640, 
                139726652679936, 695474137587463280, 693564106158191728}, mask_was_saved = 0}}, priv = {pad = {0x0, 
              0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#25 0x00007f17c58d047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
(gdb) 
(gdb) bt
#0  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:79
#1  0x00007f17306f18eb in memcpy (__len=<optimized out>, __src=<optimized out>, __dest=0x7f11b038714a)
    at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#2  gst_rtp_h264_set_src_caps (rtph264depay=rtph264depay@entry=0x7f14d4061570) at gstrtph264depay.c:367
#3  0x00007f17306f2432 in gst_rtp_h264_depay_handle_nal (rtph264depay=rtph264depay@entry=0x7f14d4061570, 
    nal=nal@entry=0x7f1317e723e0, in_timestamp=in_timestamp@entry=308597263135, marker=marker@entry=1)
    at gstrtph264depay.c:826
#4  0x00007f17306f302b in gst_rtp_h264_depay_process (depayload=depayload@entry=0x7f14d4061570, 
    rtp=rtp@entry=0x7f14a57f9380) at gstrtph264depay.c:1213
#5  0x00007f17c3d0db23 in gst_rtp_base_depayload_handle_buffer (filter=filter@entry=0x7f14d4061570, 
    in=in@entry=0x7f1317e72a40, bclass=<optimized out>, bclass=<optimized out>) at gstrtpbasedepayload.c:444
#6  0x00007f17c3d0e325 in gst_rtp_base_depayload_chain (pad=pad@entry=0x7f146400d020, 
    parent=parent@entry=0x7f14d4061570, in=in@entry=0x7f1317e72a40) at gstrtpbasedepayload.c:508
#7  0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400d020)
    at gstpad.c:4088
#8  gst_pad_push_data (pad=pad@entry=0x7f172c3e2c80, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#9  0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
#10 0x00007f17c6faf9fb in gst_proxy_pad_chain_default (pad=pad@entry=0x7f172c3e1b20, 
    parent=parent@entry=0x7f172c3e2c80, buffer=buffer@entry=0x7f1317e72a40) at gstghostpad.c:126
#11 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f172c3e1b20)
    at gstpad.c:4088
#12 gst_pad_push_data (pad=pad@entry=0x7f146400c960, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#13 0x00007f17c6fc5e5e in gst_pad_push (pad=pad@entry=0x7f146400c960, buffer=buffer@entry=0x7f1317e72a40)
    at gstpad.c:4459
#14 0x00007f176c3375ab in gst_rtp_pt_demux_chain (pad=pad@entry=0x7f146400c4e0, parent=parent@entry=0x7f16e41eb900, 
    buf=buf@entry=0x7f1317e72a40) at gstrtpptdemux.c:442
#15 0x00007f17c6fbe4d5 in gst_pad_chain_data_unchecked (data=0x7f1317e72a40, type=<optimized out>, pad=0x7f146400c4e0)
    at gstpad.c:4088
#16 gst_pad_push_data (pad=pad@entry=0x7f146400c060, type=type@entry=4112, data=<optimized out>, 
    data@entry=0x7f1317e72a40) at gstpad.c:4340
#17 0x00007f17c6fc5e5e in gst_pad_push (pad=0x7f146400c060, buffer=buffer@entry=0x7f1317e72a40) at gstpad.c:4459
#18 0x00007f176c331fbe in pop_and_push_next (jitterbuffer=jitterbuffer@entry=0x7f156802b610, seqnum=39076)
    at gstrtpjitterbuffer.c:2979
#19 0x00007f176c332c69 in handle_next_buffer (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3078
#20 gst_rtp_jitter_buffer_loop (jitterbuffer=0x7f156802b610) at gstrtpjitterbuffer.c:3513
#21 0x00007f17c6feeaae in gst_task_func (task=0x7f16e41eba70) at gsttask.c:343
#22 0x00007f17c6a637cc in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007f17c6a62e45 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#24 0x00007f17c80ab182 in start_thread (arg=0x7f14a57fa700) at pthread_create.c:312
#25 0x00007f17c58d047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) 

If you have any ideas, let me know. thanks!

Henry

[igracia]

Henry, could you please tell us how to reproduce the bug. Next time, please attach the log files, and paste only the significant lines. Otherwise it's quite hard to read ;-)

@jcaden I'm assigning this to you for triage

[jcaden]

Stack trace is fine in this case instead of the log.

@tsonghong is this error happening all the times? I mean is a random error or is it consistent?

Video source should be also important in case it is consistent.

[tsonghong]

Sorry for the long post, I'll attach files next time.

Right now, it seems to be random as I have not figured out a pattern to cause this. So, it appears to crash maybe average 3 times a week with this. but there are multiple streams going on the server so it's hard to say it's not reproducible.

We use H.264 streams and in terms of endpoints, only the WebRTCEndpoint and CV Filter plugin endpoints.

[tsonghong]

Ok, I think I know how to reproduce this issue more easily and was able to reproduce it with the ChromaFilter. But in general, it appears any CV Filter (with possible exception of the FaceDetection one) can reproduce this.

So the pipeline setup is as such:

WebRTCEndpoint#1 --> ChromaFilter --> WebRTCEndpoint#2

where a H.264 camera source is streaming to WebRTCEndpoint#1 and the user on Firefox is viewing on WebRCEndpoint#2.

To reproduce the issue:

1) Setup the pipeline as described above so that the User BrowserFirefox is able to see the video. The feed we are using is H.264. (Not sure if this isn't a problem in VP8 or not.) 2) Then, disconnect the link between the ChromaFilter and the WebRTCEndpoint#2 using the disconnect() function ChromaFilter side and wait for maybe 30 seconds or so. (Seems like this issue is more likely to happen if the wait is not short). 3) Then, reconnect back the ChromaFilter to the WebRTCEndpoint#2. 4) Then try to call any kurento client api command (like query the getMediaState() of the WebRTCEndpoint) and it could be possible you'll get a crash as the KurentoClient usually gets hosed at this point and throws the exception I pasted after my name below. If not, you may have to repeat from 2) again. If the first disconnect/reconnect doesn't crash the first time, I've noticed I'm usually able to crash on the 2nd or 3rd attempt of this.

Thanks!

Henry

P.S. I tried to attach text file ,but get this error: " Attaching documents requires write permission to this repository. Try again with a PNG, GIF, or JPG. "

Anyway, Kurento client javascript thrown exception error from step 4:

"

Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [SyntaxError: Client has been disconnected] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11) Trace: [TypeError: Cannot read property '1' of undefined] at callback2 (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/promisecallback/index.js:32:17) at lib$es6$promise$$internal$$tryCatch (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:331:16) at lib$es6$promise$$internal$$invokeCallback (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:343:17) at lib$es6$promise$$internal$$publish (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:314:11) at lib$es6$promise$$internal$$publishRejection (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:264:7) at lib$es6$promise$asap$$flush (/home/umbo/LaughingOcto/ss/node_modules/kurento-client/node_modules/es6-promise/dist/es6-promise.js:125:9) at process._tickDomainCallback (node.js:381:11)

"

[tsonghong]

Ok, the puzzle is much more clearer now with some more testing: It appears that if the CV Filter is running and feeding 1 or more WebRTCEndpoint sinks, but if you then disconnect all the WebRTCEndpoint sinks from the CV Filter, we see the memory of the Kurento Media Server skyrocket eating multiple Gigabytes of memory every few seconds. I.e. So long as one sink is still present, the memory won't increase much. But once all are disconnected, then we see the problem. I suspect the behaviour is because it's buffering everything since it has no 'sink' to successfully feed to.

Anyway, this explains why the core dumps were so large (multi gig files):

tsonghong@topper:~$ ls -rlt /tmp/cores total 242926480 -rw------- 1 nobody nogroup 20127789056 Oct 15 09:35 core.rtpjitterbuffer.2173.topper.1444919632 -rw------- 1 nobody nogroup 27295145984 Oct 16 02:08 core.rtpjitterbuffer.3590.topper.1444979143 -rw------- 1 nobody nogroup 16649363456 Oct 18 00:48 core.rtpjitterbuffer.7274.topper.1445147282 -rw------- 1 nobody nogroup 53773533184 Oct 18 01:21 core.rtpjitterbuffer.21281.topper.1445148852 -rw------- 1 nobody nogroup 37356347392 Oct 18 01:54 core.rtpjitterbuffer.21853.topper.1445151002 -rw------- 1 nobody nogroup 19904397312 Oct 18 02:25 core.rtpjitterbuffer.22252.topper.1445153028 -rw------- 1 nobody nogroup 17891684352 Oct 18 06:31 core.rtpjitterbuffer.22584.topper.1445167829 -rw------- 1 nobody nogroup 25838473216 Oct 18 06:44 core.rtpjitterbuffer.23685.topper.1445168517 -rw------- 1 nobody nogroup 15030820864 Oct 18 07:21 core.rtpjitterbuffer.24019.topper.1445170819 -rw------- 1 nobody nogroup 15541346304 Oct 18 07:32 core.rtpjitterbuffer.24377.topper.1445171472 -rw------- 1 nobody nogroup 15485710336 Oct 18 08:40 core.rtpjitterbuffer.32621.topper.1445175588 tsonghong@topper:~$

Any thoughts on this? The use case I have for this 'disconnect' behavior is that the User/WebRTC Endpoint is allowed to 'pause' from seeing the CV Fitler results. And all users can pause if they want to at any time they want. My preferred behavior is that the CV Filter not buffer and just drop when it has no sink to feed to.

[jcaden]

Reading the stack trace and your last comment, It seems that you are running out of memory and because of this, depayloader fails when trying to get memory for mapping the buffers.

Are the opencv filters generated by you? Have you observed that this filters does not leak memory?

[jcaden]

One more question, are you using this elements on a custom pipeline or are they used inside Kurento Media Server?

[tsonghong]

Thanks, I'm using the ChromaFilter provided by Kurento. From what I observed, it doesn't leak if 1) it hasn't connected to any sink yet or 2) it's at least connected to one sink. If all previosuly connected sinks are disconnected, I see the memory sky rocket. Normally, when the memory doesn't increase the KMS is just hovering around 800 MB.

If I don't have any Filter and just use WebRTC Endpoint to WebRTC Endpoint directly, I don't see any leaks even if I disconnect all the sink WebRTC Endpoints.

I'm using pure Javascript KurentoClient and calling disconnect() and connect() to detach and attach Endpoints. I presume this means non-custom pipeline.

[jcaden]

Thanks, so it seems that CrhomaFilter is leaking when there are no downstream elements. Have you observed that with other filters or just with chroma?

[tsonghong]

I observed this with the others as well with the possible exception of the FaceDetectionOne. Also, i verified my own CV Filter plugin has the same issue as well.

[sancane]

I could not reproduce this bug neither using vp8 nor h264. I've tested using a WebRtcEndpoint connected to a chroma and that chroma connectect back to the WebRtcEndpoint, then I disconnect the media flowing from the Chroma to the WebRtcEndpoint, at this point I only have the WebRtcEndPoint connected to the Chroma (no media flows from chroma to anywhere). CPU consuption remains stable, memory does not grow, everything keeps working as expected as long as I keep the demo running. I got tired after 15 minutes without experimenting any abnormal behaviour. I'm using the develop branch with Gstreamer 1.7

[tsonghong]

Instead of connecting back to the same WebRtcEndpoint (causing a loopback), can you connect to a different WebRtcEndpoint that feeds to another user ?

Thanks!

Henry

On Fri, Oct 30, 2015 at 4:36 PM, Santiago Carot-Nemesio < notifications@github.com> wrote:

I could not reproduce this bug neither using vp8 nor h264. I've tested using a WebRtcEndpoint connected to a chroma and that chroma connectect back to the WebRtcEndpoint, then I disconnect the media flowing from the Chroma to the WebRtcEndpoint, at this point I only have the WebRtcEndPoint connected to the Chroma (no media flows from chroma to anywhere). CPU consuption remains stable, memory does not grow, everything keeps working as expected as long as I keep the demo running. I got tired after 15 minutes without experimenting any abnormal behaviour. I'm using the develop branch with Gstreamer 1.7

— Reply to this email directly or view it on GitHub https://github.com/Kurento/bugtracker/issues/5#issuecomment-152460210.

We do everything against time in search of a complete life.

[jcaden]

We have perform several tests and we are not able to reproduce the issue.

If you think the issue is still there in the development version feel free to re open the issue adding a test we can run to debug the issue.