double free crash when stopping an H.265 RTSP stream

[zchunhai]

Prerequisites

• [x] I have read the SUPPORT document

• [x] I have checked the Troubleshooting Guide

• [x] I have tested with the latest version of Kurento

Issue description

I am testing to play RTSP video stream. The test code is: https://github.com/lulop-k/kurento-rtsp2webrtc

The output source of the RTSP video stream is the NVR device of hikvision. The video stream address format is: rtsp://player:1234qwer@172.16.0.168:554/Streaming/tracks/201?transportmode=unicast&starttime=20200429t063818z&endtime=20200429t063828z

When the video ends playing, Kurento will crash:

0:07:26.551899963     1 0x7f1d18008800 FIXME           rtph265depay gstrtph265depay.c:1151:gst_rtp_h265_depay_process:<rtph265depay0> Assuming DONL field is not present
0:07:26.551910199     1 0x7f1d18008800 FIXME           rtph265depay gstrtph265depay.c:1151:gst_rtp_h265_depay_process:<rtph265depay0> Assuming DONL field is not present
0:07:26.551931041     1 0x7f1d18008800 FIXME              h265parse gsth265parse.c:1639:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
0:07:26.596075664     1 0x7f1d18008800 FIXME              h265parse gsth265parse.c:1639:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message
0:07:26.606947779     1 0x7f1b98002c00 WARN               rtpsource rtpsource.c:1171:update_receiver_stats: unacceptable seqnum received (seqnr 18, delta -3427, packet_rate: 278, max_dropout: 16680, max_misorder: 556)
0:07:26.607130253     1 0x7f1b98002c00 WARN               rtpsource rtpsource.c:983:calculate_jitter: cannot get clock-rate for pt 112
*** Error in `/usr/bin/kurento-media-server': double free or corruption (!prev): 0x00007f1bd80ba9e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f1d715b97e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f1d715c237a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f1d715c653c]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlibav.so(+0x8980c)[0x7f1cc279c80c]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlibav.so(+0xa4158)[0x7f1cc27b7158]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlibav.so(+0xce498)[0x7f1cc27e1498]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlibav.so(+0xde3b3)[0x7f1cc27f13b3]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstlibav.so(+0xde4cf)[0x7f1cc27f14cf]
/usr/lib/x86_64-linux-gnu/libgstvideo-1.5.so.0(+0x38647)[0x7f1d7030e647]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(gst_element_change_state+0x3e)[0x7f1d7310624e]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x599c7)[0x7f1d731069c7]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x38515)[0x7f1d730e5515]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstplayback.so(+0x11e57)[0x7f1d5c327e57]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(gst_element_change_state+0x3e)[0x7f1d7310624e]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x599c7)[0x7f1d731069c7]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x38515)[0x7f1d730e5515]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstplayback.so(+0x1c865)[0x7f1d5c332865]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(gst_element_change_state+0x3e)[0x7f1d7310624e]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x599c7)[0x7f1d731069c7]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x38515)[0x7f1d730e5515]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(gst_element_change_state+0x3e)[0x7f1d7310624e]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(gst_element_change_state+0x95)[0x7f1d731062a5]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0(+0x599c7)[0x7f1d731069c7]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libkmselementsplugins.so(+0x9adc)[0x7f1d5d57dadc]
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libkmselementsplugins.so(+0x9b77)[0x7f1d5d57db77]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x15a)[0x7f1d72b9304a]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x4a3f0)[0x7f1d72b933f0]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_loop_run+0xc2)[0x7f1d72b93712]
/usr/lib/x86_64-linux-gnu/libkmsgstcommons.so.6(+0x21315)[0x7f1d7057c315]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x70c55)[0x7f1d72bb9c55]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f1d73c906ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1d7164941d]
======= Memory map: ========
55fac7253000-55fac745c000 r-xp 00000000 08:12 5378357                    /usr/bin/kurento-media-server
55fac765b000-55fac7663000 r--p 00208000 08:12 5378357                    /usr/bin/kurento-media-server
55fac7663000-55fac7665000 rw-p 00210000 08:12 5378357                    /usr/bin/kurento-media-server
55fac7a0c000-55fac7ca0000 rw-p 00000000 00:00 0                          [heap]
7f1b90000000-7f1b90022000 rw-p 00000000 00:00 0
7f1b90022000-7f1b94000000 ---p 00000000 00:00 0
7f1b949ff000-7f1b98000000 rw-p 00000000 00:00 0
7f1b98000000-7f1b98185000 rw-p 00000000 00:00 0
7f1b98185000-7f1b9c000000 ---p 00000000 00:00 0
7f1b9c000000-7f1b9c022000 rw-p 00000000 00:00 0
7f1b9c022000-7f1ba0000000 ---p 00000000 00:00 0
7f1ba0000000-7f1ba00ab000 rw-p 00000000 00:00 0
7f1ba00ab000-7f1ba4000000 ---p 00000000 00:00 0
7f1ba4000000-7f1ba4022000 rw-p 00000000 00:00 0
7f1ba4022000-7f1ba8000000 ---p 00000000 00:00 0
7f1ba8000000-7f1ba802c000 rw-p 00000000 00:00 0
7f1ba802c000-7f1bac000000 ---p 00000000 00:00 0
7f1bacd08000-7f1bad795000 rw-p 00000000 00:00 0
7f1bad795000-7f1bad79c000 r-xp 00000000 08:12 5378771                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoscale.so
7f1bad79c000-7f1bad99b000 ---p 00007000 08:12 5378771                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoscale.so
7f1bad99b000-7f1bad99c000 r--p 00006000 08:12 5378771                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoscale.so
7f1bad99c000-7f1bad99d000 rw-p 00007000 08:12 5378771                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoscale.so
7f1bad99d000-7f1bad9a2000 r-xp 00000000 08:12 5378762                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoconvert.so
7f1bad9a2000-7f1badba1000 ---p 00005000 08:12 5378762                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoconvert.so
7f1badba1000-7f1badba2000 r--p 00004000 08:12 5378762                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoconvert.so
7f1badba2000-7f1badba3000 rw-p 00005000 08:12 5378762                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideoconvert.so
7f1badba3000-7f1badbab000 r-xp 00000000 08:12 5378769                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideorate.so
7f1badbab000-7f1baddaa000 ---p 00008000 08:12 5378769                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideorate.so
7f1baddaa000-7f1baddab000 r--p 00007000 08:12 5378769                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideorate.so
7f1baddab000-7f1baddac000 rw-p 00008000 08:12 5378769                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvideorate.so
7f1baddac000-7f1badfcb000 r-xp 00000000 08:12 5379407                    /usr/lib/x86_64-linux-gnu/libvpx.so.3.0.0
7f1badfcb000-7f1bae1ca000 ---p 0021f000 08:12 5379407                    /usr/lib/x86_64-linux-gnu/libvpx.so.3.0.0
7f1bae1ca000-7f1bae1cc000 r--p 0021e000 08:12 5379407                    /usr/lib/x86_64-linux-gnu/libvpx.so.3.0.0
7f1bae1cc000-7f1bae1cd000 rw-p 00220000 08:12 5379407                    /usr/lib/x86_64-linux-gnu/libvpx.so.3.0.0
7f1bae1cd000-7f1bae1d0000 rw-p 00000000 00:00 0
7f1bae1d0000-7f1bae1e0000 r-xp 00000000 08:12 5378779                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvpx.so
7f1bae1e0000-7f1bae3e0000 ---p 00010000 08:12 5378779                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvpx.so
7f1bae3e0000-7f1bae3e1000 r--p 00010000 08:12 5378779                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvpx.so
7f1bae3e1000-7f1bae3e2000 rw-p 00011000 08:12 5378779                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstvpx.so
7f1baebe3000-7f1baebe4000 ---p 00000000 00:00 0
7f1baebe4000-7f1baf3e4000 rw-p 00000000 00:00 0
7f1baf3e4000-7f1baf3e5000 ---p 00000000 00:00 0
7f1baf3e5000-7f1bafbe5000 rw-p 00000000 00:00 0
7f1bafbe5000-7f1bafbec000 r-xp 00000000 08:12 5379291                    /usr/lib/x86_64-linux-gnu/liborc-test-0.4.so.0.25.0
7f1bafbec000-7f1bafdeb000 ---p 00007000 08:12 5379291                    /usr/lib/x86_64-linux-gnu/liborc-test-0.4.so.0.25.0
7f1bafdeb000-7f1bafdec000 r--p 00006000 08:12 5379291                    /usr/lib/x86_64-linux-gnu/liborc-test-0.4.so.0.25.0
7f1bafdec000-7f1bafded000 rw-p 00007000 08:12 5379291                    /usr/lib/x86_64-linux-gnu/liborc-test-0.4.so.0.25.0
7f1bafded000-7f1bafdfe000 r-xp 00000000 08:12 5378598                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioresample.so
7f1bafdfe000-7f1bafffe000 ---p 00011000 08:12 5378598                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioresample.so
7f1bafffe000-7f1baffff000 r--p 00011000 08:12 5378598                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioresample.so
7f1baffff000-7f1bb0000000 rw-p 00012000 08:12 5378598                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioresample.so
7f1bb0000000-7f1bb0031000 rw-p 00000000 00:00 0
7f1bb0031000-7f1bb4000000 ---p 00000000 00:00 0
7f1bb4000000-7f1bb4222000 rw-p 00000000 00:00 0
7f1bb4222000-7f1bb8000000 ---p 00000000 00:00 0
7f1bb8000000-7f1bb8022000 rw-p 00000000 00:00 0
7f1bb8022000-7f1bbc000000 ---p 00000000 00:00 0
7f1bbc000000-7f1bbc089000 rw-p 00000000 00:00 0
7f1bbc089000-7f1bc0000000 ---p 00000000 00:00 0
7f1bc0017000-7f1bc001c000 r-xp 00000000 08:12 5378592                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioconvert.so
7f1bc001c000-7f1bc021b000 ---p 00005000 08:12 5378592                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioconvert.so
7f1bc021b000-7f1bc021c000 r--p 00004000 08:12 5378592                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioconvert.so
7f1bc021c000-7f1bc021d000 rw-p 00005000 08:12 5378592                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstaudioconvert.so
7f1bc021d000-7f1bc0266000 r-xp 00000000 08:12 5379287                    /usr/lib/x86_64-linux-gnu/libopus.so.0.5.2
7f1bc0266000-7f1bc0465000 ---p 00049000 08:12 5379287                    /usr/lib/x86_64-linux-gnu/libopus.so.0.5.2
7f1bc0465000-7f1bc0466000 r--p 00048000 08:12 5379287                    /usr/lib/x86_64-linux-gnu/libopus.so.0.5.2
7f1bc0466000-7f1bc0467000 rw-p 00049000 08:12 5379287                    /usr/lib/x86_64-linux-gnu/libopus.so.0.5.2
7f1bc0467000-7f1bc0474000 r-xp 00000000 08:12 5378708                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopus.so
7f1bc0474000-7f1bc0673000 ---p 0000d000 08:12 5378708                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopus.so
7f1bc0673000-7f1bc0674000 r--p 0000c000 08:12 5378708                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopus.so
7f1bc0674000-7f1bc0675000 rw-p 0000d000 08:12 5378708                    /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstopus.so

Context

I am developing an application to play RTSP video stream through webRTC. When the video stream ends, it will inevitably lead to crash in kurento.

How to reproduce?

Reproduce steps:

https://github.com/lulop-k/kurento-rtsp2webrtc/blob/master/js/index.js

Expected & current behavior

kurento should not crash.

(Optional) Possible solution

Environment info

I use the docker image of kms, with tag 6.13.

docker run --name kms -d --net=host kurento/kurento-media-server:6.13

INFO: Kurento Media Server

• Kurento version: 6.13
• Server OS: docker
• Installation method:
  • [ ] apt-get
  • [x] Docker
  • [ ] AWS
  • [ ] Built from sources

INFO: Application Server

• Language:
• Kurento Client version:

INFO: End-user clients

• Device(s):
• OS(es):
• Browser(s):

Version details

root@cpu0:/# lsb_release -a
bash: lsb_release: command not found
root@cpu0:/#
root@cpu0:/# kurento-media-server -v
Kurento Media Server version: 6.13.1~7.g93fee4a
Found modules:
    'core' version 6.13.1~7.ga704765
    'elements' version 6.13.1~7.g61435ac
    'filters' version 6.13.1~1.gbee162d
root@cpu0:/#
root@cpu0:/# dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.5-alsa:amd64              1.8.1-1kurento2.16.04~20200416152946.gbp7fb324  amd64        GStreamer plugin for ALSA
ii  gstreamer1.5-libav:amd64             1.8.1-1kurento1.16.04~20200416152946.gbpb33143  amd64        libav plugin for GStreamer
ii  gstreamer1.5-libav-dbg:amd64         1.8.1-1kurento1.16.04~20200416152946.gbpb33143  amd64        libav plugin for GStreamer (debug symbols)
ii  gstreamer1.5-nice:amd64              0.1.16-1.16.04~20200416152946.gbp1854ac         amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-nice-dbgsym:amd64       0.1.16-1.16.04~20200416152946.gbp1854ac         amd64        debug symbols for package gstreamer1.5-nice
ii  gstreamer1.5-plugins-bad:amd64       1.8.1-1kurento4.16.04~20200416152946.gbp17d48d  amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-bad-dbg:amd64   1.8.1-1kurento4.16.04~20200416152946.gbp17d48d  amd64        GStreamer plugins from the "bad" set (debug symbols)
ii  gstreamer1.5-plugins-base:amd64      1.8.1-1kurento2.16.04~20200416152946.gbp7fb324  amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-base-dbg:amd64  1.8.1-1kurento2.16.04~20200416152946.gbp7fb324  amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64      1.8.1-1kurento3.16.04~20200416152946.gbpbe46c7  amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-good-dbg:amd64  1.8.1-1kurento3.16.04~20200416152946.gbpbe46c7  amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64      1.8.1-1kurento1.16.04~20200416152946.gbpbed8b0  amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-plugins-ugly-dbg:amd64  1.8.1-1kurento1.16.04~20200416152946.gbpbed8b0  amd64        GStreamer plugins from the "ugly" set (debug symbols)
ii  gstreamer1.5-pulseaudio:amd64        1.8.1-1kurento3.16.04~20200416152946.gbpbe46c7  amd64        GStreamer plugin for PulseAudio
ii  gstreamer1.5-x:amd64                 1.8.1-1kurento2.16.04~20200416152946.gbp7fb324  amd64        GStreamer plugins for X11 and Pango
ii  kms-core                             6.13.1-0kurento1.16.04~20200416152946.gbpa70476 amd64        Kurento Core module
ii  kms-core-dbg                         6.13.1-0kurento1.16.04~20200416152946.gbpa70476 amd64        Kurento Core module
ii  kms-elements                         6.13.1-0kurento1.16.04~20200416152946.gbp61435a amd64        Kurento Elements module
ii  kms-elements-dbg                     6.13.1-0kurento1.16.04~20200416152946.gbp61435a amd64        Kurento Elements module
ii  kms-filters                          6.13.1-0kurento1.16.04~20200416152946.gbpbee162 amd64        Kurento Filters module
ii  kms-filters-dbg                      6.13.1-0kurento1.16.04~20200416152946.gbpbee162 amd64        Kurento Filters module
ii  kms-jsonrpc                          6.13.1-0kurento1.16.04~20200416152946.gbpc45b46 amd64        Kurento JSON-RPC library
ii  kms-jsonrpc-dbg                      6.13.1-0kurento1.16.04~20200416152946.gbpc45b46 amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                           1.6.3-1kurento1.16.04~20200416152946.gbp15940b  amd64        Kurento jsoncpp library
ii  kmsjsoncpp-dbg                       1.6.3-1kurento1.16.04~20200416152946.gbp15940b  amd64        Kurento jsoncpp library
ii  kurento-dbg                          6.13.1-0kurento1.16.04~20200416152946.gbp93fee4 amd64        Meta-package that installs debug symbols
ii  kurento-media-server                 6.13.1-0kurento1.16.04~20200416152946.gbp93fee4 amd64        Kurento Media Server
ii  kurento-media-server-dbg             6.13.1-0kurento1.16.04~20200416152946.gbp93fee4 amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64  1.8.1-1kurento4.16.04~20200416152946.gbp17d48d  amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.16.04~20200416152946.gbp7fb324  amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64              1.8.1-1kurento2.16.04~20200416152946.gbp84c84b  amd64        Core GStreamer libraries and elements
ii  libgstreamer1.5-0-dbg:amd64          1.8.1-1kurento2.16.04~20200416152946.gbp84c84b  amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                      0.1.16-1.16.04~20200416152946.gbp1854ac         amd64        ICE library (shared library)
ii  libnice10-dbgsym:amd64               0.1.16-1.16.04~20200416152946.gbp1854ac         amd64        debug symbols for package libnice10
ii  libsrtp0:amd64                       1.6.0-0kurento1.16.04~20200416152946.gbp577e0a  amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                           0.9.2-1kurento1.16.04~20200416152946.gbpfd1d0e  amd64        sctp-refimpl library
ii  openh264                             1.4.0-1kurento1.16.04~20200416152946.gbpb8cb17  amd64        OpenH264 library
ii  openh264-gst-plugins-bad-1.5:amd64   1.8.1-1kurento4.16.04~20200416152946.gbp17d48d  amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins               0.10.0-1kurento1.16.04~20200416152946.gbp6a0d22 amd64        OpenWebRTC specific GStreamer plugins
ii  openwebrtc-gst-plugins-dbg           0.10.0-1kurento1.16.04~20200416152946.gbp6a0d22 amd64        OpenWebRTC specific GStreamer plugins

[j1elo]

Hi, thanks for reporting the crash.

Could you please...

a) Try again with the official kurento-player tutorial, which can also play RTSP streams.

b) Try with this Docker image: kurento/kurento-media-server:6.13.1-asan. It is a special build to analyze memory corruption bugs (like this one). Let me know if you get any message from AddressSanitizer with this image.

Note that the very first message is a known issue:

=================================================================
==10==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 49152 byte(s) in 2 object(s) allocated from:
    #0 0x7f1155a80f88 in malloc (/kurento-asan/libasan.so+0x10bf88)
    #1 0x7f11553897b8 in g_malloc /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gmem.c:94

so that one is not important and you can ignore it.

[zchunhai]

@j1elo I used the kurento-player tutorial and kurento/kurento-media-server:6.13.1-asan, but still can not play rtsp， the error is:

==10==LeakSanitizer has encountered a fatal error.
==10==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==10==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
10:01:22.608674559     9 0x61d0001f09e0 ERROR        rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
10:01:22.609588574     9 0x61d0001f09e0 ERROR        rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
10:01:24.610377050     9 0x61d0001f09e0 ERROR        rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
=================================================================
==9==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200017623a at pc 0x7fcf79c259ed bp 0x7fcf3c4674b0 sp 0x7fcf3c466c58
WRITE of size 12 at 0x60200017623a thread T49 (rtpjitterbuffer)
10:01:24.613687657     9 0x61d0001f09e0 ERROR        rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
    #0 0x7fcf79c259ec  (/kurento-asan/libasan.so+0x9a9ec)
    #1 0x7fcf449bb356 in strcpy /usr/include/x86_64-linux-gnu/bits/string3.h:110
    #2 0x7fcf449bb356 in gst_rtp_h265_depay_negotiate /workdir/gst/rtp/gstrtph265depay.c:216
    #3 0x7fcf449bb356 in gst_rtp_h265_depay_setcaps /workdir/gst/rtp/gstrtph265depay.c:773
    #4 0x7fcf74fad800 in gst_rtp_base_depayload_setcaps /workdir/gst-libs/gst/rtp/gstrtpbasedepayload.c:325
    #5 0x7fcf74fad800 in gst_rtp_base_depayload_handle_event /workdir/gst-libs/gst/rtp/gstrtpbasedepayload.c:597
    #6 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
    #7 0x7fcf77c7bba8 in gst_pad_send_event /workdir/gst/gstpad.c:5752
    #8 0x7fcf4a65ba27 in send_sticky_event /workdir/gst/playback/gstdecodebin2.c:1979
    #9 0x7fcf77c6e8c8 in foreach_dispatch_function /workdir/gst/gstpad.c:5851
    #10 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
    #11 0x7fcf77c7c6f3 in gst_pad_sticky_events_foreach /workdir/gst/gstpad.c:5882
    #12 0x7fcf4a658d8a in send_sticky_events /workdir/gst/playback/gstdecodebin2.c:1994
    #13 0x7fcf4a658d8a in connect_pad /workdir/gst/playback/gstdecodebin2.c:2494
    #14 0x7fcf4a664201 in analyze_new_pad /workdir/gst/playback/gstdecodebin2.c:1808
    #15 0x7fcf4a664994 in type_found /workdir/gst/playback/gstdecodebin2.c:2861
    #16 0x7fcf7542fe3f in ffi_call_unix64 (/usr/lib/x86_64-linux-gnu/libffi.so.6+0x5e3f)
    #17 0x7fcf7542f8aa in ffi_call (/usr/lib/x86_64-linux-gnu/libffi.so.6+0x58aa)
    #18 0x7fcf773117c8 in g_cclosure_marshal_generic /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:1490
    #19 0x7fcf77310fa4 in g_closure_invoke /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:804
    #20 0x7fcf77322fc0 in signal_emit_unlocked_R /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3629
    #21 0x7fcf7732bd5b in g_signal_emit_valist /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3385
    #22 0x7fcf7732c08e in g_signal_emit /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3441
    #23 0x7fcf47656a2e in gst_type_find_element_setcaps /workdir/plugins/elements/gsttypefindelement.c:784
    #24 0x7fcf47656a2e in gst_type_find_element_sink_event /workdir/plugins/elements/gsttypefindelement.c:676
    #25 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
    #26 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
    #27 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
    #28 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
    #29 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
    #30 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
    #31 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
    #32 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
    #33 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
    #34 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
    #35 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
    #36 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
    #37 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
    #38 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
    #39 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
    #40 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
    #41 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
    #42 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
    #43 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
    #44 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
    #45 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
    #46 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
    #47 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
    #48 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
    #49 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
    #50 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
    #51 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
    #52 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
    #53 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
    #54 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
    #55 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
    #56 0x7fcf77c72010 in check_sticky /workdir/gst/gstpad.c:3842
    #57 0x7fcf77c72010 in gst_pad_push_data /workdir/gst/gstpad.c:4413
    #58 0x7fcf77c7a532 in gst_pad_push /workdir/gst/gstpad.c:4554
    #59 0x7fcf4a1fe751 in gst_rtp_pt_demux_chain /workdir/gst/rtpmanager/gstrtpptdemux.c:442
    #60 0x7fcf77c725ce in gst_pad_chain_data_unchecked /workdir/gst/gstpad.c:4183
    #61 0x7fcf77c725ce in gst_pad_push_data /workdir/gst/gstpad.c:4435
    #62 0x7fcf77c7a532 in gst_pad_push /workdir/gst/gstpad.c:4554
    #63 0x7fcf4a1f89ef in pop_and_push_next /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3058
    #64 0x7fcf4a1f9a19 in handle_next_buffer /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3157
    #65 0x7fcf4a1f9a19 in gst_rtp_jitter_buffer_loop /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3593
    #66 0x7fcf77ca4c5a in gst_task_func /workdir/gst/gsttask.c:344
    #67 0x7fcf770615ed in g_thread_pool_thread_proxy /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthreadpool.c:307
    #68 0x7fcf77060c54 in g_thread_proxy /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread.c:780
    #69 0x7fcf78ddc6b9 in start_thread /build/glibc-LK5gWL/glibc-2.23/nptl/pthread_create.c:333
    #70 0x7fcf7673241c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

0x60200017623a is located 0 bytes to the right of 10-byte region [0x602000176230,0x60200017623a)
allocated by thread T49 (rtpjitterbuffer) here:
    #0 0x7fcf79c96f88 in malloc (/kurento-asan/libasan.so+0x10bf88)
    #1 0x7fcf7703f7b8 in g_malloc /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gmem.c:94

Thread T49 (rtpjitterbuffer) created by T44 (udpsrc0:src) here:
    #0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T44 (udpsrc0:src) created by T37 (task2) here:
    #0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T37 (task2) created by T16 here:
    #0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T16 created by T0 here:
    #0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fcf76cddad4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0ad4)
    #2 0x55906935f5a7 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
    #3 0x55906935f5a7 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:324
    #4 0x5590691d0b36 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
    #5 0x7fcf7664b82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/kurento-asan/libasan.so+0x9a9ec)
Shadow bytes around the buggy address:
  0x0c0480026bf0: fa fa fd fa fa fa 04 fa fa fa fd fa fa fa 04 fa
  0x0c0480026c00: fa fa 00 fa fa fa 07 fa fa fa 00 00 fa fa fd fa
  0x0c0480026c10: fa fa fd fa fa fa fd fa fa fa 05 fa fa fa fd fa
  0x0c0480026c20: fa fa 05 fa fa fa fd fa fa fa fd fa fa fa 05 fa
  0x0c0480026c30: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 04 fa
=>0x0c0480026c40: fa fa fd fa fa fa 00[02]fa fa 00 06 fa fa fd fd
  0x0c0480026c50: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fd
  0x0c0480026c60: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fa fa
  0x0c0480026c70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0480026c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0480026c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==9==ABORTING

[j1elo]

Hi @zchunhai , the "-asan" image is just to detect memory corruption errors. And it worked, it detected one heap overflow error!

I see in the stack trace that all levels are inside GStreamer code. There is no Kurento code seen here... so it must be a GStreamer bug.

Can you provide a link to the RTSP stream, so I can test some fixes? (in a comment here or privately, email address is in my profile)

[zchunhai]

Hi @zchunhai , the "-asan" image is just to detect memory corruption errors. And it worked, it detected one heap overflow error!

I see in the stack trace that all levels are inside GStreamer code. There is no Kurento code seen here... so it must be a GStreamer bug.

Can you provide a link to the RTSP stream, so I can test some fixes? (in a comment here or privately, email address is in my profile)

I have emailed you the rtsp stream address.

[j1elo]

Hi, thanks for sharing a test stream.

I was able to find out the GStreamer commit where they fixed the memory buffer overflow error, and backported it to Kurento.

I am now able to play the RTSP stream with kurento-player tutorial and the latest nightly build of KMS. I used these commands:

$ docker pull kurento/kurento-media-server-dev:latest
$ docker run --rm --net=host kurento/kurento-media-server-dev:latest

Let me know if it the nightly build works better than the 6.13

[sksamal]

I am getting the exact same issue with h264 stream. I am using kms 6.14.0 version. Kurento media player crashes with similar logs.. I see that this issue was with v 6.13. Has this fix been merged into the mainstream ?
Where can i get a version where this issue is fixed?

Thanks so much Suraj

[j1elo]

Please have a look at the latest version: https://doc-kurento.readthedocs.io/en/latest/user/installation.html Kurento 6.15 has received several improvements to GStreamer, so this issue is probably fixed.

Please, if you can verify that the crash doesn't happen, let us know so this issue report can be closed.

Thanks!

[sksamal]

The problem exists with 6.15.0 as well and i can confim its not fixed. At first we thought the issue is with specific camera h265 stream, but I can confirm it occurs with all the h265 streams. The streaming starts perfectly, but when we send the stop request, the docker crashes. (I used the kurento-player tutorial - https://doc-kurento.readthedocs.io/en/latest/tutorials/java/tutorial-player.html) I used the docker kurento/kurento-media-server:6.15.0-asan and generated these logs..

============kurento/kurento-media-server:6.15.0-asan================

0:07:26.598528751    13 0x619000411160 DEBUG         playerendpoint kmsplayerendpoint.c:919:kms_player_endpoint_stopped:<kmsplayerendpoint0> Pipeline stopped
0:07:26.598848219    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd WAIT
0:07:26.598882621    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:5024:gst_rtspsrc_loop_send_cmd:<source> cancel previous request LOOP
0:07:26.598904544    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:5032:gst_rtspsrc_loop_send_cmd:<source> connection flush busy LOOP
0:07:26.598923168    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 1
0:07:26.598943409    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:4185:gst_rtspsrc_connection_flush:<source> connection flush
0:07:26.599051898    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4782:gst_rtspsrc_loop_udp:<source> got interrupted
0:07:26.599087021    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5075:gst_rtspsrc_loop:<source> pausing task, reason flushing
0:07:26.599101435    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd WAIT
0:07:26.599110284    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5032:gst_rtspsrc_loop_send_cmd:<source> connection flush busy LOOP
0:07:26.599117464    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 1
0:07:26.599255584    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd PAUSE
0:07:26.599305748    13 0x619000411160 DEBUG                rtspsrc gstrtspsrc.c:5037:gst_rtspsrc_loop_send_cmd:<source> not interrupting busy cmd WAIT
0:07:26.599385561    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:7628:gst_rtspsrc_thread:<source> got command PAUSE
0:07:26.599450442    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 0
0:07:26.599535419    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4185:gst_rtspsrc_connection_flush:<source> connection flush
0:07:26.599578129    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:7424:gst_rtspsrc_pause:<source> PAUSE...
0:07:26.599921519    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5401:gst_rtspsrc_try_send:<source> sending message
0:07:26.601693429    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5430:gst_rtspsrc_try_send:<source> received response message
0:07:26.601764734    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:5445:gst_rtspsrc_try_send:<source> got response message 551
0:07:26.601819623    13 0x61d0001e54f0 WARN                 rtspsrc gstrtspsrc.c:5662:gst_rtspsrc_send:<source> error: Got error response: 551 (Option not supported).
0:07:26.602069437    13 0x61d0001e54f0 WARN                 rtspsrc gstrtspsrc.c:7526:gst_rtspsrc_pause:<source> error: Could not send message. (Generic error)
0:07:26.602106470    13 0x61d000103800 DEBUG         playerendpoint kmsplayerendpoint.c:1166:kms_player_endpoint_emit_invalid_uri_signal: Emit 'Invalid URI' signal
0:07:26.602126541    13 0x6190000d44f0 ERROR         playerendpoint kmsplayerendpoint.c:1335:process_bus_message:<kmsplayerendpoint0> Error code 9: 'Could not read from resource.', element: source, parent: internalpipeline
0:07:26.602202082    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:7628:gst_rtspsrc_thread:<source> got command LOOP
0:07:26.602252649    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 0
0:07:26.602246462    13 0x6190000d44f0 ERROR         playerendpoint kmsplayerendpoint.c:1340:process_bus_message:<kmsplayerendpoint0> Debugging info: gstrtspsrc.c(5662): gst_rtspsrc_send (): /GstPipeline:internalpipeline/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source:
Got error response: 551 (Option not supported).
0:07:26.602273896    13 0x61d0001e54f0 DEBUG                rtspsrc gstrtspsrc.c:4695:gst_rtspsrc_loop_udp:<source> doing receive with timeout 54 seconds
0:07:26.602331844    13 0x61d000103800 DEBUG         playerendpoint kmsplayerendpoint.c:1166:kms_player_endpoint_emit_invalid_uri_signal: Emit 'Invalid URI' signal
0:07:26.602318316    13 0x6190000d44f0 ERROR         playerendpoint kmsplayerendpoint.c:1335:process_bus_message:<kmsplayerendpoint0> Error code 10: 'Could not write to resource.', element: source, parent: internalpipeline
0:07:26.602388394    13 0x6190000d44f0 ERROR         playerendpoint kmsplayerendpoint.c:1340:process_bus_message:<kmsplayerendpoint0> Debugging info: gstrtspsrc.c(7526): gst_rtspsrc_pause (): /GstPipeline:internalpipeline/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source:
Could not send message. (Generic error)
=================================================================
==13==ERROR: AddressSanitizer: attempting double-free on 0x62c000010200 in thread T61:
    #0 0x7fa5b3d82c7f in __interceptor_free (/kurento-asan/libasan.so+0x10bc7f)
    #1 0x7fa54975d80b in hevc_decode_free libavcodec/hevc.c:3096
    #2 0x7fa549778157 in avcodec_close libavcodec/utils.c:2545
    #3 0x7fa5497a2497 in gst_ffmpeg_avcodec_close /build/ext/libav/gstav.c:79
    #4 0x7fa5497b23b2 in gst_ffmpegviddec_close /build/ext/libav/gstavviddec.c:320
    #5 0x7fa5497b24ce in gst_ffmpegviddec_stop /build/ext/libav/gstavviddec.c:1712
    #6 0x7fa5afb61646 in gst_video_decoder_change_state /build/gst-libs/gst/video/gstvideodecoder.c:2556
    #7 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
    #8 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
    #9 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
    #10 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
    #11 0x7fa56eb2ee56 in gst_decode_bin_change_state /build/gst/playback/gstdecodebin2.c:5296
    #12 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
    #13 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
    #14 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
    #15 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
    #16 0x7fa56eb39864 in gst_uri_decode_bin_change_state /build/gst/playback/gsturidecodebin.c:2752
    #17 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
    #18 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
    #19 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
    #20 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
    #21 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
    #22 0x7fa5b1d4b2a4 in gst_element_change_state /build/gst/gstelement.c:2694
    #23 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
    #24 0x7fa56f5a0b5c in kms_player_endpoint_mark_reset_base_time_and_set_state /kms-omni-build/kms-elements/src/gst-plugins/kmsplayerendpoint.c:444
    #25 0x7fa56f5a0b5c in kms_player_endpoint_stopped /kms-omni-build/kms-elements/src/gst-plugins/kmsplayerendpoint.c:922
    #26 0x7fa5afdd3544 in kms_uri_endpoint_set_state /build/src/gst-plugins/commons/kmsuriendpoint.c:152
    #27 0x7fa5b29a84b0 in kurento::UriEndpointImpl::stop() /build/src/server/implementation/objects/UriEndpointImpl.cpp:177
    #28 0x7fa5b2a1bcb8 in kurento::UriEndpointMethodStop::invoke(std::shared_ptr<kurento::UriEndpoint>) /build/obj-x86_64-linux-gnu/src/server/interface/generated-cpp/UriEndpointInternal.cpp:39
    #29 0x7fa5b2a00d90 in kurento::UriEndpointImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&, Json::Value&) /build/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/UriEndpointImplInternal.cpp:37
    #30 0x7fa592dc4789 in kurento::PlayerEndpointImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&, Json::Value&) /build/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/PlayerEndpointImplInternal.cpp:89
    #31 0x556aefdfad3c in kurento::ServerMethods::invoke(Json::Value const&, Json::Value&) /kms-omni-build/kurento-media-server/server/ServerMethods.cpp:603
    #32 0x556aefe1381c in void std::__invoke_impl<void, void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>(std::__invoke_memfun_deref, void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/invoke.h:73
    #33 0x556aefe1381c in std::__invoke_result<void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>::type std::__invoke<void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>(void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/invoke.h:95
    #34 0x556aefe1381c in void std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)>::__call<void, Json::Value const&, Json::Value&, 0ul, 1ul, 2ul>(std::tuple<Json::Value const&, Json::Value&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/9/functional:400
    #35 0x556aefe1381c in void std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)>::operator()<Json::Value const&, Json::Value&, void>(Json::Value const&, Json::Value&) /usr/include/c++/9/functional:484
    #36 0x556aefe1381c in std::_Function_handler<void (Json::Value const&, Json::Value&), std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)> >::_M_invoke(std::_Any_data const&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/std_function.h:300
    #37 0x7fa5b26ce7f9 in std::function<void (Json::Value const&, Json::Value&)>::operator()(Json::Value const&, Json::Value&) const /usr/include/c++/5/functional:2267
    #38 0x7fa5b26ce7f9 in kurento::JsonRpc::Handler::process(Json::Value const&, Json::Value&) /build/src/JsonRpcHandler.cpp:122
    #39 0x556aefe1075f in kurento::ServerMethods::process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /kms-omni-build/kurento-media-server/server/ServerMethods.cpp:277
    #40 0x556af005262d in void kurento::WebSocketTransport::processMessage<websocketpp::server<websocketpp::config::asio> >(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, websocketpp::server<websocketpp::config::asio>::message_ptr) (/kurento-asan/kurento-media-server+0x3b462d)
    #41 0x556aeff87da6 in void std::__invoke_impl<void, void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >(std::__invoke_memfun_deref, void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/invoke.h:73
    #42 0x556aeff87da6 in std::__invoke_result<void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >::type std::__invoke<void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >(void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/invoke.h:95
    #43 0x556aeff87da6 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::__call<void, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&, 0ul, 1ul, 2ul, 3ul>(std::tuple<std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /usr/include/c++/9/functional:400
    #44 0x556aeff87da6 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::operator()<std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >, void>(std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/functional:484
    #45 0x556aeff87da6 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/std_function.h:300
    #46 0x556af00e3219 in std::function<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::operator()(std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >) const /usr/include/c++/9/bits/std_function.h:688
    #47 0x556af00e3219 in websocketpp::connection<websocketpp::config::asio>::handle_read_frame(std::error_code const&, unsigned long) /usr/include/websocketpp/impl/connection_impl.hpp:1100
    #48 0x556aeff51b06 in void std::__invoke_impl<void, void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>(std::__invoke_memfun_deref, void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/invoke.h:73
    #49 0x556aeff51b06 in std::__invoke_result<void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>::type std::__invoke<void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>(void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/invoke.h:95
    #50 0x556aeff51b06 in void std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)>::__call<void, std::error_code const&, unsigned long&&, 0ul, 1ul, 2ul>(std::tuple<std::error_code const&, unsigned long&&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/9/functional:400
    #51 0x556aeff51b06 in void std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)>::operator()<std::error_code const&, unsigned long, void>(std::error_code const&, unsigned long&&) /usr/include/c++/9/functional:484
    #52 0x556aeff51b06 in std::_Function_handler<void (std::error_code const&, unsigned long), std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)> >::_M_invoke(std::_Any_data const&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/std_function.h:300
    #53 0x556af0015be2 in std::function<void (std::error_code const&, unsigned long)>::operator()(std::error_code const&, unsigned long) const /usr/include/c++/9/bits/std_function.h:688
    #54 0x556af0015be2 in websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_async_read(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long) /usr/include/websocketpp/transport/asio/connection.hpp:896
    #55 0x556af009ce80 in void std::__invoke_impl<void, void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>(std::__invoke_memfun_deref, void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&) /usr/include/c++/9/bits/invoke.h:73
    #56 0x556af009ce80 in std::__invoke_result<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>::type std::__invoke<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>(void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&) /usr/include/c++/9/bits/invoke.h:95
    #57 0x556af009ce80 in void std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)>::__call<void, boost::system::error_code&, unsigned long&, 0ul, 1ul, 2ul, 3ul>(std::tuple<boost::system::error_code&, unsigned long&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /usr/include/c++/9/functional:400
    #58 0x556af009ce80 in void std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)>::operator()<boost::system::error_code&, unsigned long&, void>(boost::system::error_code&, unsigned long&) /usr/include/c++/9/functional:484
    #59 0x556af009ce80 in void websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >::operator()<boost::system::error_code, unsigned long>(boost::system::error_code, unsigned long) /usr/include/websocketpp/transport/asio/base.hpp:106
    #60 0x556af009ce80 in boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>::operator()() /usr/include/boost/asio/detail/bind_handler.hpp:127
    #61 0x556af009ce80 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, ...) /usr/include/boost/asio/handler_invoke_hook.hpp:69
    #62 0x556af009ce80 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #63 0x556af009ce80 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>*) /usr/include/boost/asio/detail/bind_handler.hpp:170
    #64 0x556af009ce80 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #65 0x556af009ce80 in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&) /usr/include/boost/asio/detail/impl/strand_service.hpp:62
    #66 0x556af009e4bd in boost::asio::async_result<boost::asio::handler_type<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, void ()>::type>::type boost::asio::io_service::strand::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&&) /usr/include/boost/asio/strand.hpp:158
    #67 0x556af009e4bd in void boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>::operator()<boost::system::error_code, unsigned long>(boost::system::error_code const&, unsigned long const&) /usr/include/boost/asio/detail/wrapped_handler.hpp:99
    #68 0x556af009e4bd in boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >::operator()(boost::system::error_code const&, unsigned long, int) /usr/include/boost/asio/impl/read.hpp:282
    #69 0x556af0149eff in boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>::operator()() /usr/include/boost/asio/detail/bind_handler.hpp:127
    #70 0x556af0149eff in boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >::operator()() /usr/include/boost/asio/detail/wrapped_handler.hpp:191
    #71 0x556af0149eff in void boost::asio::asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, ...) /usr/include/boost/asio/handler_invoke_hook.hpp:69
    #72 0x556af0149eff in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #73 0x556af0149eff in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >*) /usr/include/boost/asio/detail/wrapped_handler.hpp:275
    #74 0x556af0149eff in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #75 0x556af0149eff in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/completion_handler.hpp:68
    #76 0x556af014b105 in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&) /usr/include/boost/asio/detail/impl/strand_service.hpp:88
    #77 0x556af014cb87 in boost::asio::async_result<boost::asio::handler_type<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, void ()>::type>::type boost::asio::io_service::strand::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&&) /usr/include/boost/asio/strand.hpp:158
    #78 0x556af014cb87 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>*) /usr/include/boost/asio/detail/wrapped_handler.hpp:232
    #79 0x556af014cb87 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #80 0x556af014cb87 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >*) /usr/include/boost/asio/impl/read.hpp:502
    #81 0x556af014cb87 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> > >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
    #82 0x556af014cb87 in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
    #83 0x556aeff6e5ed in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
    #84 0x556aeff6e5ed in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
    #85 0x556aeff6e5ed in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:641
    #86 0x556aeff6ad2e in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
    #87 0x556aeff6ad2e in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) /usr/include/boost/asio/detail/impl/task_io_service.ipp:372
    #88 0x556aeff6ad2e in boost::asio::detail::task_io_service::run(boost::system::error_code&) /usr/include/boost/asio/detail/impl/task_io_service.ipp:149
    #89 0x556aeff312bf in boost::asio::io_service::run() /usr/include/boost/asio/impl/io_service.ipp:59
    #90 0x556aeff312bf in kurento::WebSocketTransport::run() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:103
    #91 0x556aeff52273 in void std::__invoke_impl<void, void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>(std::__invoke_memfun_deref, void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&) /usr/include/c++/9/bits/invoke.h:73
    #92 0x556aeff52273 in std::__invoke_result<void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>::type std::__invoke<void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>(void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&) /usr/include/c++/9/bits/invoke.h:95
    #93 0x556aeff52273 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/9/functional:400
    #94 0x556aeff52273 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>::operator()<, void>() /usr/include/c++/9/functional:484
    #95 0x556aeff52273 in void std::__invoke_impl<void, std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>(std::__invoke_other, std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/invoke.h:60
    #96 0x556aeff52273 in std::__invoke_result<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>::type std::__invoke<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/invoke.h:95
    #97 0x556aeff52273 in void std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/9/thread:244
    #98 0x556aeff52273 in std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > >::operator()() /usr/include/c++/9/thread:251
    #99 0x556aeff52273 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > > >::_M_run() /usr/include/c++/9/thread:195
    #100 0x7fa5b0dcbb0f  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0b0f)
    #101 0x7fa5b2ec86b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #102 0x7fa5b08204dc in clone (/lib/x86_64-linux-gnu/libc.so.6+0x1074dc)

0x62c000010200 is located 0 bytes inside of 31392-byte region [0x62c000010200,0x62c000017ca0)
freed by thread T61 here:
    #0 0x7fa5b3d82c7f in __interceptor_free (/kurento-asan/libasan.so+0x10bc7f)
    #1 0x7fa54975d802 in hevc_decode_free libavcodec/hevc.c:3095

previously allocated by thread T132 (rtpjitterbuffer) here:
    #0 0x7fa5b3d83f3d in posix_memalign (/kurento-asan/libasan.so+0x10cf3d)
    #1 0x7fa54a303ddf in av_malloc libavutil/mem.c:97

Thread T61 created by T0 here:
    #0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fa5b0dcbda4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0da4)
    #2 0x556aeff48887 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
    #3 0x556aeff48887 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:376
    #4 0x556aefdb7c16 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
    #5 0x7fa5b073983f in __libc_start_main ../csu/libc-start.c:291

Thread T132 (rtpjitterbuffer) created by T129 (udpsrc0:src) here:
    #0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T129 (udpsrc0:src) created by T122 (task2) here:
    #0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T122 (task2) created by T57 here:
    #0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169

Thread T57 created by T0 here:
    #0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
    #1 0x7fa5b0dcbda4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0da4)
    #2 0x556aeff48887 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
    #3 0x556aeff48887 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:376
    #4 0x556aefdb7c16 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
    #5 0x7fa5b073983f in __libc_start_main ../csu/libc-start.c:291

SUMMARY: AddressSanitizer: double-free (/kurento-asan/libasan.so+0x10bc7f) in __interceptor_free
==13==ABORTING
++ on_error
++ echo '[Docker entrypoint] ERROR (1)'
++ exit 1
[Docker entrypoint] ERROR (1)

Let me know if you need any further information, I am happy to assist as this seems to be a critical issue for us since we use a lot of h265 cameras for our application.

[j1elo]

This seems to point to a different bug, in this case in the GStreamer H.265 decoder.

Note to the future me: the AddressSanitizer logs point to these locations:

AddressSanitizer: attempting double-free in thread T61:
    #1 0x7fa54975d80b in hevc_decode_free libavcodec/hevc.c:3096

freed by thread T61 here:
    #1 0x7fa54975d802 in hevc_decode_free libavcodec/hevc.c:3095

So these two lines in libavcodec: https://github.com/FFmpeg/FFmpeg/blob/n3.0.2/libavcodec/hevc.c#L3095-L3096

        for (i = 1; i < s->threads_number; i++) {
            HEVCLocalContext *lc = s->HEVClcList[i];
            if (lc) {
3095            av_freep(&s->HEVClcList[i]);
3096            av_freep(&s->sList[i]);
            }
        }

Which means that the memory pointed to by &s->HEVClcList[i] and the memory pointed by &s->sList[i] is actually the same, thus the double-free error.

This must be a bug in how GStreamer uses libavcodec, and is possibly fixed in versions of GStreamer which are newer than the one used in Kurento. If we find a commit which fixes this issue we might be able to backport it; however after reviewing the history of gst-libav I haven't been able to locate it.

Another possibility that should not be discarded is that this was a FFmpeg bug which got fixed at some point. If that's the case, of course there is nothing to be found in the gst-libav history.

[j1elo]

@sksamal are you able to get the crash if, instead of an RTSP stream (rtsp://...) you use a local file? (file:///path/to/file).

I have been trying with an H.265+AAC file, the same tutorial-player, and the same ASAN Docker image, but I wasn't able to make Kurento crash. So it would be interesting if you can test that too. Maybe record a small portion of your RTSP to a file (with some other program), then try to load it on the tutorial-player.

[sksamal]

@j1elo . I used a local H.265 video file with the tutorial player and ASAN docker image. Stream works fine and was able to play/stop a few times without any crash. Here are the logs for a single play/stop call:

16:34:12.254409840     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track audio, stream default
16:34:12.254455897     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.254493289     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track video, stream default
16:34:12.254503329     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.254528578     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track data, stream default
16:34:12.254537810     1 0x7f39b8008b00 DEBUG             kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.268929318     1 0x7f38d8002f20 FIXME               basesink gstbasesink.c:3125:gst_base_sink_default_event:<nicesink6> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
16:34:12.270338804     1 0x7f39d008d850 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<nicesrc6:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
16:34:12.345483727     1 0x7f39c80038d0 DEBUG         playerendpoint kmsplayerendpoint.c:935:kms_player_endpoint_started:<kmsplayerendpoint6> Pipeline started
16:34:12.345733273     1 0x7f39c80038d0 DEBUG           uridecodebin gsturidecodebin.c:2758:gst_uri_decode_bin_change_state: ready to paused
16:34:12.345754892     1 0x7f39c80038d0 DEBUG           uridecodebin gsturidecodebin.c:2204:setup_source:<uridecodebin6> setup source
16:34:12.345782988     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1316:gen_source_element:<uridecodebin6> finding source for file:///hevc-1080p.mp4
16:34:12.346147611     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1329:gen_source_element:<uridecodebin6> found source type GstFileSrc
16:34:12.346332624     1 0x7f39c80038d0 WARN                 basesrc gstbasesrc.c:3489:gst_base_src_start_complete:<source> pad not activated yet
16:34:12.346374569     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1339:gen_source_element:<uridecodebin6> source is stream: 0
16:34:12.346386705     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1342:gen_source_element:<uridecodebin6> source needs queue: 0
16:34:12.346446791     1 0x7f39c80038d0 DEBUG           uridecodebin gsturidecodebin.c:1462:has_all_raw_caps:<source:src> have caps ANY
16:34:12.346476025     1 0x7f39c80038d0 DEBUG           uridecodebin gsturidecodebin.c:2279:setup_source:<uridecodebin6> Plugging decodebin to source
16:34:12.346485121     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1798:make_decoder:<uridecodebin6> making new decodebin
16:34:12.346746566     1 0x7f39c80038d0 LOG             uridecodebin gsturidecodebin.c:1871:make_decoder:<uridecodebin6> have 1 pending dynamic objects
16:34:12.346975394     1 0x7f39c80038d0 WARN                 basesrc gstbasesrc.c:3489:gst_base_src_start_complete:<source> pad not activated yet
16:34:12.347997416     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.348049424     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.349996512     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38900098a0
16:34:12.350056973     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f3890009940
16:34:12.350082054     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.350140068     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.350164385     1 0x7f39cc0b3190 DEBUG           uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.355550193     1 0x7f36f8002630 WARN                 qtdemux qtdemux_types.c:222:qtdemux_type_get: unknown QuickTime node type pasp
16:34:12.355676092     1 0x7f36f8002630 WARN                 qtdemux qtdemux.c:2651:qtdemux_parse_trex:<qtdemux3> failed to find fragment defaults for stream 1
16:34:12.360398563     1 0x7f36f8002630 WARN                 basesrc gstbasesrc.c:2396:gst_base_src_update_length:<source> processing at or past EOS
16:34:12.360779690     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.360820332     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.361191717     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38a838dc80
16:34:12.361218600     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f38a838dc40
16:34:12.361236573     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.361484697     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.361506316     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.361687584     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.361924744     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f36f8001ec0
16:34:12.361957360     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f36f8001e00
16:34:12.361979948     1 0x7f36f8002630 DEBUG           uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.363023835     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363224184     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363376685     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363497323     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.363534455     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.363832528     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38a8388460
16:34:12.363864563     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x55a0d8e50420
16:34:12.363885801     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.363984855     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.364009733     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.653447723     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1158:new_decoded_pad_added_cb:<decodebin5> new decoded pad, name: <src_0>
16:34:12.653644881     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1142:copy_sticky_events:<'':src_0> store sticky event stream-start event: 0x7f37780035d0, time 99:99:99.999999999, seq-num 4253, GstEventStreamStart, stream-id=(string)adda3e4f412187fb2ea6a184d0a0530419be40cdb9cd31a78d0b9a9f1182f58d/001, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)30;
16:34:12.653706034     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1142:copy_sticky_events:<'':src_0> store sticky event caps event: 0x7f3888006520, time 99:99:99.999999999, seq-num 4345, GstEventCaps, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)30000/1001";
16:34:12.653766975     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:830:kms_player_endpoint_uridecodebin_pad_added:<uridecodebin6:src_0> Pad added
16:34:12.653795993     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:745:kms_player_end_point_get_agnostic_for_pad:<uridecodebin6:src_0> Prepare for input caps: video/x-raw, format=(string){ I420, YUY2, RGB, BGR, Y42B, Y444, YUV9, Y41B, GRAY8, RGB8P, I420, Y42B, Y444, UYVY, NV12, NV21, ARGB, RGBA, ABGR, BGRA, GRAY16_BE, GRAY16_LE, A420, RGB16, RGB15, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE }
16:34:12.653853055     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:753:kms_player_end_point_get_agnostic_for_pad:<uridecodebin6:src_0> Detected video caps
16:34:12.653864960     1 0x7f39a4004d40 DEBUG             kmselement kmselement.c:724:kms_element_get_output_element:<kmsplayerendpoint6> Output element requested for track video, stream default
16:34:12.654191540     1 0x7f39a4004d40 DEBUG             kmselement kmselement.c:694:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property codec-config
16:34:12.654222872     1 0x7f39a4004d40 DEBUG             kmselement kmselement.c:697:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property max-bitrate
16:34:12.654237615     1 0x7f39a4004d40 DEBUG             kmselement kmselement.c:700:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property min-bitrate
16:34:12.654881768     1 0x55a0d8e244f0 FIXME                default gstutils.c:3766:gst_pad_create_stream_id_internal:<appsrc3:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
16:34:12.654885542     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:964:no_more_pads_full:<decodebin5> no more pads, 1 pending
16:34:12.654936197     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:917:do_async_done:<uridecodebin6> posting ASYNC_DONE
16:34:12.655091578     1 0x7f39a4004d40 LOG             uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> stream-start, decoder 0x7f377803cf00
16:34:12.655131426     1 0x7f39a4004d40 LOG             uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> caps, decoder 0x7f377803cf00
16:34:12.655198081     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:781:appsink_probe_set_appsrc_caps:<appsrc3> Set new caps: video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30000/1001
16:34:12.658834585     1 0x7f39a4004d40 LOG             uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> segment, decoder 0x7f377803cf00
16:34:12.658883330     1 0x7f39a4004d40 LOG             uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> tag, decoder 0x7f377803cf00
16:34:12.658895268     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1113:decoded_pad_event_probe:<decodebin5:src_0> found bitrate 593435
16:34:12.658904562     1 0x7f39a4004d40 DEBUG           uridecodebin gsturidecodebin.c:1068:configure_stream_buffering:<uridecodebin6> overall bitrate 593435
16:34:12.658987789     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:500:process_sample:<appsink3> Preroll: reset base time
16:34:12.659006549     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:410:kms_player_endpoint_get_or_generate_base_time:<kmsplayerendpoint6> Setting base time to: 0:00:00.426031436, is preroll: 1
16:34:12.659222216     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:410:kms_player_endpoint_get_or_generate_base_time:<kmsplayerendpoint6> Setting base time to: 0:00:00.426246562, is preroll: 0
16:34:12.659238424     1 0x7f39a4004d40 DEBUG         playerendpoint kmsplayerendpoint.c:546:process_sample:<appsink3> Original PTS equals last PTS (original PTS: 0:00:00.000000000, is preroll: 0). Seems to be already pushed.
16:34:12.660181134     1 0x55a0d8e244f0 DEBUG            agnosticbin kmsagnosticbin.c:899:input_bin_src_caps_probe:<kmsagnosticbin2-3> Set input caps: video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30000/1001
16:34:13.872179650     1 0x7f39d008d850 DEBUG             kmselement kmselement.c:959:kms_element_set_sink_input_stats:<kmswebrtcendpoint6> Generating average stats for pad <kmswebrtcendpoint6:sink_audio_default>
16:34:13.872353495     1 0x7f39d008d850 DEBUG             kmselement kmselement.c:424:kms_element_set_target_on_linked:<kmsplayerendpoint6:video_src_default_0> Setting target <kmsagnosticbin2-3:src_0>
16:34:13.872387010     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: ANY
16:34:13.872418143     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.897032964     1 0x7f39d008d850 INFO             agnosticbin kmsagnosticbin.c:729:kms_agnostic_bin2_find_or_create_bin_for_caps:<kmsagnosticbin2-3> TRANSCODING ACTIVE for video
16:34:13.897890391     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: video/x-vp8, profile=(string){ 0, 1, 2, 3 }
16:34:13.897950692     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.898789509     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: video/x-vp8, profile=(string){ 0, 1, 2, 3 }
16:34:13.898850709     1 0x7f39d008d850 DEBUG            agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.899321100     1 0x7f39d008d850 DEBUG             kmselement kmselement.c:959:kms_element_set_sink_input_stats:<kmswebrtcendpoint6> Generating average stats for pad <kmswebrtcendpoint6:sink_video_default>
16:34:14.380407987     1 0x7f39cc006130 FIXME                    bin gstbin.c:4149:gst_bin_query: implement duration caching in GstBin again
16:34:32.631647791     1 0x7f39b8008b00 DEBUG         playerendpoint kmsplayerendpoint.c:919:kms_player_endpoint_stopped:<kmsplayerendpoint6> Pipeline stopped
16:34:32.655352475     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:1224:pad_removed_cb:<decodebin5> pad removed name: <decodebin5:src_0>
16:34:32.655422154     1 0x7f39b8008b00 DEBUG         playerendpoint kmsplayerendpoint.c:893:kms_player_endpoint_uridecodebin_pad_removed:<uridecodebin6:src_0> Pad removed
16:34:32.665196497     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:2784:gst_uri_decode_bin_change_state: paused to ready
16:34:32.665216535     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:1650:remove_decoders:<uridecodebin6> removing old decoder element
16:34:32.665250795     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:2086:remove_source:<uridecodebin6> removing old src element
16:34:32.665281595     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:2793:gst_uri_decode_bin_change_state: ready to null
16:34:32.665314702     1 0x7f39b8008b00 DEBUG           uridecodebin gsturidecodebin.c:1224:pad_removed_cb:<decodebin5> pad removed name: <decodebin5:sink>
16:34:32.667453857     1 0x7f39c804acf0 DEBUG             kmselement kmselement.c:853:accept_eos_probe:<kmswebrtcendpoint6:sink_video_default> Event flush-start dropped
16:34:32.667551548     1 0x7f39c804acf0 DEBUG             kmselement kmselement.c:853:accept_eos_probe:<kmswebrtcendpoint6:sink_video_default> Event flush-stop dropped
16:34:32.667577719     1 0x7f39c804acf0 DEBUG             kmselement kmselement.c:442:kms_element_remove_target_on_unlinked:<kmsplayerendpoint6:video_src_default_0> No target pad
16:34:32.667726323     1 0x7f39c804acf0 ERROR   KurentoUriEndpointImpl UriEndpointImpl.cpp:179:stop:<kmsplayerendpoint6> Error: Already in state stop
16:34:32.675010764     1 0x55a0d8e39730 DEBUG             kmselement kmselement.c:1369:kms_element_finalize:<kmswebrtcendpoint6> finalize
16:34:32.702201200     1 0x7f39c804acf0 DEBUG         playerendpoint kmsplayerendpoint.c:367:kms_player_endpoint_finalize:<kmsplayerendpoint6> finalize
16:34:32.702234918     1 0x7f39c804acf0 DEBUG             kmselement kmselement.c:1369:kms_element_finalize:<kmsplayerendpoint6> finalize

[j1elo]

So the same than me... it looks like the crash only happens with RTSP streams, then... not with just any stream.

[j1elo]

I just tested with an HTTP server and this also worked fine. Which rules out both file:// and http:// type URLs, only leaving rtsp:// as the way to cause the crash.

I've tried creating a quick and dirty localhost RTSP server, based on VLC...

... without transcoding, directly from the H.265 video:

VLC_RTP="sdp=rtsp://localhost:8554/video.sdp"

vlc -I dummy -vvv --repeat "video_H265.mp4" \
    :sout="#rtp{$VLC_RTP}"

... with transcoding, from some other source video:

VLC_TRANSCODE="vcodec=hevc,venc=x265{min-keyint=10,keyint=10,repeat-headers=1},scale=Auto"
VLC_RTP="sdp=rtsp://localhost:8554/video.sdp"

vlc -I dummy -vvv --repeat "video_VP8.webm" \
    :sout="#transcode{$VLC_TRANSCODE}:rtp{$VLC_RTP}"

but that didn't work, because of limitations in VLC. Turns out they implemented all codec parameters for x264 (keyint, repeat-headers), but lack those same parameters for x265. And repeat-headers is required, otherwise the stream metadata (H.265 SPS and PPS) won't be repeated and Kurento/GStreamer won't be able to decode it.

[sksamal]

Hi @j1elo, I found a gst-libav issue which seems to be related to this: https://gitlab.freedesktop.org/gstreamer/gst-libav/-/issues/72 The issue says "GStreamer 1.14.5 and FFmpeg 4.2.2 (Gentoo)"

Suggested solution is to increase the #define MAX_NB_THREADS 16 to 64 in libavcodec/hevcdec.h (in FFmpeg)

Just noticed, hevcdec.h doesnt exist in FFmpeg libav n3.0.2. But exists at hevc.h https://github.com/FFmpeg/FFmpeg/blob/n3.0.2/libavcodec/hevc.h#L42

[sksamal]

I was able to build https://github.com/Kurento/gst-libav after making the change, generate a new debian package and install it inside a kurento 6.14.0 container. It seems to fix the issue. Tested with a few h265 cameras, and the crash doesnot occur as yet.

[sksamal]

@j1elo , Any idea, if this will be fixed soon in next release ?

[gnc10]

Hi, facing the same issue and the fix mentioned by @sksamal is working for us. Is there any ETA on when this will be available in the next release? Thanks in advance!

[j1elo]

I just tested again with current pre-release version of Kurento 7.0 (experimental branch gstreamer-bionic in all Kurento git repos) and using VLC to simulate an RTSP server as shown in this comment. The PlayerEndpoint was able to run through the whole video, and Kurento didn't crash when the video finished. It also didn't crash when the Stop is sent (Stop button of kurento-player tutorial).

So I believe this issue is fixed in latest versions of GStreamer.

Note that the gstreamer-bionic git branches, and the Docker image kurento/kurento-media-server-exp:bionic-gstreamer, is built with Ubuntu 20.04 "Focal", and GStreamer 1.16.

This issue is categorized as a task for Kurento 7.0.0, because it requires updating GStreamer to solve the issue (something impossible in Kurento 6.x). As such, I think this issue can be closed because the current Kurento 7.0 preview doesn't show this problem any more.