Closed zchunhai closed 2 years ago
Hi, thanks for reporting the crash.
Could you please...
a) Try again with the official kurento-player tutorial, which can also play RTSP streams.
b) Try with this Docker image: kurento/kurento-media-server:6.13.1-asan
. It is a special build to analyze memory corruption bugs (like this one). Let me know if you get any message from AddressSanitizer
with this image.
Note that the very first message is a known issue:
=================================================================
==10==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 49152 byte(s) in 2 object(s) allocated from:
#0 0x7f1155a80f88 in malloc (/kurento-asan/libasan.so+0x10bf88)
#1 0x7f11553897b8 in g_malloc /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gmem.c:94
so that one is not important and you can ignore it.
@j1elo
I used the kurento-player tutorial
and kurento/kurento-media-server:6.13.1-asan
, but still can not play rtsp, the error is:
==10==LeakSanitizer has encountered a fatal error.
==10==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==10==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
10:01:22.608674559 9 0x61d0001f09e0 ERROR rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
10:01:22.609588574 9 0x61d0001f09e0 ERROR rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
10:01:24.610377050 9 0x61d0001f09e0 ERROR rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
=================================================================
==9==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200017623a at pc 0x7fcf79c259ed bp 0x7fcf3c4674b0 sp 0x7fcf3c466c58
WRITE of size 12 at 0x60200017623a thread T49 (rtpjitterbuffer)
10:01:24.613687657 9 0x61d0001f09e0 ERROR rtpjitterbuffer gstrtpjitterbuffer.c:1248:gst_jitter_buffer_sink_parse_caps:<rtpjitterbuffer0> Got caps with wrong payload type (got 96, expected 112)
#0 0x7fcf79c259ec (/kurento-asan/libasan.so+0x9a9ec)
#1 0x7fcf449bb356 in strcpy /usr/include/x86_64-linux-gnu/bits/string3.h:110
#2 0x7fcf449bb356 in gst_rtp_h265_depay_negotiate /workdir/gst/rtp/gstrtph265depay.c:216
#3 0x7fcf449bb356 in gst_rtp_h265_depay_setcaps /workdir/gst/rtp/gstrtph265depay.c:773
#4 0x7fcf74fad800 in gst_rtp_base_depayload_setcaps /workdir/gst-libs/gst/rtp/gstrtpbasedepayload.c:325
#5 0x7fcf74fad800 in gst_rtp_base_depayload_handle_event /workdir/gst-libs/gst/rtp/gstrtpbasedepayload.c:597
#6 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
#7 0x7fcf77c7bba8 in gst_pad_send_event /workdir/gst/gstpad.c:5752
#8 0x7fcf4a65ba27 in send_sticky_event /workdir/gst/playback/gstdecodebin2.c:1979
#9 0x7fcf77c6e8c8 in foreach_dispatch_function /workdir/gst/gstpad.c:5851
#10 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
#11 0x7fcf77c7c6f3 in gst_pad_sticky_events_foreach /workdir/gst/gstpad.c:5882
#12 0x7fcf4a658d8a in send_sticky_events /workdir/gst/playback/gstdecodebin2.c:1994
#13 0x7fcf4a658d8a in connect_pad /workdir/gst/playback/gstdecodebin2.c:2494
#14 0x7fcf4a664201 in analyze_new_pad /workdir/gst/playback/gstdecodebin2.c:1808
#15 0x7fcf4a664994 in type_found /workdir/gst/playback/gstdecodebin2.c:2861
#16 0x7fcf7542fe3f in ffi_call_unix64 (/usr/lib/x86_64-linux-gnu/libffi.so.6+0x5e3f)
#17 0x7fcf7542f8aa in ffi_call (/usr/lib/x86_64-linux-gnu/libffi.so.6+0x58aa)
#18 0x7fcf773117c8 in g_cclosure_marshal_generic /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:1490
#19 0x7fcf77310fa4 in g_closure_invoke /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:804
#20 0x7fcf77322fc0 in signal_emit_unlocked_R /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3629
#21 0x7fcf7732bd5b in g_signal_emit_valist /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3385
#22 0x7fcf7732c08e in g_signal_emit /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3441
#23 0x7fcf47656a2e in gst_type_find_element_setcaps /workdir/plugins/elements/gsttypefindelement.c:784
#24 0x7fcf47656a2e in gst_type_find_element_sink_event /workdir/plugins/elements/gsttypefindelement.c:676
#25 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
#26 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
#27 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
#28 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
#29 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
#30 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
#31 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
#32 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
#33 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
#34 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
#35 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
#36 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
#37 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
#38 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
#39 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
#40 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
#41 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
#42 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
#43 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
#44 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
#45 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
#46 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
#47 0x7fcf77c7b913 in check_sticky /workdir/gst/gstpad.c:3842
#48 0x7fcf77c7b913 in gst_pad_push_event /workdir/gst/gstpad.c:5371
#49 0x7fcf77c7ba8a in event_forward_func /workdir/gst/gstpad.c:2988
#50 0x7fcf77c7750d in gst_pad_forward /workdir/gst/gstpad.c:2942
#51 0x7fcf77c77662 in gst_pad_event_default /workdir/gst/gstpad.c:3039
#52 0x7fcf77c71396 in gst_pad_send_event_unchecked /workdir/gst/gstpad.c:5582
#53 0x7fcf77c717cd in gst_pad_push_event_unchecked /workdir/gst/gstpad.c:5240
#54 0x7fcf77c71c0f in push_sticky /workdir/gst/gstpad.c:3785
#55 0x7fcf77c6f8ee in events_foreach /workdir/gst/gstpad.c:601
#56 0x7fcf77c72010 in check_sticky /workdir/gst/gstpad.c:3842
#57 0x7fcf77c72010 in gst_pad_push_data /workdir/gst/gstpad.c:4413
#58 0x7fcf77c7a532 in gst_pad_push /workdir/gst/gstpad.c:4554
#59 0x7fcf4a1fe751 in gst_rtp_pt_demux_chain /workdir/gst/rtpmanager/gstrtpptdemux.c:442
#60 0x7fcf77c725ce in gst_pad_chain_data_unchecked /workdir/gst/gstpad.c:4183
#61 0x7fcf77c725ce in gst_pad_push_data /workdir/gst/gstpad.c:4435
#62 0x7fcf77c7a532 in gst_pad_push /workdir/gst/gstpad.c:4554
#63 0x7fcf4a1f89ef in pop_and_push_next /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3058
#64 0x7fcf4a1f9a19 in handle_next_buffer /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3157
#65 0x7fcf4a1f9a19 in gst_rtp_jitter_buffer_loop /workdir/gst/rtpmanager/gstrtpjitterbuffer.c:3593
#66 0x7fcf77ca4c5a in gst_task_func /workdir/gst/gsttask.c:344
#67 0x7fcf770615ed in g_thread_pool_thread_proxy /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthreadpool.c:307
#68 0x7fcf77060c54 in g_thread_proxy /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread.c:780
#69 0x7fcf78ddc6b9 in start_thread /build/glibc-LK5gWL/glibc-2.23/nptl/pthread_create.c:333
#70 0x7fcf7673241c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
0x60200017623a is located 0 bytes to the right of 10-byte region [0x602000176230,0x60200017623a)
allocated by thread T49 (rtpjitterbuffer) here:
#0 0x7fcf79c96f88 in malloc (/kurento-asan/libasan.so+0x10bf88)
#1 0x7fcf7703f7b8 in g_malloc /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gmem.c:94
Thread T49 (rtpjitterbuffer) created by T44 (udpsrc0:src) here:
#0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T44 (udpsrc0:src) created by T37 (task2) here:
#0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T37 (task2) created by T16 here:
#0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fcf7707e8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T16 created by T0 here:
#0 0x7fcf79bc4eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fcf76cddad4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0ad4)
#2 0x55906935f5a7 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
#3 0x55906935f5a7 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:324
#4 0x5590691d0b36 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
#5 0x7fcf7664b82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
SUMMARY: AddressSanitizer: heap-buffer-overflow (/kurento-asan/libasan.so+0x9a9ec)
Shadow bytes around the buggy address:
0x0c0480026bf0: fa fa fd fa fa fa 04 fa fa fa fd fa fa fa 04 fa
0x0c0480026c00: fa fa 00 fa fa fa 07 fa fa fa 00 00 fa fa fd fa
0x0c0480026c10: fa fa fd fa fa fa fd fa fa fa 05 fa fa fa fd fa
0x0c0480026c20: fa fa 05 fa fa fa fd fa fa fa fd fa fa fa 05 fa
0x0c0480026c30: fa fa fd fa fa fa fd fa fa fa fd fa fa fa 04 fa
=>0x0c0480026c40: fa fa fd fa fa fa 00[02]fa fa 00 06 fa fa fd fd
0x0c0480026c50: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fd
0x0c0480026c60: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fa fa
0x0c0480026c70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480026c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480026c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==9==ABORTING
Hi @zchunhai , the "-asan" image is just to detect memory corruption errors. And it worked, it detected one heap overflow error!
I see in the stack trace that all levels are inside GStreamer code. There is no Kurento code seen here... so it must be a GStreamer bug.
Can you provide a link to the RTSP stream, so I can test some fixes? (in a comment here or privately, email address is in my profile)
Hi @zchunhai , the "-asan" image is just to detect memory corruption errors. And it worked, it detected one heap overflow error!
I see in the stack trace that all levels are inside GStreamer code. There is no Kurento code seen here... so it must be a GStreamer bug.
Can you provide a link to the RTSP stream, so I can test some fixes? (in a comment here or privately, email address is in my profile)
I have emailed you the rtsp stream address.
Hi, thanks for sharing a test stream.
I was able to find out the GStreamer commit where they fixed the memory buffer overflow error, and backported it to Kurento.
I am now able to play the RTSP stream with kurento-player tutorial and the latest nightly build of KMS. I used these commands:
$ docker pull kurento/kurento-media-server-dev:latest
$ docker run --rm --net=host kurento/kurento-media-server-dev:latest
Let me know if it the nightly build works better than the 6.13
I am getting the exact same issue with h264 stream. I am using kms 6.14.0 version. Kurento media player crashes with similar logs.. I see that this issue was with v 6.13. Has this fix been merged into the mainstream ?
Where can i get a version where this issue is fixed?
Thanks so much Suraj
Please have a look at the latest version: https://doc-kurento.readthedocs.io/en/latest/user/installation.html Kurento 6.15 has received several improvements to GStreamer, so this issue is probably fixed.
Please, if you can verify that the crash doesn't happen, let us know so this issue report can be closed.
Thanks!
The problem exists with 6.15.0 as well and i can confim its not fixed. At first we thought the issue is with specific camera h265 stream, but I can confirm it occurs with all the h265 streams. The streaming starts perfectly, but when we send the stop request, the docker crashes. (I used the kurento-player tutorial - https://doc-kurento.readthedocs.io/en/latest/tutorials/java/tutorial-player.html) I used the docker kurento/kurento-media-server:6.15.0-asan
and generated these logs..
============kurento/kurento-media-server:6.15.0-asan================
0:07:26.598528751 13 0x619000411160 DEBUG playerendpoint kmsplayerendpoint.c:919:kms_player_endpoint_stopped:<kmsplayerendpoint0> Pipeline stopped
0:07:26.598848219 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd WAIT
0:07:26.598882621 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:5024:gst_rtspsrc_loop_send_cmd:<source> cancel previous request LOOP
0:07:26.598904544 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:5032:gst_rtspsrc_loop_send_cmd:<source> connection flush busy LOOP
0:07:26.598923168 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 1
0:07:26.598943409 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:4185:gst_rtspsrc_connection_flush:<source> connection flush
0:07:26.599051898 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4782:gst_rtspsrc_loop_udp:<source> got interrupted
0:07:26.599087021 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5075:gst_rtspsrc_loop:<source> pausing task, reason flushing
0:07:26.599101435 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd WAIT
0:07:26.599110284 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5032:gst_rtspsrc_loop_send_cmd:<source> connection flush busy LOOP
0:07:26.599117464 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 1
0:07:26.599255584 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:5012:gst_rtspsrc_loop_send_cmd:<source> sending cmd PAUSE
0:07:26.599305748 13 0x619000411160 DEBUG rtspsrc gstrtspsrc.c:5037:gst_rtspsrc_loop_send_cmd:<source> not interrupting busy cmd WAIT
0:07:26.599385561 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:7628:gst_rtspsrc_thread:<source> got command PAUSE
0:07:26.599450442 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 0
0:07:26.599535419 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4185:gst_rtspsrc_connection_flush:<source> connection flush
0:07:26.599578129 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:7424:gst_rtspsrc_pause:<source> PAUSE...
0:07:26.599921519 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5401:gst_rtspsrc_try_send:<source> sending message
0:07:26.601693429 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5430:gst_rtspsrc_try_send:<source> received response message
0:07:26.601764734 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:5445:gst_rtspsrc_try_send:<source> got response message 551
0:07:26.601819623 13 0x61d0001e54f0 WARN rtspsrc gstrtspsrc.c:5662:gst_rtspsrc_send:<source> error: Got error response: 551 (Option not supported).
0:07:26.602069437 13 0x61d0001e54f0 WARN rtspsrc gstrtspsrc.c:7526:gst_rtspsrc_pause:<source> error: Could not send message. (Generic error)
0:07:26.602106470 13 0x61d000103800 DEBUG playerendpoint kmsplayerendpoint.c:1166:kms_player_endpoint_emit_invalid_uri_signal: Emit 'Invalid URI' signal
0:07:26.602126541 13 0x6190000d44f0 ERROR playerendpoint kmsplayerendpoint.c:1335:process_bus_message:<kmsplayerendpoint0> Error code 9: 'Could not read from resource.', element: source, parent: internalpipeline
0:07:26.602202082 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:7628:gst_rtspsrc_thread:<source> got command LOOP
0:07:26.602252649 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4182:gst_rtspsrc_connection_flush:<source> set flushing 0
0:07:26.602246462 13 0x6190000d44f0 ERROR playerendpoint kmsplayerendpoint.c:1340:process_bus_message:<kmsplayerendpoint0> Debugging info: gstrtspsrc.c(5662): gst_rtspsrc_send (): /GstPipeline:internalpipeline/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source:
Got error response: 551 (Option not supported).
0:07:26.602273896 13 0x61d0001e54f0 DEBUG rtspsrc gstrtspsrc.c:4695:gst_rtspsrc_loop_udp:<source> doing receive with timeout 54 seconds
0:07:26.602331844 13 0x61d000103800 DEBUG playerendpoint kmsplayerendpoint.c:1166:kms_player_endpoint_emit_invalid_uri_signal: Emit 'Invalid URI' signal
0:07:26.602318316 13 0x6190000d44f0 ERROR playerendpoint kmsplayerendpoint.c:1335:process_bus_message:<kmsplayerendpoint0> Error code 10: 'Could not write to resource.', element: source, parent: internalpipeline
0:07:26.602388394 13 0x6190000d44f0 ERROR playerendpoint kmsplayerendpoint.c:1340:process_bus_message:<kmsplayerendpoint0> Debugging info: gstrtspsrc.c(7526): gst_rtspsrc_pause (): /GstPipeline:internalpipeline/GstURIDecodeBin:uridecodebin0/GstRTSPSrc:source:
Could not send message. (Generic error)
=================================================================
==13==ERROR: AddressSanitizer: attempting double-free on 0x62c000010200 in thread T61:
#0 0x7fa5b3d82c7f in __interceptor_free (/kurento-asan/libasan.so+0x10bc7f)
#1 0x7fa54975d80b in hevc_decode_free libavcodec/hevc.c:3096
#2 0x7fa549778157 in avcodec_close libavcodec/utils.c:2545
#3 0x7fa5497a2497 in gst_ffmpeg_avcodec_close /build/ext/libav/gstav.c:79
#4 0x7fa5497b23b2 in gst_ffmpegviddec_close /build/ext/libav/gstavviddec.c:320
#5 0x7fa5497b24ce in gst_ffmpegviddec_stop /build/ext/libav/gstavviddec.c:1712
#6 0x7fa5afb61646 in gst_video_decoder_change_state /build/gst-libs/gst/video/gstvideodecoder.c:2556
#7 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
#8 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
#9 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
#10 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
#11 0x7fa56eb2ee56 in gst_decode_bin_change_state /build/gst/playback/gstdecodebin2.c:5296
#12 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
#13 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
#14 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
#15 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
#16 0x7fa56eb39864 in gst_uri_decode_bin_change_state /build/gst/playback/gsturidecodebin.c:2752
#17 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
#18 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
#19 0x7fa5b1d2a514 in gst_bin_element_set_state /build/gst/gstbin.c:2414
#20 0x7fa5b1d2a514 in gst_bin_change_state_func /build/gst/gstbin.c:2756
#21 0x7fa5b1d4b24d in gst_element_change_state /build/gst/gstelement.c:2648
#22 0x7fa5b1d4b2a4 in gst_element_change_state /build/gst/gstelement.c:2694
#23 0x7fa5b1d4b9c6 in gst_element_set_state_func /build/gst/gstelement.c:2602
#24 0x7fa56f5a0b5c in kms_player_endpoint_mark_reset_base_time_and_set_state /kms-omni-build/kms-elements/src/gst-plugins/kmsplayerendpoint.c:444
#25 0x7fa56f5a0b5c in kms_player_endpoint_stopped /kms-omni-build/kms-elements/src/gst-plugins/kmsplayerendpoint.c:922
#26 0x7fa5afdd3544 in kms_uri_endpoint_set_state /build/src/gst-plugins/commons/kmsuriendpoint.c:152
#27 0x7fa5b29a84b0 in kurento::UriEndpointImpl::stop() /build/src/server/implementation/objects/UriEndpointImpl.cpp:177
#28 0x7fa5b2a1bcb8 in kurento::UriEndpointMethodStop::invoke(std::shared_ptr<kurento::UriEndpoint>) /build/obj-x86_64-linux-gnu/src/server/interface/generated-cpp/UriEndpointInternal.cpp:39
#29 0x7fa5b2a00d90 in kurento::UriEndpointImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&, Json::Value&) /build/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/UriEndpointImplInternal.cpp:37
#30 0x7fa592dc4789 in kurento::PlayerEndpointImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&, Json::Value&) /build/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/PlayerEndpointImplInternal.cpp:89
#31 0x556aefdfad3c in kurento::ServerMethods::invoke(Json::Value const&, Json::Value&) /kms-omni-build/kurento-media-server/server/ServerMethods.cpp:603
#32 0x556aefe1381c in void std::__invoke_impl<void, void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>(std::__invoke_memfun_deref, void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/invoke.h:73
#33 0x556aefe1381c in std::__invoke_result<void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>::type std::__invoke<void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&>(void (kurento::ServerMethods::*&)(Json::Value const&, Json::Value&), kurento::ServerMethods*&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/invoke.h:95
#34 0x556aefe1381c in void std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)>::__call<void, Json::Value const&, Json::Value&, 0ul, 1ul, 2ul>(std::tuple<Json::Value const&, Json::Value&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/9/functional:400
#35 0x556aefe1381c in void std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)>::operator()<Json::Value const&, Json::Value&, void>(Json::Value const&, Json::Value&) /usr/include/c++/9/functional:484
#36 0x556aefe1381c in std::_Function_handler<void (Json::Value const&, Json::Value&), std::_Bind<void (kurento::ServerMethods::*(kurento::ServerMethods*, std::_Placeholder<1>, std::_Placeholder<2>))(Json::Value const&, Json::Value&)> >::_M_invoke(std::_Any_data const&, Json::Value const&, Json::Value&) /usr/include/c++/9/bits/std_function.h:300
#37 0x7fa5b26ce7f9 in std::function<void (Json::Value const&, Json::Value&)>::operator()(Json::Value const&, Json::Value&) const /usr/include/c++/5/functional:2267
#38 0x7fa5b26ce7f9 in kurento::JsonRpc::Handler::process(Json::Value const&, Json::Value&) /build/src/JsonRpcHandler.cpp:122
#39 0x556aefe1075f in kurento::ServerMethods::process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /kms-omni-build/kurento-media-server/server/ServerMethods.cpp:277
#40 0x556af005262d in void kurento::WebSocketTransport::processMessage<websocketpp::server<websocketpp::config::asio> >(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, websocketpp::server<websocketpp::config::asio>::message_ptr) (/kurento-asan/kurento-media-server+0x3b462d)
#41 0x556aeff87da6 in void std::__invoke_impl<void, void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >(std::__invoke_memfun_deref, void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/invoke.h:73
#42 0x556aeff87da6 in std::__invoke_result<void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >::type std::__invoke<void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> > >(void (kurento::WebSocketTransport::*&)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), kurento::WebSocketTransport*&, websocketpp::server<websocketpp::config::asio>*&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/invoke.h:95
#43 0x556aeff87da6 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::__call<void, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&, 0ul, 1ul, 2ul, 3ul>(std::tuple<std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /usr/include/c++/9/functional:400
#44 0x556aeff87da6 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::operator()<std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >, void>(std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/functional:484
#45 0x556aeff87da6 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) /usr/include/c++/9/bits/std_function.h:300
#46 0x556af00e3219 in std::function<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)>::operator()(std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >) const /usr/include/c++/9/bits/std_function.h:688
#47 0x556af00e3219 in websocketpp::connection<websocketpp::config::asio>::handle_read_frame(std::error_code const&, unsigned long) /usr/include/websocketpp/impl/connection_impl.hpp:1100
#48 0x556aeff51b06 in void std::__invoke_impl<void, void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>(std::__invoke_memfun_deref, void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/invoke.h:73
#49 0x556aeff51b06 in std::__invoke_result<void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>::type std::__invoke<void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long>(void (websocketpp::connection<websocketpp::config::asio>::*&)(std::error_code const&, unsigned long), websocketpp::connection<websocketpp::config::asio>*&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/invoke.h:95
#50 0x556aeff51b06 in void std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)>::__call<void, std::error_code const&, unsigned long&&, 0ul, 1ul, 2ul>(std::tuple<std::error_code const&, unsigned long&&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/9/functional:400
#51 0x556aeff51b06 in void std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)>::operator()<std::error_code const&, unsigned long, void>(std::error_code const&, unsigned long&&) /usr/include/c++/9/functional:484
#52 0x556aeff51b06 in std::_Function_handler<void (std::error_code const&, unsigned long), std::_Bind<void (websocketpp::connection<websocketpp::config::asio>::*(websocketpp::connection<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>))(std::error_code const&, unsigned long)> >::_M_invoke(std::_Any_data const&, std::error_code const&, unsigned long&&) /usr/include/c++/9/bits/std_function.h:300
#53 0x556af0015be2 in std::function<void (std::error_code const&, unsigned long)>::operator()(std::error_code const&, unsigned long) const /usr/include/c++/9/bits/std_function.h:688
#54 0x556af0015be2 in websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_async_read(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long) /usr/include/websocketpp/transport/asio/connection.hpp:896
#55 0x556af009ce80 in void std::__invoke_impl<void, void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>(std::__invoke_memfun_deref, void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&) /usr/include/c++/9/bits/invoke.h:73
#56 0x556af009ce80 in std::__invoke_result<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>::type std::__invoke<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&>(void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*&)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long), std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >&, std::function<void (std::error_code const&, unsigned long)>&, boost::system::error_code&, unsigned long&) /usr/include/c++/9/bits/invoke.h:95
#57 0x556af009ce80 in void std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)>::__call<void, boost::system::error_code&, unsigned long&, 0ul, 1ul, 2ul, 3ul>(std::tuple<boost::system::error_code&, unsigned long&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /usr/include/c++/9/functional:400
#58 0x556af009ce80 in void std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)>::operator()<boost::system::error_code&, unsigned long&, void>(boost::system::error_code&, unsigned long&) /usr/include/c++/9/functional:484
#59 0x556af009ce80 in void websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >::operator()<boost::system::error_code, unsigned long>(boost::system::error_code, unsigned long) /usr/include/websocketpp/transport/asio/base.hpp:106
#60 0x556af009ce80 in boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>::operator()() /usr/include/boost/asio/detail/bind_handler.hpp:127
#61 0x556af009ce80 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, ...) /usr/include/boost/asio/handler_invoke_hook.hpp:69
#62 0x556af009ce80 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#63 0x556af009ce80 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>*) /usr/include/boost/asio/detail/bind_handler.hpp:170
#64 0x556af009ce80 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#65 0x556af009ce80 in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&) /usr/include/boost/asio/detail/impl/strand_service.hpp:62
#66 0x556af009e4bd in boost::asio::async_result<boost::asio::handler_type<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>, void ()>::type>::type boost::asio::io_service::strand::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::system::error_code, unsigned long>&&) /usr/include/boost/asio/strand.hpp:158
#67 0x556af009e4bd in void boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>::operator()<boost::system::error_code, unsigned long>(boost::system::error_code const&, unsigned long const&) /usr/include/boost/asio/detail/wrapped_handler.hpp:99
#68 0x556af009e4bd in boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >::operator()(boost::system::error_code const&, unsigned long, int) /usr/include/boost/asio/impl/read.hpp:282
#69 0x556af0149eff in boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>::operator()() /usr/include/boost/asio/detail/bind_handler.hpp:127
#70 0x556af0149eff in boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >::operator()() /usr/include/boost/asio/detail/wrapped_handler.hpp:191
#71 0x556af0149eff in void boost::asio::asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, ...) /usr/include/boost/asio/handler_invoke_hook.hpp:69
#72 0x556af0149eff in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#73 0x556af0149eff in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >*) /usr/include/boost/asio/detail/wrapped_handler.hpp:275
#74 0x556af0149eff in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#75 0x556af0149eff in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/completion_handler.hpp:68
#76 0x556af014b105 in void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&) /usr/include/boost/asio/detail/impl/strand_service.hpp:88
#77 0x556af014cb87 in boost::asio::async_result<boost::asio::handler_type<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >, void ()>::type>::type boost::asio::io_service::strand::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > > >(boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> > >&&) /usr/include/boost/asio/strand.hpp:158
#78 0x556af014cb87 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>*) /usr/include/boost/asio/detail/wrapped_handler.hpp:232
#79 0x556af014cb87 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running>&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#80 0x556af014cb87 in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >*) /usr/include/boost/asio/impl/read.hpp:502
#81 0x556af014cb87 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> > >(boost::asio::detail::binder2<boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>&, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> >&) /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#82 0x556af014cb87 in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_at_least_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*(std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>))(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> >, boost::asio::detail::is_continuation_if_running> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
#83 0x556aeff6e5ed in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
#84 0x556aeff6e5ed in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#85 0x556aeff6e5ed in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:641
#86 0x556aeff6ad2e in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
#87 0x556aeff6ad2e in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) /usr/include/boost/asio/detail/impl/task_io_service.ipp:372
#88 0x556aeff6ad2e in boost::asio::detail::task_io_service::run(boost::system::error_code&) /usr/include/boost/asio/detail/impl/task_io_service.ipp:149
#89 0x556aeff312bf in boost::asio::io_service::run() /usr/include/boost/asio/impl/io_service.ipp:59
#90 0x556aeff312bf in kurento::WebSocketTransport::run() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:103
#91 0x556aeff52273 in void std::__invoke_impl<void, void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>(std::__invoke_memfun_deref, void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&) /usr/include/c++/9/bits/invoke.h:73
#92 0x556aeff52273 in std::__invoke_result<void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>::type std::__invoke<void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&>(void (kurento::WebSocketTransport::*&)(), kurento::WebSocketTransport*&) /usr/include/c++/9/bits/invoke.h:95
#93 0x556aeff52273 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/9/functional:400
#94 0x556aeff52273 in void std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>::operator()<, void>() /usr/include/c++/9/functional:484
#95 0x556aeff52273 in void std::__invoke_impl<void, std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>(std::__invoke_other, std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/invoke.h:60
#96 0x556aeff52273 in std::__invoke_result<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>::type std::__invoke<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>>(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/invoke.h:95
#97 0x556aeff52273 in void std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/include/c++/9/thread:244
#98 0x556aeff52273 in std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > >::operator()() /usr/include/c++/9/thread:251
#99 0x556aeff52273 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> > > >::_M_run() /usr/include/c++/9/thread:195
#100 0x7fa5b0dcbb0f (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0b0f)
#101 0x7fa5b2ec86b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
#102 0x7fa5b08204dc in clone (/lib/x86_64-linux-gnu/libc.so.6+0x1074dc)
0x62c000010200 is located 0 bytes inside of 31392-byte region [0x62c000010200,0x62c000017ca0)
freed by thread T61 here:
#0 0x7fa5b3d82c7f in __interceptor_free (/kurento-asan/libasan.so+0x10bc7f)
#1 0x7fa54975d802 in hevc_decode_free libavcodec/hevc.c:3095
previously allocated by thread T132 (rtpjitterbuffer) here:
#0 0x7fa5b3d83f3d in posix_memalign (/kurento-asan/libasan.so+0x10cf3d)
#1 0x7fa54a303ddf in av_malloc libavutil/mem.c:97
Thread T61 created by T0 here:
#0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fa5b0dcbda4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0da4)
#2 0x556aeff48887 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
#3 0x556aeff48887 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:376
#4 0x556aefdb7c16 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
#5 0x7fa5b073983f in __libc_start_main ../csu/libc-start.c:291
Thread T132 (rtpjitterbuffer) created by T129 (udpsrc0:src) here:
#0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T129 (udpsrc0:src) created by T122 (task2) here:
#0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T122 (task2) created by T57 here:
#0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fa5b116c8ff in g_system_thread_new /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/gthread-posix.c:1169
Thread T57 created by T0 here:
#0 0x7fa5b3cb0eb1 in __interceptor_pthread_create (/kurento-asan/libasan.so+0x39eb1)
#1 0x7fa5b0dcbda4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xd0da4)
#2 0x556aeff48887 in void std::vector<std::thread, std::allocator<std::thread> >::emplace_back<std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()> >(std::_Bind<void (kurento::WebSocketTransport::*(kurento::WebSocketTransport*))()>&&) /usr/include/c++/9/bits/vector.tcc:121
#3 0x556aeff48887 in kurento::WebSocketTransport::start() /kms-omni-build/kurento-media-server/server/transport/websocket/WebSocketTransport.cpp:376
#4 0x556aefdb7c16 in main /kms-omni-build/kurento-media-server/server/main.cpp:257
#5 0x7fa5b073983f in __libc_start_main ../csu/libc-start.c:291
SUMMARY: AddressSanitizer: double-free (/kurento-asan/libasan.so+0x10bc7f) in __interceptor_free
==13==ABORTING
++ on_error
++ echo '[Docker entrypoint] ERROR (1)'
++ exit 1
[Docker entrypoint] ERROR (1)
Let me know if you need any further information, I am happy to assist as this seems to be a critical issue for us since we use a lot of h265 cameras for our application.
This seems to point to a different bug, in this case in the GStreamer H.265 decoder.
Note to the future me: the AddressSanitizer logs point to these locations:
AddressSanitizer: attempting double-free in thread T61:
#1 0x7fa54975d80b in hevc_decode_free libavcodec/hevc.c:3096
freed by thread T61 here:
#1 0x7fa54975d802 in hevc_decode_free libavcodec/hevc.c:3095
So these two lines in libavcodec: https://github.com/FFmpeg/FFmpeg/blob/n3.0.2/libavcodec/hevc.c#L3095-L3096
for (i = 1; i < s->threads_number; i++) {
HEVCLocalContext *lc = s->HEVClcList[i];
if (lc) {
3095 av_freep(&s->HEVClcList[i]);
3096 av_freep(&s->sList[i]);
}
}
Which means that the memory pointed to by &s->HEVClcList[i]
and the memory pointed by &s->sList[i]
is actually the same, thus the double-free error.
This must be a bug in how GStreamer uses libavcodec, and is possibly fixed in versions of GStreamer which are newer than the one used in Kurento. If we find a commit which fixes this issue we might be able to backport it; however after reviewing the history of gst-libav I haven't been able to locate it.
Another possibility that should not be discarded is that this was a FFmpeg bug which got fixed at some point. If that's the case, of course there is nothing to be found in the gst-libav history.
@sksamal are you able to get the crash if, instead of an RTSP stream (rtsp://...
) you use a local file? (file:///path/to/file
).
I have been trying with an H.265+AAC file, the same tutorial-player, and the same ASAN Docker image, but I wasn't able to make Kurento crash. So it would be interesting if you can test that too. Maybe record a small portion of your RTSP to a file (with some other program), then try to load it on the tutorial-player.
@j1elo . I used a local H.265 video file with the tutorial player and ASAN docker image. Stream works fine and was able to play/stop a few times without any crash. Here are the logs for a single play/stop call:
16:34:12.254409840 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track audio, stream default
16:34:12.254455897 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.254493289 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track video, stream default
16:34:12.254503329 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.254528578 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1422:kms_element_request_new_srcpad:<kmsplayerendpoint6> New output element for track data, stream default
16:34:12.254537810 1 0x7f39b8008b00 DEBUG kmselement kmselement.c:1471:kms_element_request_new_src_element_default:<kmsplayerendpoint6> src pads on sometimes (by default)
16:34:12.268929318 1 0x7f38d8002f20 FIXME basesink gstbasesink.c:3125:gst_base_sink_default_event:<nicesink6> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
16:34:12.270338804 1 0x7f39d008d850 FIXME default gstutils.c:3766:gst_pad_create_stream_id_internal:<nicesrc6:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
16:34:12.345483727 1 0x7f39c80038d0 DEBUG playerendpoint kmsplayerendpoint.c:935:kms_player_endpoint_started:<kmsplayerendpoint6> Pipeline started
16:34:12.345733273 1 0x7f39c80038d0 DEBUG uridecodebin gsturidecodebin.c:2758:gst_uri_decode_bin_change_state: ready to paused
16:34:12.345754892 1 0x7f39c80038d0 DEBUG uridecodebin gsturidecodebin.c:2204:setup_source:<uridecodebin6> setup source
16:34:12.345782988 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1316:gen_source_element:<uridecodebin6> finding source for file:///hevc-1080p.mp4
16:34:12.346147611 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1329:gen_source_element:<uridecodebin6> found source type GstFileSrc
16:34:12.346332624 1 0x7f39c80038d0 WARN basesrc gstbasesrc.c:3489:gst_base_src_start_complete:<source> pad not activated yet
16:34:12.346374569 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1339:gen_source_element:<uridecodebin6> source is stream: 0
16:34:12.346386705 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1342:gen_source_element:<uridecodebin6> source needs queue: 0
16:34:12.346446791 1 0x7f39c80038d0 DEBUG uridecodebin gsturidecodebin.c:1462:has_all_raw_caps:<source:src> have caps ANY
16:34:12.346476025 1 0x7f39c80038d0 DEBUG uridecodebin gsturidecodebin.c:2279:setup_source:<uridecodebin6> Plugging decodebin to source
16:34:12.346485121 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1798:make_decoder:<uridecodebin6> making new decodebin
16:34:12.346746566 1 0x7f39c80038d0 LOG uridecodebin gsturidecodebin.c:1871:make_decoder:<uridecodebin6> have 1 pending dynamic objects
16:34:12.346975394 1 0x7f39c80038d0 WARN basesrc gstbasesrc.c:3489:gst_base_src_start_complete:<source> pad not activated yet
16:34:12.347997416 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.348049424 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.349996512 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38900098a0
16:34:12.350056973 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f3890009940
16:34:12.350082054 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.350140068 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.350164385 1 0x7f39cc0b3190 DEBUG uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.355550193 1 0x7f36f8002630 WARN qtdemux qtdemux_types.c:222:qtdemux_type_get: unknown QuickTime node type pasp
16:34:12.355676092 1 0x7f36f8002630 WARN qtdemux qtdemux.c:2651:qtdemux_parse_trex:<qtdemux3> failed to find fragment defaults for stream 1
16:34:12.360398563 1 0x7f36f8002630 WARN basesrc gstbasesrc.c:2396:gst_base_src_update_length:<source> processing at or past EOS
16:34:12.360779690 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.360820332 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.361191717 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38a838dc80
16:34:12.361218600 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f38a838dc40
16:34:12.361236573 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.361484697 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.361506316 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.361687584 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.361924744 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f36f8001ec0
16:34:12.361957360 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x7f36f8001e00
16:34:12.361979948 1 0x7f36f8002630 DEBUG uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.363023835 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363224184 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363376685 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1771:proxy_autoplug_query_signal:<uridecodebin6> autoplug-query returned 0
16:34:12.363497323 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1711:proxy_autoplug_continue_signal:<uridecodebin6> autoplug-continue returned 1
16:34:12.363534455 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:337:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> finding factories
16:34:12.363832528 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:359:gst_uri_decode_bin_autoplug_factories:<uridecodebin6> autoplug-factories returns 0x7f38a8388460
16:34:12.363864563 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1726:proxy_autoplug_factories_signal:<uridecodebin6> autoplug-factories returned 0x55a0d8e50420
16:34:12.363885801 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1741:proxy_autoplug_sort_signal:<uridecodebin6> autoplug-sort returned (nil)
16:34:12.363984855 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:375:gst_uri_decode_bin_autoplug_select:<uridecodebin6> default autoplug-select returns TRY
16:34:12.364009733 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1756:proxy_autoplug_select_signal:<uridecodebin6> autoplug-select returned 0
16:34:12.653447723 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1158:new_decoded_pad_added_cb:<decodebin5> new decoded pad, name: <src_0>
16:34:12.653644881 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1142:copy_sticky_events:<'':src_0> store sticky event stream-start event: 0x7f37780035d0, time 99:99:99.999999999, seq-num 4253, GstEventStreamStart, stream-id=(string)adda3e4f412187fb2ea6a184d0a0530419be40cdb9cd31a78d0b9a9f1182f58d/001, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)30;
16:34:12.653706034 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1142:copy_sticky_events:<'':src_0> store sticky event caps event: 0x7f3888006520, time 99:99:99.999999999, seq-num 4345, GstEventCaps, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ chroma-site\=\(string\)mpeg2\,\ colorimetry\=\(string\)bt709\,\ framerate\=\(fraction\)30000/1001";
16:34:12.653766975 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:830:kms_player_endpoint_uridecodebin_pad_added:<uridecodebin6:src_0> Pad added
16:34:12.653795993 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:745:kms_player_end_point_get_agnostic_for_pad:<uridecodebin6:src_0> Prepare for input caps: video/x-raw, format=(string){ I420, YUY2, RGB, BGR, Y42B, Y444, YUV9, Y41B, GRAY8, RGB8P, I420, Y42B, Y444, UYVY, NV12, NV21, ARGB, RGBA, ABGR, BGRA, GRAY16_BE, GRAY16_LE, A420, RGB16, RGB15, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE }
16:34:12.653853055 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:753:kms_player_end_point_get_agnostic_for_pad:<uridecodebin6:src_0> Detected video caps
16:34:12.653864960 1 0x7f39a4004d40 DEBUG kmselement kmselement.c:724:kms_element_get_output_element:<kmsplayerendpoint6> Output element requested for track video, stream default
16:34:12.654191540 1 0x7f39a4004d40 DEBUG kmselement kmselement.c:694:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property codec-config
16:34:12.654222872 1 0x7f39a4004d40 DEBUG kmselement kmselement.c:697:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property max-bitrate
16:34:12.654237615 1 0x7f39a4004d40 DEBUG kmselement kmselement.c:700:kms_element_set_video_output_properties:<kmsagnosticbin2-3> Setting property min-bitrate
16:34:12.654881768 1 0x55a0d8e244f0 FIXME default gstutils.c:3766:gst_pad_create_stream_id_internal:<appsrc3:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
16:34:12.654885542 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:964:no_more_pads_full:<decodebin5> no more pads, 1 pending
16:34:12.654936197 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:917:do_async_done:<uridecodebin6> posting ASYNC_DONE
16:34:12.655091578 1 0x7f39a4004d40 LOG uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> stream-start, decoder 0x7f377803cf00
16:34:12.655131426 1 0x7f39a4004d40 LOG uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> caps, decoder 0x7f377803cf00
16:34:12.655198081 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:781:appsink_probe_set_appsrc_caps:<appsrc3> Set new caps: video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30000/1001
16:34:12.658834585 1 0x7f39a4004d40 LOG uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> segment, decoder 0x7f377803cf00
16:34:12.658883330 1 0x7f39a4004d40 LOG uridecodebin gsturidecodebin.c:1098:decoded_pad_event_probe:<decodebin5:src_0> tag, decoder 0x7f377803cf00
16:34:12.658895268 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1113:decoded_pad_event_probe:<decodebin5:src_0> found bitrate 593435
16:34:12.658904562 1 0x7f39a4004d40 DEBUG uridecodebin gsturidecodebin.c:1068:configure_stream_buffering:<uridecodebin6> overall bitrate 593435
16:34:12.658987789 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:500:process_sample:<appsink3> Preroll: reset base time
16:34:12.659006549 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:410:kms_player_endpoint_get_or_generate_base_time:<kmsplayerendpoint6> Setting base time to: 0:00:00.426031436, is preroll: 1
16:34:12.659222216 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:410:kms_player_endpoint_get_or_generate_base_time:<kmsplayerendpoint6> Setting base time to: 0:00:00.426246562, is preroll: 0
16:34:12.659238424 1 0x7f39a4004d40 DEBUG playerendpoint kmsplayerendpoint.c:546:process_sample:<appsink3> Original PTS equals last PTS (original PTS: 0:00:00.000000000, is preroll: 0). Seems to be already pushed.
16:34:12.660181134 1 0x55a0d8e244f0 DEBUG agnosticbin kmsagnosticbin.c:899:input_bin_src_caps_probe:<kmsagnosticbin2-3> Set input caps: video/x-raw, format=(string)I420, width=(int)1920, height=(int)1080, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, colorimetry=(string)bt709, framerate=(fraction)30000/1001
16:34:13.872179650 1 0x7f39d008d850 DEBUG kmselement kmselement.c:959:kms_element_set_sink_input_stats:<kmswebrtcendpoint6> Generating average stats for pad <kmswebrtcendpoint6:sink_audio_default>
16:34:13.872353495 1 0x7f39d008d850 DEBUG kmselement kmselement.c:424:kms_element_set_target_on_linked:<kmsplayerendpoint6:video_src_default_0> Setting target <kmsagnosticbin2-3:src_0>
16:34:13.872387010 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: ANY
16:34:13.872418143 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.897032964 1 0x7f39d008d850 INFO agnosticbin kmsagnosticbin.c:729:kms_agnostic_bin2_find_or_create_bin_for_caps:<kmsagnosticbin2-3> TRANSCODING ACTIVE for video
16:34:13.897890391 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: video/x-vp8, profile=(string){ 0, 1, 2, 3 }
16:34:13.897950692 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.898789509 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:774:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Upstream provided caps: video/x-vp8, profile=(string){ 0, 1, 2, 3 }
16:34:13.898850709 1 0x7f39d008d850 DEBUG agnosticbin kmsagnosticbin.c:783:kms_agnostic_bin2_link_pad:<kmsagnosticbin2-3> Downstream wanted caps: video/x-vp8
16:34:13.899321100 1 0x7f39d008d850 DEBUG kmselement kmselement.c:959:kms_element_set_sink_input_stats:<kmswebrtcendpoint6> Generating average stats for pad <kmswebrtcendpoint6:sink_video_default>
16:34:14.380407987 1 0x7f39cc006130 FIXME bin gstbin.c:4149:gst_bin_query: implement duration caching in GstBin again
16:34:32.631647791 1 0x7f39b8008b00 DEBUG playerendpoint kmsplayerendpoint.c:919:kms_player_endpoint_stopped:<kmsplayerendpoint6> Pipeline stopped
16:34:32.655352475 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:1224:pad_removed_cb:<decodebin5> pad removed name: <decodebin5:src_0>
16:34:32.655422154 1 0x7f39b8008b00 DEBUG playerendpoint kmsplayerendpoint.c:893:kms_player_endpoint_uridecodebin_pad_removed:<uridecodebin6:src_0> Pad removed
16:34:32.665196497 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:2784:gst_uri_decode_bin_change_state: paused to ready
16:34:32.665216535 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:1650:remove_decoders:<uridecodebin6> removing old decoder element
16:34:32.665250795 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:2086:remove_source:<uridecodebin6> removing old src element
16:34:32.665281595 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:2793:gst_uri_decode_bin_change_state: ready to null
16:34:32.665314702 1 0x7f39b8008b00 DEBUG uridecodebin gsturidecodebin.c:1224:pad_removed_cb:<decodebin5> pad removed name: <decodebin5:sink>
16:34:32.667453857 1 0x7f39c804acf0 DEBUG kmselement kmselement.c:853:accept_eos_probe:<kmswebrtcendpoint6:sink_video_default> Event flush-start dropped
16:34:32.667551548 1 0x7f39c804acf0 DEBUG kmselement kmselement.c:853:accept_eos_probe:<kmswebrtcendpoint6:sink_video_default> Event flush-stop dropped
16:34:32.667577719 1 0x7f39c804acf0 DEBUG kmselement kmselement.c:442:kms_element_remove_target_on_unlinked:<kmsplayerendpoint6:video_src_default_0> No target pad
16:34:32.667726323 1 0x7f39c804acf0 ERROR KurentoUriEndpointImpl UriEndpointImpl.cpp:179:stop:<kmsplayerendpoint6> Error: Already in state stop
16:34:32.675010764 1 0x55a0d8e39730 DEBUG kmselement kmselement.c:1369:kms_element_finalize:<kmswebrtcendpoint6> finalize
16:34:32.702201200 1 0x7f39c804acf0 DEBUG playerendpoint kmsplayerendpoint.c:367:kms_player_endpoint_finalize:<kmsplayerendpoint6> finalize
16:34:32.702234918 1 0x7f39c804acf0 DEBUG kmselement kmselement.c:1369:kms_element_finalize:<kmsplayerendpoint6> finalize
So the same than me... it looks like the crash only happens with RTSP streams, then... not with just any stream.
I just tested with an HTTP server and this also worked fine. Which rules out both file://
and http://
type URLs, only leaving rtsp://
as the way to cause the crash.
I've tried creating a quick and dirty localhost RTSP server, based on VLC...
... without transcoding, directly from the H.265 video:
VLC_RTP="sdp=rtsp://localhost:8554/video.sdp"
vlc -I dummy -vvv --repeat "video_H265.mp4" \
:sout="#rtp{$VLC_RTP}"
... with transcoding, from some other source video:
VLC_TRANSCODE="vcodec=hevc,venc=x265{min-keyint=10,keyint=10,repeat-headers=1},scale=Auto"
VLC_RTP="sdp=rtsp://localhost:8554/video.sdp"
vlc -I dummy -vvv --repeat "video_VP8.webm" \
:sout="#transcode{$VLC_TRANSCODE}:rtp{$VLC_RTP}"
but that didn't work, because of limitations in VLC. Turns out they implemented all codec parameters for x264 (keyint
, repeat-headers
), but lack those same parameters for x265. And repeat-headers
is required, otherwise the stream metadata (H.265 SPS and PPS) won't be repeated and Kurento/GStreamer won't be able to decode it.
Hi @j1elo, I found a gst-libav issue which seems to be related to this: https://gitlab.freedesktop.org/gstreamer/gst-libav/-/issues/72 The issue says "GStreamer 1.14.5 and FFmpeg 4.2.2 (Gentoo)"
Suggested solution is to increase the #define MAX_NB_THREADS 16 to 64 in libavcodec/hevcdec.h (in FFmpeg)
Just noticed, hevcdec.h doesnt exist in FFmpeg libav n3.0.2. But exists at hevc.h https://github.com/FFmpeg/FFmpeg/blob/n3.0.2/libavcodec/hevc.h#L42
I was able to build https://github.com/Kurento/gst-libav after making the change, generate a new debian package and install it inside a kurento 6.14.0 container. It seems to fix the issue. Tested with a few h265 cameras, and the crash doesnot occur as yet.
@j1elo , Any idea, if this will be fixed soon in next release ?
Hi, facing the same issue and the fix mentioned by @sksamal is working for us. Is there any ETA on when this will be available in the next release? Thanks in advance!
I just tested again with current pre-release version of Kurento 7.0 (experimental branch gstreamer-bionic
in all Kurento git repos) and using VLC to simulate an RTSP server as shown in this comment. The PlayerEndpoint was able to run through the whole video, and Kurento didn't crash when the video finished. It also didn't crash when the Stop is sent (Stop button of kurento-player tutorial).
So I believe this issue is fixed in latest versions of GStreamer.
Note that the gstreamer-bionic
git branches, and the Docker image kurento/kurento-media-server-exp:bionic-gstreamer
, is built with Ubuntu 20.04 "Focal", and GStreamer 1.16.
This issue is categorized as a task for Kurento 7.0.0, because it requires updating GStreamer to solve the issue (something impossible in Kurento 6.x). As such, I think this issue can be closed because the current Kurento 7.0 preview doesn't show this problem any more.
Prerequisites
[x] I have read the SUPPORT document
[x] I have checked the Troubleshooting Guide
[x] I have tested with the latest version of Kurento
Issue description
I am testing to play RTSP video stream. The test code is: https://github.com/lulop-k/kurento-rtsp2webrtc
The output source of the RTSP video stream is the NVR device of hikvision. The video stream address format is: rtsp://player:1234qwer@172.16.0.168:554/Streaming/tracks/201?transportmode=unicast&starttime=20200429t063818z&endtime=20200429t063828z
When the video ends playing, Kurento will crash:
Context
I am developing an application to play RTSP video stream through webRTC. When the video stream ends, it will inevitably lead to crash in kurento.
How to reproduce?
Reproduce steps:
https://github.com/lulop-k/kurento-rtsp2webrtc/blob/master/js/index.js
Expected & current behavior
kurento should not crash.
(Optional) Possible solution
Environment info
I use the docker image of kms, with tag 6.13.
docker run --name kms -d --net=host kurento/kurento-media-server:6.13
INFO: Kurento Media Server
INFO: Application Server
INFO: End-user clients
Version details