[0.x] janus.plugin.streaming: deadlock

[tmatth]

Summary

We've observed a deadlock in janus.plugin.streaming in production several times since at least February and possibly earlier, however it'd be catastrophic enough that we'd never had the chance to do analysis on it.

Thankfully, @lionelnicolas was able to cordon off the deadlocked janus instances so that we could examine them with GDB and thus we finally have some more detailed context. We're still missing some information, but in short:

• While holding the mountpoints_mutex as of here, janus_streaming_mountpoint_destroy (called from here) gets stuck waiting for the mountpoint->thread to join here: https://github.com/meetecho/janus-gateway/blob/4292ce5b33b1a2dd4ef1e653ea99f43857ab3fde/plugins/janus_streaming.c#L1309

• While holding the mountpoint->mutex as of here the mountpoint->thread never joins because it is stuck waiting to lock session->mutex for one of its viewers here: https://github.com/meetecho/janus-gateway/blob/4292ce5b33b1a2dd4ef1e653ea99f43857ab3fde/plugins/janus_streaming.c#L8417

• What we can't figure out is who is holding the lock for session->mutex in this context. The most likely candidate would be in the handling of the watch request, but from looking at the code paths it seems like wherever session->mutex is locked there is a path to it being unlocked.

What version of Janus is this happening on? 34e3f4abc87ac546d22aeea3dc1bb1af0959cfe7 (0.x)

Have you tested a more recent version of Janus too? No, as we've only hit this in production and are only using 0.x there.

Was this working before? No, we've been trying to track down this deadlock for quite some time so it's not a recent regression.

Is there a gdb or libasan trace of the issue? We have stacktraces for 2 different workloads which exhibit this deadlock (albeit with different impacts on the surrounding threads):

This workload was with janus.plugin.videoroom + janus.plugin.streaming

``` Thread 24 (Thread 0x7ffa16ffd700 (LWP 123)): #0 __pthread_clockjoin_ex (threadid=140712087631616, thread_return=0x0, clockid=, abstime=, block=) at pthread_join_common.c:145 #1 0x00007ffa6e04254b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6e01efaf in g_thread_join () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6477f53c in janus_streaming_mountpoint_destroy (mountpoint=0x7ffa0c110b10) at plugins/janus_streaming.c:1309 #4 0x00007ffa6dfe2229 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa647a5d1a in janus_streaming_process_synchronous_request (session=0x7ffa340063a0, message=0x7ffa1c013430) at plugins/janus_streaming.c:3859 #6 0x00007ffa647b18ae in janus_streaming_handle_message (handle=0x7ffa34032590, transaction=0x7ffa0c018000 "2qxqwaq275zf", message=0x7ffa1c013430, jsep=0x0) at plugins/janus_streaming.c:4364 #7 0x000055cec361460f in janus_process_incoming_request (request=0x7ffa1c04c2c0) at janus.c:1717 #8 0x000055cec3622edf in janus_transport_task (data=0x7ffa1c04c2c0, user_data=0x0) at janus.c:3432 #9 0x00007ffa6e01f374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #11 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #12 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 23 (Thread 0x7ffa15ffb700 (LWP 110)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa647e9220 in janus_streaming_relay_thread (data=0x7ffa0c110b10) at plugins/janus_streaming.c:8417 #3 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #5 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 22 (Thread 0x7ffa167fc700 (LWP 73)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x7ffa6001b0c0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff54a3 in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6dff54f1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 21 (Thread 0x7ffa177fe700 (LWP 71)): #0 0x00007ffa6d91546e in epoll_wait (epfd=22, events=0x7ffa177fcf60, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 #1 0x00007ffa647622fa in ?? () from /lib/x86_64-linux-gnu/libmicrohttpd.so.12 #2 0x00007ffa6476359f in ?? () from /lib/x86_64-linux-gnu/libmicrohttpd.so.12 #3 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #4 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 20 (Thread 0x7ffa17fff700 (LWP 70)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x55cec3a3fdf0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6c02751b in janus_http_timer (user_data=0x0) at transports/janus_http.c:380 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 19 (Thread 0x7ffa30ff9700 (LWP 69)): --Type for more, q to quit, c to continue without paging--c #0 0x00007ffa6d90899f in __GI___poll (fds=0x7ffa317fb010, nfds=4, timeout=13071) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6c0e9890 in poll (__timeout=, __nfds=, __fds=) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 _lws_plat_service_tsi (tsi=0, timeout_ms=, context=0x55cec3a3fe10) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:152 #3 _lws_plat_service_tsi (context=0x55cec3a3fe10, timeout_ms=, tsi=0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:72 #4 0x00007ffa6c105a75 in lws_service (context=0x55cec3a3fe10, timeout_ms=) at /tmp/libwebsockets/lib/core-net/service.c:838 #5 0x00007ffa6c052a82 in janus_websockets_thread (data=0x55cec3a3fe10) at transports/janus_websockets.c:1110 #6 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 18 (Thread 0x7ffa327fc700 (LWP 68)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e042623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dfc559b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6dfc5a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa647b6131 in janus_streaming_handler (data=0x0) at plugins/janus_streaming.c:4653 #5 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 17 (Thread 0x7ffa32ffd700 (LWP 67)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e042623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dfc559b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6dfc5a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa6c0a8522 in janus_videoroom_handler (data=0x0) at plugins/janus_videoroom.c:6247 #5 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 16 (Thread 0x7ffa337fe700 (LWP 66)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x55cec3a3a4f0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6c0cdcc0 in janus_videoroom_rtp_forwarder_rtcp_thread (data=0x0) at plugins/janus_videoroom.c:8664 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 15 (Thread 0x7ffa33fff700 (LWP 65)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e042623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dfc559b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6dfc5a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x000055cec35d1a3f in janus_events_thread (data=0x0) at events.c:283 #5 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 14 (Thread 0x7ffa4cff9700 (LWP 64)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e042623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dfc559b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007ffa6dfc5a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa6c141e83 in janus_wsevh_handler (data=0x0) at events/janus_wsevh.c:640 #5 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 13 (Thread 0x7ffa4d7fa700 (LWP 63)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x7ffa4dffc010, nfds=2, timeout=30000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6c0e9890 in poll (__timeout=, __nfds=, __fds=) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 _lws_plat_service_tsi (tsi=0, timeout_ms=, context=0x55cec3a37dc0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:152 #3 _lws_plat_service_tsi (context=0x55cec3a37dc0, timeout_ms=, tsi=0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:72 #4 0x00007ffa6c105a75 in lws_service (context=0x55cec3a37dc0, timeout_ms=) at /tmp/libwebsockets/lib/core-net/service.c:838 #5 0x00007ffa6c14180d in janus_wsevh_thread (data=0x0) at events/janus_wsevh.c:587 #6 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 12 (Thread 0x7ffa4effd700 (LWP 62)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6478c715 in janus_streaming_create_session (handle=0x7ffa34011da0, error=0x7ffa4effc244) at plugins/janus_streaming.c:2364 #3 0x000055cec35e0682 in janus_ice_handle_attach_plugin (core_session=0x7ffa34035a70, handle=0x7ffa340120f0, plugin=0x7ffa647f7e20 , loop_index=-1) at ice.c:1343 #4 0x000055cec360f7f9 in janus_process_incoming_request (request=0x7ffa1c006680) at janus.c:1218 #5 0x000055cec36234db in janus_transport_requests (data=0x0) at janus.c:3472 #6 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 11 (Thread 0x7ffa4f7fe700 (LWP 61)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x55cec3a2ce20, nfds=1, timeout=1998) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x000055cec360af80 in janus_sessions_watchdog (user_data=0x55cec3a2d7c0) at janus.c:737 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 10 (Thread 0x7ffa4ffff700 (LWP 60)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa647b4e63 in janus_streaming_hangup_media (handle=0x7ffa34008f50) at plugins/janus_streaming.c:4574 #3 0x000055cec3602b95 in janus_ice_outgoing_traffic_handle (handle=0x7ffa3405f290, pkt=0x55cec36a9f40 ) at ice.c:4702 #4 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa3405bb70, callback=0x0, user_data=0x0) at ice.c:474 #5 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a07f70) at ice.c:196 #9 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #11 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 9 (Thread 0x7ffa64ff9700 (LWP 59)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa647b547f in janus_streaming_hangup_media_internal (handle=0x7ffa1c02d220) at plugins/janus_streaming.c:4613 #3 0x00007ffa647b4eab in janus_streaming_hangup_media (handle=0x7ffa1c02d220) at plugins/janus_streaming.c:4575 #4 0x000055cec3602b95 in janus_ice_outgoing_traffic_handle (handle=0x7ffa34056eb0, pkt=0x55cec36a9f40 ) at ice.c:4702 #5 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa34062780, callback=0x0, user_data=0x0) at ice.c:474 #6 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #9 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a07c80) at ice.c:196 #10 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #11 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #12 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 8 (Thread 0x7ffa657fa700 (LWP 58)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x7ffa4801a660, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a04270) at ice.c:196 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 7 (Thread 0x7ffa65ffb700 (LWP 57)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6478c854 in janus_streaming_destroy_session (handle=0x7ffa34013630, error=0x7ffa65ffa7d0) at plugins/janus_streaming.c:2376 #3 0x000055cec3602f26 in janus_ice_outgoing_traffic_handle (handle=0x7ffa34057410, pkt=0x55cec36aa0a0 ) at ice.c:4734 #4 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa340606d0, callback=0x0, user_data=0x0) at ice.c:474 #5 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a03fb0) at ice.c:196 #9 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #11 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7ffa667fc700 (LWP 56)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6478c854 in janus_streaming_destroy_session (handle=0x7ffa34031f00, error=0x7ffa667fb7d0) at plugins/janus_streaming.c:2376 #3 0x000055cec3602f26 in janus_ice_outgoing_traffic_handle (handle=0x7ffa340113a0, pkt=0x55cec36aa0a0 ) at ice.c:4734 #4 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa340116b0, callback=0x0, user_data=0x0) at ice.c:474 #5 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a04b00) at ice.c:196 #9 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #11 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 5 (Thread 0x7ffa66ffd700 (LWP 55)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa647b4e63 in janus_streaming_hangup_media (handle=0x7ffa340794b0) at plugins/janus_streaming.c:4574 #3 0x000055cec3602b95 in janus_ice_outgoing_traffic_handle (handle=0x7ffa34024e80, pkt=0x55cec36a9f40 ) at ice.c:4702 #4 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa34025190, callback=0x0, user_data=0x0) at ice.c:474 #5 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a04840) at ice.c:196 #9 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #11 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 4 (Thread 0x7ffa677fe700 (LWP 54)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e041a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa647b4e63 in janus_streaming_hangup_media (handle=0x7ffa1c0322e0) at plugins/janus_streaming.c:4574 #3 0x000055cec3602b95 in janus_ice_outgoing_traffic_handle (handle=0x7ffa3409d600, pkt=0x55cec36a9f40 ) at ice.c:4702 #4 0x000055cec35d609d in janus_ice_outgoing_traffic_dispatch (source=0x7ffa34007380, callback=0x0, user_data=0x0) at ice.c:474 #5 0x00007ffa6dff517d in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffa6dff5400 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a03a20) at ice.c:196 #9 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #11 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 3 (Thread 0x7ffa67fff700 (LWP 53)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x7ffa601bb9b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x000055cec35d3868 in janus_ice_static_event_loop_thread (data=0x55cec3a03b50) at ice.c:196 #4 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 2 (Thread 0x7ffa6c949700 (LWP 52)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa6e042623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x000055cec3641531 in janus_log_thread (ctx=0x0) at log.c:106 #3 0x00007ffa6e01ead1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007ffa6d9f0609 in start_thread (arg=) at pthread_create.c:477 #5 0x00007ffa6d915133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 1 (Thread 0x7ffa6c951a80 (LWP 13)): #0 0x00007ffa6d90899f in __GI___poll (fds=0x55cec3a461e0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007ffa6dff536e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007ffa6dff56f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x000055cec36401a9 in main (argc=9, argv=0x7fff4b686958) at janus.c:5700 ```

This workload was with just the janus.plugin.streaming plugin

``` Thread 43 (Thread 0x7f34deffd700 (LWP 507)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c020770, message=0x7f3544024310) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c001960, transaction=0x7f35440248f0 "zlnuvkaw13tq", message=0x7f3544024310, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544024520) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544024520, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 42 (Thread 0x7f34df7fe700 (LWP 506)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c020770, message=0x7f354400a8e0) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c001960, transaction=0x7f3544024e40 "f4icox8m7n6i", message=0x7f354400a8e0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544023e10) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544023e10, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 41 (Thread 0x7f34dffff700 (LWP 503)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c020290, message=0x7f35440126b0) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016f40, transaction=0x7f3544012290 "xss8z6fkae47", message=0x7f35440126b0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544023cd0) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544023cd0, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 40 (Thread 0x7f34fcff9700 (LWP 501)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c020290, message=0x7f354401a980) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016f40, transaction=0x7f354401a5a0 "5aspv1ras6b6", message=0x7f354401a980, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544011ad0) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544011ad0, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 --Type for more, q to quit, c to continue without paging--c #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 39 (Thread 0x7f34fd7fa700 (LWP 228)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c019d30, message=0x7f354401a680) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c009750, transaction=0x7f354400d210 "i7ndvxk8i2qj", message=0x7f354401a680, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354400ce60) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354400ce60, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 38 (Thread 0x7f34fdffb700 (LWP 227)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c019d30, message=0x7f354401b800) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c009750, transaction=0x7f35440111b0 "ynsya5jxswxu", message=0x7f354401b800, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354401a500) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354401a500, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 37 (Thread 0x7f34fe7fc700 (LWP 226)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c018830, message=0x7f354401b8c0) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016780, transaction=0x7f354400dea0 "vr1ojhyhucz2", message=0x7f354401b8c0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354400db20) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354400db20, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 36 (Thread 0x7f34feffd700 (LWP 225)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c018830, message=0x7f354401bab0) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016780, transaction=0x7f354402a4f0 "7kyznzotyfw7", message=0x7f354401bab0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354401b740) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354401b740, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 35 (Thread 0x7f34ff7fe700 (LWP 224)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c0101b0, message=0x7f3544011520) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c010930, transaction=0x7f3544019a40 "f2veeejclfy3", message=0x7f3544011520, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354401ac20) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354401ac20, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 34 (Thread 0x7f34fffff700 (LWP 223)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c0101b0, message=0x7f3544009fa0) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c010930, transaction=0x7f354400a8a0 "vk79q4rap7bc", message=0x7f3544009fa0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544010eb0) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544010eb0, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 33 (Thread 0x7f351cff9700 (LWP 222)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c012bd0, message=0x7f3544019890) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016ba0, transaction=0x7f354400a590 "fjm96s0qsr4d", message=0x7f3544019890, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544018e00) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544018e00, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 32 (Thread 0x7f351d7fa700 (LWP 221)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c012bd0, message=0x7f354400f480) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c016ba0, transaction=0x7f355c0037a0 "zibur7f33ts2", message=0x7f354400f480, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544019710) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544019710, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 31 (Thread 0x7f351dffb700 (LWP 220)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c018070, message=0x7f354400f6d0) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c008a70, transaction=0x7f354402a700 "i1diglb71otr", message=0x7f354400f6d0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354402a720) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354402a720, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 30 (Thread 0x7f351e7fc700 (LWP 219)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c018070, message=0x7f3544030890) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c008a70, transaction=0x7f354400afb0 "ih73h77a83x3", message=0x7f3544030890, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354402a200) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354402a200, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 29 (Thread 0x7f351effd700 (LWP 218)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c0173b0, message=0x7f354400edf0) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c01a010, transaction=0x7f354400ece0 "1608lixk09r0", message=0x7f354400edf0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354400f100) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354400f100, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 28 (Thread 0x7f351f7fe700 (LWP 217)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c0173b0, message=0x7f3544008ca0) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c01a010, transaction=0x7f354400ae30 "61xnikyf6pbr", message=0x7f3544008ca0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544030680) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544030680, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 27 (Thread 0x7f351ffff700 (LWP 216)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c015030, message=0x7f354400ab40) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c012ba0, transaction=0x7f35440091d0 "8gnsgr0wp8d8", message=0x7f354400ab40, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544007090) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544007090, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 26 (Thread 0x7f3540ff9700 (LWP 215)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c015030, message=0x7f354400ffc0) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c012ba0, transaction=0x7f3544018870 "481k90dwy8dn", message=0x7f354400ffc0, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544007040) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544007040, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 25 (Thread 0x7f35417fa700 (LWP 214)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594553d5f in janus_streaming_process_synchronous_request (session=0x7f355c014b40, message=0x7f3544000e40) at plugins/janus_streaming.c:3838 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c0197f0, transaction=0x7f354401c100 "q68fykbk1e7i", message=0x7f3544000e40, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f3544000f20) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f3544000f20, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 24 (Thread 0x7f3541ffb700 (LWP 213)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35945428e5 in janus_streaming_process_synchronous_request (session=0x7f355c014b40, message=0x7f3544008500) at plugins/janus_streaming.c:2754 #3 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c0197f0, transaction=0x7f3544001970 "llck2do97cjq", message=0x7f3544008500, jsep=0x0) at plugins/janus_streaming.c:4364 #4 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354401bde0) at janus.c:1717 #5 0x00005579f7c89edf in janus_transport_task (data=0x7f354401bde0, user_data=0x0) at janus.c:3432 #6 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #8 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #9 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 23 (Thread 0x7f3543fff700 (LWP 212)): #0 __pthread_clockjoin_ex (threadid=139866742449920, thread_return=0x0, clockid=, abstime=, block=) at pthread_join_common.c:145 #1 0x00007f359651354b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964effaf in g_thread_join () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f359452e53c in janus_streaming_mountpoint_destroy (mountpoint=0x7f353801a280) at plugins/janus_streaming.c:1309 #4 0x00007f35964b3229 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3594554d1a in janus_streaming_process_synchronous_request (session=0x7f355c00aa80, message=0x7f3544012830) at plugins/janus_streaming.c:3859 #6 0x00007f35945608ae in janus_streaming_handle_message (handle=0x7f355c001f00, transaction=0x7f3544001aa0 "oz94e9dzbri8", message=0x7f3544012830, jsep=0x0) at plugins/janus_streaming.c:4364 #7 0x00005579f7c7b60f in janus_process_incoming_request (request=0x7f354400b830) at janus.c:1717 #8 0x00005579f7c89edf in janus_transport_task (data=0x7f354400b830, user_data=0x0) at janus.c:3432 #9 0x00007f35964f0374 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #10 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #11 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #12 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 22 (Thread 0x7f35437fe700 (LWP 210)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596512a5c in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f3594598220 in janus_streaming_relay_thread (data=0x7f353801a280) at plugins/janus_streaming.c:8417 #3 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #5 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 21 (Thread 0x7f35427fc700 (LWP 203)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f35427fb3c0, nfds=5, timeout=1000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35945926da in janus_streaming_relay_thread (data=0x7f3538003700) at plugins/janus_streaming.c:7928 #2 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #4 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 20 (Thread 0x7f3542ffd700 (LWP 73)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f35840190b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c64a3 in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f35964c64f1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 19 (Thread 0x7f3550ff9700 (LWP 69)): #0 0x00007f3595de646e in epoll_wait (epfd=21, events=0x7f3550ff7f60, maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 #1 0x00007f35944d62fa in ?? () from /lib/x86_64-linux-gnu/libmicrohttpd.so.12 #2 0x00007f35944d759f in ?? () from /lib/x86_64-linux-gnu/libmicrohttpd.so.12 #3 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #4 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 18 (Thread 0x7f35517fa700 (LWP 68)): #0 0x00007f3595dd999f in __GI___poll (fds=0x5579f87601b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f35944f151b in janus_http_timer (user_data=0x0) at transports/janus_http.c:380 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 17 (Thread 0x7f3551ffb700 (LWP 67)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f35527fd010, nfds=3, timeout=862) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35945ba890 in poll (__timeout=, __nfds=, __fds=) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 _lws_plat_service_tsi (tsi=0, timeout_ms=, context=0x5579f875a430) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:152 #3 _lws_plat_service_tsi (context=0x5579f875a430, timeout_ms=, tsi=0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:72 #4 0x00007f35945d6a75 in lws_service (context=0x5579f875a430, timeout_ms=) at /tmp/libwebsockets/lib/core-net/service.c:838 #5 0x00007f359451ca82 in janus_websockets_thread (data=0x5579f875a430) at transports/janus_websockets.c:1110 #6 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 16 (Thread 0x7f35537fe700 (LWP 66)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596513623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f359649659b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f3596496a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f3594565131 in janus_streaming_handler (data=0x0) at plugins/janus_streaming.c:4653 #5 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 15 (Thread 0x7f3553fff700 (LWP 65)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596513623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f359649659b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f3596496a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00005579f7c38a3f in janus_events_thread (data=0x0) at events.c:283 #5 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 14 (Thread 0x7f3574ff9700 (LWP 64)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596513623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f359649659b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f3596496a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f3594612e83 in janus_wsevh_handler (data=0x0) at events/janus_wsevh.c:640 #5 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 13 (Thread 0x7f35757fa700 (LWP 63)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f3575ffc010, nfds=2, timeout=5727) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35945ba890 in poll (__timeout=, __nfds=, __fds=) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 _lws_plat_service_tsi (tsi=0, timeout_ms=, context=0x5579f8761dc0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:152 #3 _lws_plat_service_tsi (context=0x5579f8761dc0, timeout_ms=, tsi=0) at /tmp/libwebsockets/lib/plat/unix/unix-service.c:72 #4 0x00007f35945d6a75 in lws_service (context=0x5579f8761dc0, timeout_ms=) at /tmp/libwebsockets/lib/core-net/service.c:838 #5 0x00007f359461280d in janus_wsevh_thread (data=0x0) at events/janus_wsevh.c:587 #6 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #7 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #8 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 12 (Thread 0x7f3576ffd700 (LWP 62)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596513623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f359649659b in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00007f3596496a1c in g_async_queue_pop () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00005579f7c8a11d in janus_transport_requests (data=0x0) at janus.c:3446 #5 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #7 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 11 (Thread 0x7f35777fe700 (LWP 61)): #0 0x00007f3595dd999f in __GI___poll (fds=0x5579f8756e20, nfds=1, timeout=1998) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c71f80 in janus_sessions_watchdog (user_data=0x5579f87577c0) at janus.c:737 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 10 (Thread 0x7f3577fff700 (LWP 60)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f356401a8d0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f8731f70) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 9 (Thread 0x7f358cff9700 (LWP 59)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f356c005db0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f8731c80) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 8 (Thread 0x7f358d7fa700 (LWP 58)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f35700ac380, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872e270) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 7 (Thread 0x7f358dffb700 (LWP 57)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f357c08f3b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872dfb0) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7f358e7fc700 (LWP 56)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f353018d9f0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872eb00) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 5 (Thread 0x7f358effd700 (LWP 55)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f3530130c90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872e840) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 4 (Thread 0x7f358f7fe700 (LWP 54)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f358002b810, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872da20) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 3 (Thread 0x7f358ffff700 (LWP 53)): #0 0x00007f3595dd999f in __GI___poll (fds=0x7f35880051b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7c3a868 in janus_ice_static_event_loop_thread (data=0x5579f872db50) at ice.c:196 #4 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #6 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 2 (Thread 0x7f3594e1a700 (LWP 52)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f3596513623 in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00005579f7ca8531 in janus_log_thread (ctx=0x0) at log.c:106 #3 0x00007f35964efad1 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #4 0x00007f3595ec1609 in start_thread (arg=) at pthread_create.c:477 #5 0x00007f3595de6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 1 (Thread 0x7f3594e22a80 (LWP 13)): #0 0x00007f3595dd999f in __GI___poll (fds=0x5579f87661e0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f35964c636e in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #2 0x00007f35964c66f3 in g_main_loop_run () from /lib/x86_64-linux-gnu/libglib-2.0.so.0 #3 0x00005579f7ca71a9 in main (argc=9, argv=0x7ffc80a926f8) at janus.c:5700 ```

Additional context

• We are using the WS transport and our mountpoints are all live RTP sources. We are not using SVC or helper threads.

• N.B. In some of the error cases, the order of mutexes being unlocked and the mountpoint reference count being decreased is not consistent, see here vs. here. It seems like the mountpoint's reference count should not be decreased while holding mp->mutex since janus_streaming_mountpoint_free will try to lock this mutex. This didn't show up in the backtraces so it seems unlikely to be related however, and also the reference count is incremented right before so it seems unlikely that it'd go to 0 here.

[tmatth]

This issue might be related to https://github.com/meetecho/janus-gateway/issues/3105

[atoppi]

looking into this issue (excellent analysis @tmatth)

What we can't figure out is who is holding the lock for session->mutex in this context. The most likely candidate would be in the handling of the watch request, but from looking at the code paths it seems like wherever session->mutex is locked there is a path to it being unlocked

There is no thread in stack trace holding that mutex, afaict that might be due to: 1) A code path where the mutex is not released (unlikely, should be troubleshooted through mutex lock debugging) 2) The session has been freed, and session->mutex is a garbage pointer (I guess libasan is not enabled here, it should notify a use after free)

[atoppi]

N.B. In some of the error cases, the order of mutexes being unlocked and the mountpoint reference count being decreased is not consistent, see here vs. here. It seems like the mountpoint's reference count should not be decreased while holding mp->mutex since janus_streaming_mountpoint_free will try to lock this mutex. This didn't show up in the backtraces so it seems unlikely to be related however, and also the reference count is incremented right before so it seems unlikely that it'd go to 0 here.

I agree, the order should be consistent and refcount decrement should happen after the mutex unlock.

[tmatth]

N.B. In some of the error cases, the order of mutexes being unlocked and the mountpoint reference count being decreased is not consistent, see here vs. here. It seems like the mountpoint's reference count should not be decreased while holding mp->mutex since janus_streaming_mountpoint_free will try to lock this mutex. This didn't show up in the backtraces so it seems unlikely to be related however, and also the reference count is incremented right before so it seems unlikely that it'd go to 0 here.

I agree, the order should be consistent and refcount decrement should happen after the mutex unlock.

I've addressed this specific issue here https://github.com/meetecho/janus-gateway/pull/3145

[atoppi]

Closing since #3246 has been merged.