search

Hubs-Foundation / reticulum

Phoenix web server for Hubs
Mozilla Public License 2.0
449 stars 180 forks source link

Janus crash 10/19 #77

Open gfodor opened 5 years ago

gfodor commented 5 years ago

Load spike on Janus then crash, this was in the log:

Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): ASAN:DEADLYSIGNAL
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): =================================================================
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): ==9536==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000018 (pc 0x7f1458a2e831 bp 0x7f14416781b0 sp 0x7f1441678180 T122)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): ==9536==The signal is caused by a READ memory access.
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): ==9536==Hint: address points to the zero page.
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #0 0x7f1458a2e830 in g_socket_send_message (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libgio-2.0.so.0+0x9a830)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #1 0x7f145a5a668b in socket_send_message (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x3768b)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #2 0x7f145a5a68a1 in socket_send_messages (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x378a1)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #3 0x7f145a5a4273 in nice_socket_send_messages (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x35273)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #4 0x7f145a5a6f20 in socket_send_messages (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x37f20)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #5 0x7f145a5a4273 in nice_socket_send_messages (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x35273)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #6 0x7f145a587659 in nice_agent_send_messages_nonblocking_internal (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x18659)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #7 0x7f145a587ca7 in nice_agent_send (/hab/pkgs/mozillareality/libnice/0.1.15/20180914001451/lib/libnice.so.10+0x18ca7)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #8 0x43fd95 in janus_dtls_fd_bridge /hab/cache/src/meetecho/janus-gateway/dtls.c:993
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #9 0x43c03d in janus_dtls_srtp_send_alert /hab/cache/src/meetecho/janus-gateway/dtls.c:906
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #10 0x49dac6 in janus_ice_outgoing_traffic_handle /hab/cache/src/meetecho/janus-gateway/ice.c:3582
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #11 0x44de0c in janus_ice_outgoing_traffic_dispatch /hab/cache/src/meetecho/janus-gateway/ice.c:272
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #12 0x7f14573550ca  (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x570ca)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #13 0x7f1457355f4f in g_main_context_dispatch (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x57f4f)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #14 0x7f1457356133  (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x58133)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #15 0x7f1457356559 in g_main_loop_run (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x58559)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #16 0x486325 in janus_ice_thread /hab/cache/src/meetecho/janus-gateway/ice.c:2631
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #17 0x7f1457384dbc  (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x86dbc)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #18 0x7f1456b714d6  (/hab/pkgs/core/glibc/2.27/20180608041157/lib/libpthread.so.0+0x74d6)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #19 0x7f145555c4ee in __clone (/hab/pkgs/core/glibc/2.27/20180608041157/lib/libc.so.6+0xf24ee)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): AddressSanitizer can not provide additional info.
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): SUMMARY: AddressSanitizer: SEGV (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libgio-2.0.so.0+0x9a830) in g_socket_send_message
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): Thread T122 (iceloop 4975448) created by T9 here:
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #0 0x7f145ac03de0 in pthread_create (/hab/pkgs/core/gcc/7.3.0/20180608051919/lib/../lib/libasan.so.4+0x39de0)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #1 0x7f14573aae23  (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0xace23)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #2 0x7f1457384f14  (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x86f14)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #3 0x7f1457384ea4 in g_thread_try_new (/hab/pkgs/core/glib/2.50.3/20180718153537/lib/libglib-2.0.so.0+0x86ea4)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #4 0x495d18 in janus_ice_setup_local /hab/cache/src/meetecho/janus-gateway/ice.c:3196
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #5 0x4d63cf in janus_plugin_handle_sdp /hab/cache/src/meetecho/janus-gateway/janus.c:2835
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #6 0x4d0dc0 in janus_plugin_push_event /hab/cache/src/meetecho/janus-gateway/janus.c:2710
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #7 0x7f144e1bfefc in janus_plugin_sfu::push_response::h197f831be24b55d2 (/hab/pkgs/mozillareality/janus-gateway/0.4.3/20180914003418/lib/janus/plugins/li
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): Thread T9 created by T0 here:
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #0 0x7f145ac03de0 in pthread_create (/hab/pkgs/core/gcc/7.3.0/20180608051919/lib/../lib/libasan.so.4+0x39de0)
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):     #1 0x7f144e1c0f36 in janus_plugin_sfu::init::h2032dc67500f2ced (/hab/pkgs/mozillareality/janus-gateway/0.4.3/20180914003418/lib/janus/plugins/libjanus_pl
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O):
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(O): ==9536==ABORTING
Oct 19 12:58:31 dazzling-druid.reticulum.io bash[1392]: hab-launch(SV): Child for service 'janus-gateway.default@mozillareality' with PID 9533 exited with code exit code: 0
Oct 19 12:58:32 dazzling-druid.reticulum.io bash[1392]: janus-gateway.default@mozillareality(SV): Starting service as user=hab, group=hab
Oct 19 12:58:32 dazzling-druid.reticulum.io hab[15126]: Janus commit: 3c5ad1b125f696a32c85420f31a24eec79b35028
Oct 19 12:58:32 dazzling-druid.reticulum.io hab[15126]: Compiled on:  Fri Sep 14 00:37:21 UTC 2018
Oct 19 12:58:32 dazzling-druid.reticulum.io hab[15126]:
gfodor commented 5 years ago

cc @mquander

mqp commented 5 years ago

Glad ASan is working! I'll look into it.

timfuze commented 5 years ago

@mquander This is happening for us as well. I saw the related discussion in https://github.com/meetecho/janus-gateway/issues/1292. Were you able to get more info around this?