darrenhp
commented
3 years ago It looks the same as https://gitlab.freedesktop.org/libnice/libnice/-/issues/95 (fixed); But this issue show it's not fixed;
Open darrenhp opened 4 years ago
darrenhp
commented
3 years ago It looks the same as https://gitlab.freedesktop.org/libnice/libnice/-/issues/95 (fixed); But this issue show it's not fixed;
sshsun1990
commented
3 years ago I have the same problem. Now I modify the source code of libnice
// g_assert (pair->state == NICE_CHECK_DISCOVERED);
if (pair->state != NICE_CHECK_DISCOVERED) {
nice_debug ("pair->state:%d replace g_assert", pair->state);
return ;
}
j1elo
commented
3 years ago You can build with -DG_DISABLE_ASSERT so the GLib asserts are disabled on runtime. This is what we're doing in current builds of libnice: https://github.com/Kurento/libnice/blob/ubuntu/xenial/debian/rules#L18
The packages that are right now in the nightly repository are a Release Candidate of Kurento 6.15, so if you still have this issue you can try with this newer version and help us confirm if the issue is resolved:
https://doc-kurento.readthedocs.io/en/latest/user/installation_dev.html#kurento-media-server
darrenhp
commented
3 years ago You can build with
-DG_DISABLE_ASSERTso the GLib asserts are disabled on runtime. This is what we're doing in current builds of libnice: https://github.com/Kurento/libnice/blob/ubuntu/xenial/debian/rules#L18The packages that are right now in the nightly repository are a Release Candidate of Kurento 6.15, so if you still have this issue you can try with this newer version and help us confirm if the issue is resolved:
https://doc-kurento.readthedocs.io/en/latest/user/installation_dev.html#kurento-media-server
It happend again on Kurento 6.15;
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 140254237128448, pid 1250)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346
Below is my install infomation;
$ kurento-media-server -v
Kurento Media Server version: 6.15.0
Found modules:
'core' version 6.15.0
'elements' version 6.15.0
'filters' version 6.15.0
$ dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii gstreamer1.5-alsa:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugin for ALSA
ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer
ii gstreamer1.5-libav-dbg:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer (debug symbols)
ii gstreamer1.5-nice:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (GStreamer 1.5 plugin)
ii gstreamer1.5-nice-dbgsym:amd64 0.1.18-0kurento1.18.04 amd64 debug symbols for gstreamer1.5-nice
ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "bad" set
ii gstreamer1.5-plugins-bad-dbg:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "bad" set (debug symbols)
ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-base-dbg:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-good-dbg:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set
ii gstreamer1.5-plugins-ugly-dbg:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set (debug symbols)
ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugin for PulseAudio
ii gstreamer1.5-x:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins for X11 and Pango
ii kms-core 6.15.0-0kurento1.18.04 amd64 Kurento Core module
ii kms-core-dbg 6.15.0-0kurento1.18.04 amd64 Kurento Core module
ii kms-elements 6.15.0-0kurento1.18.04 amd64 Kurento Elements module
ii kms-elements-dbg 6.15.0-0kurento1.18.04 amd64 Kurento Elements module
ii kms-filters 6.15.0-0kurento1.18.04 amd64 Kurento Filters module
ii kms-filters-dbg 6.15.0-0kurento1.18.04 amd64 Kurento Filters module
ii kms-jsonrpc 6.15.0-0kurento1.18.04 amd64 Kurento JSON-RPC library
ii kms-jsonrpc-dbg 6.15.0-0kurento1.18.04 amd64 Kurento JSON-RPC library
ii kmsjsoncpp 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library
ii kmsjsoncpp-dbg 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library
ii kurento-dbg 6.15.0-0kurento1.18.04 amd64 Meta-package that installs debug symbols
ii kurento-media-server 6.15.0-0kurento1.18.04 amd64 Kurento Media Server
ii kurento-media-server-dbg 6.15.0-0kurento1.18.04 amd64 Kurento Media Server
ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer development files for libraries from the "bad" set
ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer libraries from the "base" set
ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.18.04 amd64 Core GStreamer libraries and elements
ii libgstreamer1.5-0-dbg:amd64 1.8.1-1kurento2.18.04 amd64 Core GStreamer libraries and elements
ii libnice10:amd64 0.1.18-0kurento1.18.04 amd64 ICE library (shared library)
ii libnice10-dbgsym:amd64 0.1.18-0kurento1.18.04 amd64 debug symbols for libnice10
ii libsrtp0:amd64 1.6.0-0kurento1.18.04 amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library
ii libusrsctp 0.9.2-1kurento1.18.04 amd64 sctp-refimpl library
ii openh264 1.5.0-0kurento1.18.04 amd64 H.264 Video Codec provided by Cisco Systems, Inc.
ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento5.18.04 amd64 GStreamer plugins from openh264
ii openwebrtc-gst-plugins 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins
ii openwebrtc-gst-plugins-dbg 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer pluginsOn one hand, the issue is that libnice is asserting the state "DISCOVERED", but the actual state if "FAILED". On the other, a recent change in libnice doesn't allow me to disable those assertions in libnice, thus making the whole media server to crash, instead of simply failing one connection.
Both issues have been reported so we can work with them and find a solution:
j1elo
commented
3 years ago Hi @darrenhp, the libnice dev has replied in our bug report and asked about a libnice debug log to confirm if their solution will actually solve the problem.
Are you able to repeat the issue? If so, please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of /etc/default/kurento-media-server:
export G_MESSAGES_DEBUG="libnice,libnice-stun"
export GST_DEBUG="$GST_DEBUG,glib:5"Hi @darrenhp, the libnice dev has replied in our bug report and asked about a libnice debug log to confirm if their solution will actually solve the problem.
Are you able to repeat the issue? If so, please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of
/etc/default/kurento-media-server:export G_MESSAGES_DEBUG="libnice,libnice-stun" export GST_DEBUG="$GST_DEBUG,glib:5"I have no idea how to reproduce it; But it happen several times every week on the production environment; I will enable the debug logs this week;
darrenhp
commented
3 years ago I hope it help. the kms progress started at 2020-11-11 and crashed at 2020-12-11 (and then it restarted); Crash log; 2020-12-11T000014.00030.pid1251.log
coredump ，https://www.dropbox.com/s/7avbn9wdpewfpb3/CoreDump.zip?dl=0
2020-11-11T02:02:03+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
**
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 140032120387328, pid 1251)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346
2020-12-11T17:21:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-distCrash log; 2020-12-11T000014.00030.pid1251.log
Hi @darrenhp, I'm afraid your logs don't contain any of the advanced libnice debug messages.
Please review my previous comment where I explained how to enable them:
please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of
/etc/default/kurento-media-server:export G_MESSAGES_DEBUG="libnice,libnice-stun" export GST_DEBUG="$GST_DEBUG,glib:5"
For completeness, this is a summary of previous messages up until now, with all you need to know in order to provide the requested logs:
Please use Kurento 6.15 (6.14 will not work to get libnice logs)
Add the requested lines at the end of /etc/default/kurento-media-server
Restart the media server so the changes are applied (for example with this command: sudo service kurento-media-server restart or equivalent in your system)
Check that logs in /var/log/kurento-media-server/ contain lines like this when doing WebRTC:
2020-12-14T10:56:29,973966 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : scheduling triggered check with socket=0x7f43f00044f0 and remote cand=0x7f443000bdc0.
2020-12-14T10:56:29,974029 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : Found a matching pair 0x7f43f0005380 (1:1493399139) (SUCCEEDED) ...
2020-12-14T10:56:29,974057 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : nothing to do for pair 0x7f43f0005380.
2020-12-14T10:56:29,974079 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : marking pair 0x7f43f0005380 (1:1493399139) as nominated
2020-12-14T10:56:29,974106 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040: Pruning pending checks for s1/c1
2020-12-14T10:56:29,974130 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : selected pair priority is 782000ff:7e7f1eff:1
2020-12-14T10:56:29,974153 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : conn.check list status: 1 nominated, 1 valid, c-id 1.
2020-12-14T10:56:29,974175 15847 0x00007f43fb7fe700 debug glib libnice:0 () agent_recv_message_unlocked: Valid STUN packet received.
2020-12-14T10:56:32,474354 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040: inbound STUN packet for 1/1 (stream/component) from [192.168.1.12]:58669 (100 octets) :
2020-12-14T10:56:32,474443 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () STUN demux: OK!
2020-12-14T10:56:32,474477 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () Comparing username/ufrag of len 9 and 4, equal=0
2020-12-14T10:56:32,474503 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () username: 0x726d53333a6e564e39
2020-12-14T10:56:32,474525 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () ufrag: 0x726d5333
2020-12-14T10:56:32,474546 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () Found valid username, returning password: 'SD25M6caIpbZTk83+3tkBl'
2020-12-14T10:56:32,474578 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () Message HMAC-SHA1 fingerprint:
2020-12-14T10:56:32,474600 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () key : 0x534432354d3663614970625a546b38332b33746b426c
2020-12-14T10:56:32,474620 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () expected: 0xf42fe67e80699ed53705529d8b3b10a7d4286f05
2020-12-14T10:56:32,474639 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () received: 0xf42fe67e80699ed53705529d8b3b10a7d4286f05
2020-12-14T10:56:32,474658 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () STUN auth: OK!
2020-12-14T10:56:32,474683 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () STUN unknown: 0 mandatory attribute(s)!
2020-12-14T10:56:32,474707 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () STUN Reply (buffer size = 1300)...
2020-12-14T10:56:32,474734 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () Message HMAC-SHA1 message integrity:
2020-12-14T10:56:32,474757 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () key : 0x534432354d3663614970625a546b38332b33746b426c
2020-12-14T10:56:32,474776 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () sent : 0x8da6620352677f9f5e396e65ae0d2240eb2441d4
2020-12-14T10:56:32,474807 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () Message HMAC-SHA1 fingerprint: 0xc22a17ca
2020-12-14T10:56:32,474829 15847 0x00007f43fb7fe700 debug glib libnice-stun:0 () All done (response size: 80)
2020-12-14T10:56:32,474856 15847 0x00007f43fb7fe700 debug glib libnice:0 () Agent 0x7f4420030040 : STUN-CC RESP to '192.168.1.12:58669', socket=23, len=80, cand=0x7f443000bdc0 (c-id:1), uWatch out with the size of logs, it will grow very fast and could take a lot of space. Kurento will by default do log rotation to keep all files under 100 MB, but still, keep an eye on this.
If a crash happens with the libnice debug logs enabled, we can send the logs to the libnice devs and help them find and fix the issue. Thanks for your help!
darrenhp
commented
3 years ago the last log file;
2021-02-24T191822.00314.pid434.log.tar.gz
2021-02-03T21:36:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
**
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 139781230229248, pid 434)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346
2021-02-24T19:23:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-distthe last log file; 2021-02-24T191822.00314.pid434.log.tar.gz
2021-02-03T21:36:02+08:00 -- New execution Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist ** libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6) Aborted (thread 139781230229248, pid 434) Stack trace: [__GI_raise] sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3) [__GI_abort] /build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81 [g_assertion_message] glib/gtestutils.c:2532 [g_assertion_message_cmpnum] glib/gtestutils.c:2590 [priv_mark_pair_nominated] agent/conncheck.c:2248 (discriminator 1) [priv_reply_to_conn_check] agent/conncheck.c:3248 [agent_recv_message_unlocked] agent/agent.c:4339 [component_io_cb] agent/agent.c:5662 [socket_source_dispatch] gio/gsocket.c:3843 [g_main_dispatch] glib/gmain.c:3180 [g_main_context_iterate] glib/gmain.c:3902 [g_main_loop_run] glib/gmain.c:4097 [gst_nice_src_create] gst/gstnicesrc.c:292 [gst_base_src_get_range] ./libs/gst/base/gstbasesrc.c:2460 [gst_base_src_loop] ./libs/gst/base/gstbasesrc.c:2737 [gst_task_func] ./gst/gsttask.c:346 2021-02-24T19:23:02+08:00 -- New execution Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
@j1elo Please check whether this log contain some useful message.
brunsy
commented
3 years ago @sshsun1990 sorry to ping you on such an old message. Do you remember how you got that change compiled and into Kurento?
Prerequisites
[x] I have read the SUPPORT document
[x] I have checked the Troubleshooting Guide
[x] I have tested with the latest version of Kurento
Issue description
crash
the errors.log is
the gdb
Context
How to reproduce?
Expected & current behavior
(Optional) Possible solution
INFO about Kurento Media Server
INFO about your Application Server
INFO about end-user clients
INFO about your environment
Commands output