search

Kurento / bugtracker

[ARCHIVED] Contents migrated to monorepo: https://github.com/Kurento/kurento
46 stars 10 forks source link

Kurento crash with error #427

Open nightwolftamb opened 4 years ago

nightwolftamb commented 4 years ago

KMS Version:

6.13.1~1.g5ef465a Build from sources (changed resolution to 1280x720 and FPS to 30 in compositor)

Ubuntu Version

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

Other libraries versions:

ii gir1.2-gst-plugins-base-1.5 1.8.1-1kurento1.18.04 amd64 Description: GObject introspection data for the GStreamer Plugins Base library ii gir1.2-gstreamer-1.5 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 Description: GObject introspection data for the GStreamer library ii gir1.2-nice-0.1:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (GObject introspection) ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.18.04 amd64 libav plugin for GStreamer ii gstreamer1.5-nice:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (GStreamer 1.5 plugin) ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer plugins from the "bad" set ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "base" set ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugins from the "good" set ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins from the "ugly" set ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento2.18.04 amd64 GStreamer plugin for PulseAudio ii gstreamer1.5-x:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer plugins for X11 and Pango ii kms-cmake-utils 6.12.1-0kurento1.18.04~20191018180342.gbp91cc3e all Kurento CMake utils ii kms-cmake-utils-build-deps 6.12.1-0kurento1 all build-dependencies for kms-cmake-utils ii kms-core 6.12.1-0kurento1.18.04~20191018180342.gbpb345e3 amd64 Kurento Core module ii kms-core-build-deps 6.12.1-0kurento1 all build-dependencies for kms-core ii kms-core-dev 6.12.1-0kurento1.18.04~20191018180342.gbpb345e3 amd64 Kurento Core module ii kms-elements 6.12.1-0kurento1.18.04~20191018180342.gbpf19f0b amd64 Kurento Elements module ii kms-elements-build-deps 6.12.1-0kurento1 all build-dependencies for kms-elements ii kms-elements-dev 6.12.1-0kurento1.18.04~20191018180342.gbpf19f0b amd64 Kurento Elements module ii kms-filters 6.12.1-0kurento1.18.04~20191018180342.gbpbbef4a amd64 Kurento Filters module ii kms-filters-build-deps 6.12.1-0kurento1 all build-dependencies for kms-filters ii kms-filters-dev 6.12.1-0kurento1.18.04~20191018180342.gbpbbef4a amd64 Kurento Filters module ii kms-jsonrpc 6.12.1-0kurento1.18.04~20191018180342.gbp5017f5 amd64 Kurento JSON-RPC library ii kms-jsonrpc-build-deps 6.12.1-0kurento1 all build-dependencies for kms-jsonrpc ii kms-jsonrpc-dev 6.12.1-0kurento1.18.04~20191018180342.gbp5017f5 amd64 Kurento JSON-RPC library ii kmsjsoncpp 1.6.3-1kurento1.18.04 amd64 Kurento jsoncpp library ii kmsjsoncpp-dev 1.6.3-1kurento1.18.04 amd64 Kurento jsonrpc library ii kurento-media-server-build-deps 6.12.1-0kurento1 all build-dependencies for kurento-media-server ii kurento-module-creator 6.12.1-0kurento1.18.04~20191018180342.gbpf3922e all Kurento Module Creator ii kurento-module-creator-build-deps 6.12.1-0kurento1 all build-dependencies for kurento-module-creator ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer development files for libraries from the "bad" set ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento1.18.04 amd64 GStreamer libraries from the "base" set ii libgstreamer-plugins-base1.5-dev 1.8.1-1kurento1.18.04 amd64 GStreamer development files for libraries from the "base" set ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 Core GStreamer libraries and elements ii libgstreamer1.5-dev 1.8.1-1kurento2.18.04~20191018180342.gbp84c84b amd64 GStreamer core development files ii libnice-dev 0.1.15-1kurento3.18.04 amd64 ICE library (development files) ii libnice10:amd64 0.1.15-1kurento3.18.04 amd64 ICE library (shared library) ii libsrtp0:amd64 1.6.0-0kurento1.18.04~20191018180342.gbp577e0a amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library ii libusrsctp 0.9.2-1kurento1.18.04 amd64 sctp-refimpl library ii openh264 1.4.0-1kurento1.18.04 amd64 OpenH264 library ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento4.18.04~20191018180342.gbp17d48d amd64 GStreamer plugins from openh264 ii openwebrtc-gst-plugins 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins ii openwebrtc-gst-plugins-dev 0.10.0-1kurento1.18.04 amd64 OpenWebRTC specific GStreamer plugins

Client libraries

Browsers tested

Not sure... It crashes one time per 2-3 weeks and there is lot of users. Will try to get more info later.

System description:

KMS located on phisical server. There is coturn server on same machine.

What steps will reproduce the problem?

Kurento just crashed with error: 2020-01-10T14:01:07,445121 88021 0x00007f7dc1ffb700 error agnosticbin kmsagnosticbin.c:397 proxy_src_pad_query_function() Caps not accepted: accept-caps query: 0x7f805c0b6720, GstQueryAcceptCaps, caps=(GstCaps)"audio/x-mulaw\,\ rate\=(int)48000\,\ channels\=(int)2", result=(boolean)false;

Here is whole log file: https://www.dropbox.com/s/vpxnwey28mnev04/2020-01-10T000038.00002.pid88021.log?dl=0

We have webinar application. User can create his own room and invite participants to it. All participants combined to one stream with Composite and send to RTMP service throw RTPEndpoint. For all endpoints we add this settings: rtp_output.setMinVideoRecvBandwidth(UserSession.MIN_VIDEO_RECV_BAND); rtp_output.setMinVideoSendBandwidth(UserSession.MIN_VIDEO_SEND_BAND); rtp_output.setMaxVideoRecvBandwidth(UserSession.MAX_VIDEO_RECV_BAND); rtp_output.setMaxVideoSendBandwidth(UserSession.MAX_VIDEO_SEND_BAND); rtp_output.setMaxAudioRecvBandwidth(UserSession.MAX_AUDIO_RECV_BAND); rtp_output.setAudioFormat(new AudioCaps(UserSession.AUDIO_CODEC, UserSession.MAX_AUDIO_RECV_BAND10241024));

public static int MIN_VIDEO_RECV_BAND = 500; public static int MIN_VIDEO_SEND_BAND = 500; public static int MAX_VIDEO_RECV_BAND = 4000; public static int MAX_VIDEO_SEND_BAND = 4000; public static int MAX_AUDIO_RECV_BAND = 500; public static AudioCodec AUDIO_CODEC = AudioCodec.PCMU;

We are using PCMU rather then OPUS because there is better quality while combining videos.

What is the expected result? Kurento not crashes

What happens instead? Kurento crashes

Does it happen with one of the tutorials? Not tested

Please provide any additional information below

nightwolftamb commented 4 years ago

Some more info (maybe will help). Sometimes Kurento continue works after this error, but some processes continue use CPU. Here is error: 2020-01-11T14:35:02,102339 42166 0x00007fa929e7a700 info agnosticbin kmsagnosticbin.c:774 kms_agnostic_bin2_link_pad() Upstream provided caps: audio/x-mulaw, rate=(int)[ 8000, 192000 ], channels=(int)[ 1, 2 ] 2020-01-11T14:35:02,102416 42166 0x00007fa929e7a700 info agnosticbin kmsagnosticbin.c:783 kms_agnostic_bin2_link_pad() Downstream wanted caps: audio/x-mulaw, channels=(int)1, rate=(int)8000 2020-01-11T14:35:02,102433 42166 0x00007fa7bcff9700 error agnosticbin kmsagnosticbin.c:397 proxy_src_pad_query_function() Caps not accepted: accept-caps query: 0x7fa798003990, GstQueryAcceptCaps, caps=(GstCaps)"audio/x-mulaw\,\ rate\=(int)48000\,\ channels\=(int)2", result=(boolean)false; 2020-01-11T14:35:02,104112 42166 0x00007fa7bffff700 fixme agnosticbin kmsagnosticbin.c:1359 check_ret_error() Handling flow error, fakesink message: chain *** (agnosticbin_fakesink505:sink) (1920 bytes, dts: none, pts: 0:01:00.430000000, duration: 0:00:00.010000000, offset: 2900640, offset_end: 2901120, flags: 00000800 gap ) 0x7fa7d0673b60 Full log here: https://www.dropbox.com/s/5y0hyhef7gu62fy/2020-01-11T000015.00001.pid42166.log?dl=0

Here is the "top -H" after this issue. https://www.dropbox.com/s/7jplpiwl4n868hs/screenshot.jpg?dl=0

Will continue to check this...

nightwolftamb commented 4 years ago

Will try check if this issue happens with OPUS audio codec....

nightwolftamb commented 4 years ago

Not depends from codec. Same issue with OPUS.

j1elo commented 4 years ago

Hi, please:

nightwolftamb commented 4 years ago

Can't install debug symbols. Get this error:

Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: kurento-dbg : Depends: libglib2.0-0-dbg but it is not installable Depends: kmsjsoncpp-dbg but it is not going to be installed Depends: libnice10-dbgsym but it is not going to be installed Depends: openwebrtc-gst-plugins-dbg but it is not going to be installed Depends: gstreamer1.5-libav-dbg but it is not going to be installed Depends: gstreamer1.5-nice-dbgsym but it is not going to be installed Depends: gstreamer1.5-plugins-ugly-dbg but it is not going to be installed E: Unable to correct problems, you have held broken packages.

Also can't locate errors.log. There is no such file in /var/logs. How can I set the location of this file?

nightwolftamb commented 4 years ago

Tried on another machine. Error: The following packages have unmet dependencies: kurento-dbg : Depends: libglib2.0-0-dbg but it is not installable E: Unable to correct problems, you have held broken packages.

Where to find "libglib2.0-0-dbg" for 18.04?

j1elo commented 4 years ago

Turns out the package dependencies for kurento-dbg are working for Ubuntu 16.04 but not for 18.04; I've fixed that thanks to you noticing the issue.

Updated instructions here: https://doc-kurento.readthedocs.io/en/latest/dev/dev_guide.html#dev-dbg

Note that until the fixes are published (in next Kurento release), the correct package name is libglib2.0-0-dbgsym.

This is the full list of packages to install:

libc6-dbg
libglib2.0-0-dbgsym
kmsjsoncpp-dbg
libnice10-dbgsym
openwebrtc-gst-plugins-dbg
libgstreamer1.5-0-dbg
gstreamer1.5-libav-dbg
gstreamer1.5-nice-dbgsym
gstreamer1.5-plugins-bad-dbg
gstreamer1.5-plugins-base-dbg
gstreamer1.5-plugins-good-dbg
gstreamer1.5-plugins-ugly-dbg
kms-jsonrpc-dbg
kms-core-dbg
kms-elements-dbg
kms-filters-dbg
kurento-media-server-dbg
nightwolftamb commented 4 years ago

Now the debug tools are installed but in this mode Kurento not listening port... I am run command "./bin/kms-build-run.sh --gdb" and after that "netstat - nltp | grep kurento" OR "netstat - nltp | grep 8888" - and the result is nothing. As the result - java application can't connect to server. But the process is running - i see it with command "ps -ef | grep kurento".

When I run command "./bin/kms-build-run.sh --release" - kurento starts listening on 8888 and 8433. Not sure what I am doing wrong...

j1elo commented 4 years ago

Oh I made a mistake in my original instructions, sorry for that. I've now fixed it! After starting with GDB, the gdb program starts in a wait state, until you introduce some commands. So, you need to write the command run to actually start running the process. All this is explained in the linked page about running GDB: https://doc-kurento.readthedocs.io/en/latest/user/troubleshooting.html#gstreamer-critical-messages-in-the-log

Please follow instructions there to get a crash backtrace from Kurento. That will be helpful!

nightwolftamb commented 4 years ago

I find the reason. I am streaming from webrtcendpoint to RTPEndpoint and after that stream goes to FFMPEG. Problem appeared only with this SDP offer: v=0 o=- 0 0 IN IP4 127.0.0.1 s=- c=IN IP4 127.0.0.1 t=0 0 m=audio " + current_audio_port + " RTP/AVP 96 b=AS:192 a=rtpmap:96 PCMU/8000 a=sendrecv a=ssrc:445566 cname:user@example.com m=video " + current_video_port + " RTP/AVP 103 a=rtpmap:103 H264/90000 a=sendrecv a=ssrc:112233 cname:user@example.com

Sometimes this cause kurento crash. But I am changed audio codec to opus/48000/2 and problem not happens again. Maybe this will help...