Kurento / bugtracker

[ARCHIVED] Contents migrated to monorepo: https://github.com/Kurento/kurento
46 stars 10 forks source link

Is this normal? Two process is running and Kurento Crashed #572

Open josephmiller2000 opened 3 years ago

josephmiller2000 commented 3 years ago

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

Issue description

last day on a odd time, kurento stopped working, cam broadcasting is not working, until i manually restarted kurento.

Context

How to reproduce?

Here is the screenshot of htop, which shows two process running with same kurento name, with different PID.

Snap_Shot_01612

Not sure how to reproduce the error, because the logs are kind of weird, because two log files contains nothing in it, just empty.

Snap_Shot_01617

Attaching another file log information in here

2021-01-14T232215.00073.pid13956.log

2021-01-14T23:22:15,489620 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3888, expected 3936)
2021-01-14T23:22:15,500467 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3889, expected 3936)
2021-01-14T23:22:15,510074 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3890, expected 3936)
2021-01-14T23:22:15,519435 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3892, expected 3936)
2021-01-14T23:22:15,530267 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3893, expected 3936)
2021-01-14T23:22:15,539379 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3895, expected 3936)
2021-01-14T23:22:15,546063 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 2 ms at PTS=0:17:11.820598553 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,548457 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 2 ms at PTS=0:17:11.823001270 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,550230 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3896, expected 3936)
2021-01-14T23:22:15,554650 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 3 ms at PTS=0:17:11.829193787 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,558439 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 3 ms at PTS=0:17:11.832983584 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,560347 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3897, expected 3936)
2021-01-14T23:22:15,566080 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 3 ms at PTS=0:17:11.840627189 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,569936 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 3 ms at PTS=0:17:11.844480996 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,569972 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3899, expected 3936)
2021-01-14T23:22:15,581036 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3900, expected 3936)
2021-01-14T23:22:15,602461 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3909, expected 3937)
2021-01-14T23:22:15,613332 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3910, expected 3937)
2021-01-14T23:22:15,625298 13956 0x00007f3755941700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26751_kmsagnosticbin2-7745:sink>  GAP of 66 ms at PTS=0:15:03.085070825 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,625707 13956 0x00007f3755941700 warning kmsutils                  kmsutils.c:478 discont_detection_probe() <kmswebrtcendpoint26751_kmsagnosticbin2-7745:sink>  DISCONTINUITY at non-keyframe; will drop until keyframe
2021-01-14T23:22:15,637962 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 2 ms at PTS=0:17:11.912498535 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,656957 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:502 gap_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  GAP of 7 ms at PTS=0:17:11.931464803 (packet loss?); will request a new keyframe
2021-01-14T23:22:15,657038 13956 0x00007f3777985700 warning kmsutils                  kmsutils.c:478 discont_detection_probe() <kmswebrtcendpoint26744_kmsagnosticbin2-7742:sink>  DISCONTINUITY at non-keyframe; will drop until keyframe
2021-01-14T23:22:15,762043 13956 0x00007f37e2a5b700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 17275, expected 17286)
2021-01-14T23:22:16,030726 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3937, expected 3985)
2021-01-14T23:22:16,069415 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3940, expected 3988)
2021-01-14T23:22:16,090610 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3944, expected 3989)
2021-01-14T23:22:16,104864 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3953, expected 3989)
2021-01-14T23:22:16,126814 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3955, expected 3990)
2021-01-14T23:22:16,146020 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3959, expected 3991)
2021-01-14T23:22:16,205134 13956 0x00007f37c5a21700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 3964, expected 3996)
2021-01-14T23:22:17,294718 13956 0x00007f37e2a5b700 warning rtpsource                 rtpsource.c:1147 update_receiver_stats()  duplicate or reordered packet (seqnr 17574, expected 17580)
2021-01-14T23:22:18,084695 13956 0x00007f3bd1b72700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:108 remove_not_supported_codecs_from_array() <kmswebrtcendpoint26784>  Removing not supported codec 'AMR/8000'
2021-01-14T23:22:18,084790 13956 0x00007f3bd1b72700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:566 WebRtcEndpointImpl()  STUN port not found in config; using default value: 3478
2021-01-14T23:22:18,084811 13956 0x00007f3bd1b72700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:574 WebRtcEndpointImpl()  STUN server not found in config; remember that NAT traversal requires STUN or TURN
2021-01-14T23:22:18,084828 13956 0x00007f3bd1b72700    info KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:592 WebRtcEndpointImpl()  Using TURN relay server: <user:password>@51.161.12.163:3478?transport=udp
2021-01-14T23:22:18,090658 13956 0x00007f37d9248700   fixme basesink                  gstbasesink.c:3125 gst_base_sink_default_event() <nicesink26784>  stream-start event without group-id. Consider implementing group-id handling in the upstream elements
2021-01-14T23:22:18,090965 13956 0x00007f37901b6700   fixme default                   gstutils.c:3766 gst_pad_create_stream_id_internal() <nicesrc26784:src>  Creating random stream-id, consider implementing a deterministic way of creating a stream-id
2021-01-14T23:22:18,091357 13956 0x00007f3bcf36d700    info basertpendpoint           kmsbasertpendpoint.c:1118 kms_base_rtp_endpoint_start_transport_send() <kmswe

Expected & current behavior

Kurento should be running without crashing, not sure what went wrong.

Info about your environment

Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-128-generic x86_64) Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz cores = 6 enabledcores = 6 threads = 12 System Memory = 32GiB

About Kurento Media Server

Kurento Media Server version: 6.15.0 Found modules: 'core' version 6.15.0 'elements' version 6.15.0 'filters' version 6.15.0

About your Application Server

Run these commands

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
Kurento Media Server version: 6.15.0
ii  gstreamer1.5-libav:amd64               1.8.1-1kurento1.18.04                           amd64        libav plugin for GStreamer
ii  gstreamer1.5-nice:amd64                0.1.18-0kurento1.18.04                          amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-plugins-bad:amd64         1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-base:amd64        1.8.1-1kurento2.18.04                           amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64        1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64        1.8.1-1kurento1.18.04                           amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-pulseaudio:amd64          1.8.1-1kurento5.18.04                           amd64        GStreamer plugin for PulseAudio
ii  kms-core                               6.15.0-0kurento1.18.04                          amd64        Kurento Core module
ii  kms-elements                           6.15.0-0kurento1.18.04                          amd64        Kurento Elements module
ii  kms-filters                            6.15.0-0kurento1.18.04                          amd64        Kurento Filters module
ii  kms-jsonrpc                            6.15.0-0kurento1.18.04                          amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                             1.6.3-1kurento1.18.04                           amd64        Kurento jsoncpp library
ii  kurento-media-server                   6.15.0-0kurento1.18.04                          amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64    1.8.1-1kurento5.18.04                           amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64   1.8.1-1kurento2.18.04                           amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64                1.8.1-1kurento2.18.04                           amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                        0.1.18-0kurento1.18.04                          amd64        ICE library (shared library)
ii  libsrtp0:amd64                         1.6.0-0kurento1.18.04                           amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                             0.9.2-1kurento1.18.04                           amd64        sctp-refimpl library
ii  openh264                               1.5.0-0kurento1.18.04                           amd64        H.264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugins-bad-1.5:amd64     1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins                 0.10.0-1kurento1.18.04                          amd64        OpenWebRTC specific GStreamer plugins
github-actions[bot] commented 3 years ago

Hello @josephmiller2000! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

j1elo commented 3 years ago

do you have any info in the errors.log file?

josephmiller2000 commented 3 years ago

Ok here is the error.log file details

2021-01-04T08:04:55+00:00 -- New execution
0:00:00.026027216  9077 0x561dc5f9d530 WARN                  ladspa gstladspa.c:455:plugin_init: no LADSPA plugins found, check LADSPA_PATH
0:00:00.088263179  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.088277899  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.088282197  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.088285782  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.088288959  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.088291937  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.088295381  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.088298954  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.088303212  9077 0x561dc5f9d530 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped

2021-01-04T15:07:45+00:00 -- New execution

2021-01-04T18:43:57+00:00 -- New execution

2021-01-04T18:56:54+00:00 -- New execution

2021-01-05T10:21:18+00:00 -- New execution

2021-01-05T11:18:43+00:00 -- New execution

2021-01-05T11:29:26+00:00 -- New execution

2021-01-05T11:32:11+00:00 -- New execution

2021-01-07T07:59:20+00:00 -- New execution
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!

2021-01-22T07:14:44+00:00 -- New execution
josephmiller2000 commented 3 years ago

Ok kurento crashed again, without any errors. But this time, i could not start kurento again, until i reboot the server.

Here is the screenshot image, before i stopped kurento, you can check how long it was running without crash.

Snap_Shot_01639

Here is the error i keep getting if i try to start the kurento, after todays crash.

Snap_Shot_01640

About the error.log file, here is the details.

2021-01-05T11:29:26+00:00 -- New execution

2021-01-05T11:32:11+00:00 -- New execution

2021-01-07T07:59:20+00:00 -- New execution
[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!

2021-01-22T07:14:44+00:00 -- New execution

2021-02-09T04:34:07+00:00 -- New execution
j1elo commented 3 years ago

Before trying to start KMS (sudo service kurento-media-server start), check that there are no other KMS instances running by some chance (Check your installation):

$ ps -fC kurento-media-server
UID        PID  PPID  C STIME TTY          TIME CMD

Now if it's possible, clean the logs directory (delete or move them somewhere else just during these tests):

sudo service kurento-media-server stop

sudo mv /var/log/kurento-media-server /var/log/kurento-media-server_BACKUP

sudo service kurento-media-server start

And paste here the tail of all log files (this is why I suggest moving out the old logs, so they don't show up here):

tail -n 20 /var/log/kurento-media-server/*.log
j1elo commented 3 years ago

About the error.log file, here is the details.

[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!

This message Error: the pic_width_in_mbs exceeds the level limits comes from the H.264 parser (OpenH264). It means that the encoder is sending a wrong H.264 stream.

Please provide more info about the H.264 video: where does it come from? Is it a web browser, an IP camera? Are you sending it to Kurento with a WebRtcEndpoint? RtpEndpoint? PlayerEndpoint?

If it is an IP camera or other kind of device under your control: how is the video configuration?

Access your device settings and make sure that H.264 is encoded with Constrained baseline profile and Level 3.1

josephmiller2000 commented 3 years ago

About the error.log file, here is the details.

[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!

This message _Error: the pic_width_inmbs exceeds the level limits comes from the H.264 parser (OpenH264). It means that the encoder is sending a wrong H.264 stream.

Please provide more info about the H.264 video: where does it come from? Is it a web browser, an IP camera? Are you sending it to Kurento with a WebRtcEndpoint? RtpEndpoint? PlayerEndpoint?

If it is an IP camera or other kind of device under your control: how is the video configuration?

Access your device settings and make sure that H.264 is encoded with Constrained baseline profile and Level 3.1

Thanks for the comment, but this is different issue i guess.

[OpenH264] this = 0x0x7f3b46394680, Error: the pic_width_in_mbs exceeds the level limits!

This error happened on, 2021-01-07T07:59:20+00:00

But the next two execution logs are clean without any errors.

Im using kurento for a public chat room, where users can broadcast their webcam video and rest of the people can see them.

It is a browser based application, and H.264 is sending through browser.