Kurento / kurento

Kurento WebRTC Media Server
Apache License 2.0
305 stars 58 forks source link

set(Min/Max)EncoderBitrate not working in KMS 7.0.0 release #18

Closed neilyoung closed 1 year ago

neilyoung commented 1 year ago

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

Issue description

SetMinEncoderBitrate, SetMaxEncoderBitrate, SetEncoderBitrate not working as advertised

Details here: https://github.com/Kurento/kurento/discussions/17

Context

N/A

How to reproduce?

N/A

Expected & current behavior

Should have an impact

(Optional) Possible solution

None

Info about your environment

N/A

About Kurento Media Server

About your Application Server

About end-user clients

N/A

Run these commands

ubuntu@kms7:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
Kurento Media Server version: 7.0.0
Found modules:
    'core' version 7.0.0
    'elements' version 7.0.0
    'filters' version 7.0.0
    'webrtcarucocode' version 0.2.0~17.g4a42814
    'webrtcbarcode' version 0.2.0~8.g2dc7fb7
    'webrtccbr' version 0.2.0~9.ge481621
ubuntu@kms7:~$ dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gir1.2-nice-0.1:amd64                  0.1.18-0kurento1                  amd64        ICE library (GObject introspection)
ii  gstreamer1.0-nice:amd64                0.1.18-0kurento1                  amd64        ICE library (GStreamer plugin)
ii  gstreamer1.0-plugins-good:amd64        1:1.16.3-0ubuntu1.1kurento1       amd64        GStreamer plugins from the "good" set
ii  gstreamer1.0-plugins-good-kurento      1.0.0-0kurento1ubuntu1            amd64        Dummy package that depends on Kurento patched gstreamer1.0-plugins-good (shared library)
ii  kurento-cmake-utils                    7.0.0-1kurento1                   all          CMake utilities used when building Kurento modules.
ii  kurento-jsonrpc                        7.0.0-1kurento1                   amd64        Kurento JSON-RPC library
ii  kurento-jsonrpc-dev                    7.0.0-1kurento1                   amd64        Kurento JSON-RPC library
ii  kurento-media-server                   7.0.0-1kurento1                   amd64        Kurento Media Server
ii  kurento-media-server-dev               7.0.0-1kurento1                   amd64        Kurento Media Server
ii  kurento-module-core                    7.0.0-1kurento1                   amd64        Core module for use by Kurento Media Server
ii  kurento-module-core-dev                7.0.0-1kurento1                   amd64        Core module for use by Kurento Media Server
ii  kurento-module-creator                 7.0.0-1kurento1                   all          Kurento Module Creator
ii  kurento-module-elements                7.0.0-1kurento1                   amd64        Kurento Elements module
ii  kurento-module-elements-dev            7.0.0-1kurento1                   amd64        Kurento Elements module
ii  kurento-module-filters                 7.0.0-1kurento1                   amd64        Kurento Filters module
ii  kurento-module-filters-dev             7.0.0-1kurento1                   amd64        Kurento Filters module
ii  libgstreamer-plugins-good1.0-0:amd64   1:1.16.3-0ubuntu1.1kurento1       amd64        GStreamer development files for libraries from the "good" set
ii  libgstreamer-plugins-good1.0-dev       1:1.16.3-0ubuntu1.1kurento1       amd64        GStreamer development files for libraries from the "good" set
ii  libnice-dev                            0.1.18-0kurento1                  amd64        ICE library (development files)
ii  libnice10:amd64                        0.1.18-0kurento1                  amd64        ICE library (shared library)
ii  libsrtp-kurento                        1.0.0-0kurento1ubuntu1            amd64        Dummy package that depends on Kurento patched libsrtp (shared library)
ii  libsrtp2-1:amd64                       1:2.3.0-2kurento1                 amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  openh264                               2.3.0-0kurento1                   amd64        OpenH264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugin                    1.0.0-0kurento1ubuntu1            amd64        OpenH264 plugin for GStreamer
j1elo commented 1 year ago

Please run Kurento with this logging level enabled: enctreebin:5,agnosticbin:5. For example:

export GST_DEBUG="2,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,enctreebin:5,agnosticbin:5"

Then share the relevant part of the logs here. We're especially interested in messages from enctreebin and agnosticbin.

If you are using a Kurento built from sources, and are able to make a temporary change in the source code, please also do this:

static gint
kms_enc_tree_bin_get_bitrate (KmsEncTreeBin * self)
{
  gint bitrate;

  if (self->priv->remb_bitrate <= 0) {
    if (self->priv->tag_bitrate <= 0) {
      GST_WARNING_OBJECT(self, "*** Bitrate from min_bitrate");
    } else {
      GST_WARNING_OBJECT(self, "*** Bitrate from TAG");
    }
    bitrate = KMS_ENC_TREE_BIN_LIMIT (self, self->priv->tag_bitrate);
  } else if (self->priv->tag_bitrate <= 0) {
    GST_WARNING_OBJECT(self, "*** Bitrate from REMB");
    bitrate = KMS_ENC_TREE_BIN_LIMIT (self, self->priv->remb_bitrate);
  } else {
    GST_WARNING_OBJECT(self, "*** Bitrate from min(REMB, TAG)");
    bitrate = KMS_ENC_TREE_BIN_LIMIT (self, MIN (self->priv->remb_bitrate,
            self->priv->tag_bitrate));
  }

  if (bitrate <= 0) {
    GST_WARNING_OBJECT(self, "*** Bitrate still -1, use current");
    bitrate = self->priv->current_bitrate;
  } else {
    self->priv->current_bitrate = bitrate;
  }

  GST_WARNING_OBJECT(self, "*** Final bitrate: %d", bitrate);

  return bitrate;
}
j1elo commented 1 year ago

https://github.com/orgs/Kurento/discussions/17#discussioncomment-5440748

Also a nice one:

Ignoring requested bitrate max < min; setting min: 500000
Ignoring out of bounds requested target bitrate; setting default: 300000

Or this:

Ignoring out of bounds requested target bitrate; setting default: 300000

on this sequence:

                    await webRtcEndpoint.setMaxEncoderBitrate(10000000)
                    await webRtcEndpoint.setMinEncoderBitrate(10000000)

                    await webRtcEndpoint.setEncoderBitrate(3000000)

I think the bug is going to reside in the validation logic... however those numbers in your logs don't make too much sense, unless you had previously done some more changes before the logs posted here.

The logic is as follows:

The log messages are a bit confusing and I am changing them to be clearer and also indicate clearly what are the current and new values.

But, to explain your logs one by one:

Ignoring requested bitrate max < min; setting min: 500000

Means that the current minimum was 500000, and you tried to set a new maximum below that number. So, the new maximum was set to 500000 instead , the same value as the current minimum. Now the valid range of values is [500000, 500000].

Ignoring out of bounds requested target bitrate; setting default: 300000

Here you try to set some target bitrate, but because it falls out of the [min, max] range, the value is rejected and 300000 is used instead.

This logic is the one you currently have in your build of Kurento. I have been changing the code to make it easier to understand the logs. Also with the new code (not committed yet), the rules are more cohesive and more in line with what one would expect:

b) create a new setEncoderBitrates(min, max, target) API

That's be best to be honest, but the client code is just auto-generated from a JSON API definition file, and it generates get/set functions for each individual property.

j1elo commented 1 year ago

Also it will be interesting to see the enctreebin logs. Even the old code from Kurento 6.x had the "feature" that REMB will take priority, so even if you initially set an output target of, say, 5 Mbps, it won't matter if the subscriber/consumer (e.g. Chrome) is informing that its bandwidth is 800 Kbps (via REMB messages of the WebRTC connection)

neilyoung commented 1 year ago

Well, basically I just want to do this.

min 1.500.000 max 4.500.000 target 3.000.000

I can make sure that min < target < max, but from your comment I understand, I cannot set these values one after the other (as indicated?)

neilyoung commented 1 year ago

OK, first attempt. Above log level (btw: Added this to the init.d for the service. Couldn't find any trace. Stopped the service, exported to shell and run kurento-media-server ... better)

Application does this:

                    await webRtcEndpoint.setMinEncoderBitrate(1500000)
                    await webRtcEndpoint.setEncoderBitrate(3000000)
                    await webRtcEndpoint.setMaxEncoderBitrate(4500000)

The KMS log:

0:00:00.246606801  4558 0x5622f3b39e40 INFO    KurentoServerMethods ServerMethods.cpp:88:ServerMethods: Using above 80% of system limits will throw NOT_ENOUGH_RESOURCES exception
0:00:00.246614493  4558 0x5622f3b39e40 INFO    KurentoServerMethods ServerMethods.cpp:107:ServerMethods: System limits: 41994 threads, 1024 files
0:00:00.246670357  4558 0x5622f3b39e40 INFO       KurentoWorkerPool WorkerPool.cpp:67:WorkerPool: Worker thread pool size: 6
0:00:00.247020389  4558 0x5622f3b39e40 INFO    KurentoServerMethods ServerMethods.cpp:143:ServerMethods: RPC Request Cache is ENABLED
0:00:00.247149485  4558 0x5622f3b39e40 INFO    KurentoWebSocketTransport WebSocketTransport.cpp:203:initWebSocket: WebSocket server (ws://) listening on address '::', port 8888
0:00:00.247156883  4558 0x5622f3b39e40 INFO    KurentoWebSocketTransport WebSocketTransport.cpp:89:WebSocketTransport: Secure WebSocket server (wss://) not enabled
0:00:00.247337754  4558 0x5622f3b39e40 INFO      KurentoMediaServer main.cpp:259:main: Kurento Media Server started
0:00:07.337308656  4558 0x7f52b8002360 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:160:generateDefaultCertificates: Unable to load the RSA certificate from file. Using the default certificate.
0:00:07.366332837  4558 0x7f52b8002360 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:169:generateDefaultCertificates: Unable to load the ECDSA certificate from file. Using the default certificate.
0:00:07.366481483  4558 0x7f52b8002360 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:572:WebRtcEndpointImpl: No QOS-DSCP value set
0:00:07.366515198  4558 0x7f52b8002360 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:112:remove_not_supported_codecs_from_array:<kmswebrtcendpoint0> Removing not supported codec 'AMR/8000'
0:00:07.367941908  4558 0x7f52c40022c0 WARN                 default webrtcbarcodeOpenCVImpl.cpp:56:webrtcbarcodeOpenCVImpl: ZBar version: 0.23.0
0:00:07.368071551  4558 0x7f52ac04eaf0 WARN                 default webrtcbarcodeOpenCVImpl.cpp:250:inferenceThread: Inference thread started
0:00:07.369143718  4558 0x7f52c40022c0 DEBUG            agnosticbin kmsagnosticbin.c:1189:kms_agnostic_bin2_set_property:<kmsfilterelement0_kmsagnosticbin2-0> target-encoder-bitrate set: 300000
0:00:07.369154677  4558 0x7f52c40022c0 DEBUG            agnosticbin kmsagnosticbin.c:1223:kms_agnostic_bin2_set_property:<kmsfilterelement0_kmsagnosticbin2-0> max-encoder-bitrate set: 2147483647
0:00:07.369158264  4558 0x7f52c40022c0 DEBUG            agnosticbin kmsagnosticbin.c:1204:kms_agnostic_bin2_set_property:<kmsfilterelement0_kmsagnosticbin2-0> min-encoder-bitrate set: 0
0:00:07.369384561  4558 0x7f52c40022c0 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<'':sink_video_default> Add probe: DISCONT buffers and GAP events
0:00:07.507340241  4558 0x7f52a4002240 WARN        kmswebrtcsession kmswebrtcsession.c:818:kms_webrtc_session_set_stun_server_info:<kmswebrtcsession0> STUN server not configured! NAT traversal requires STUN or TURN
0:00:07.507361504  4558 0x7f52a4002240 WARN        kmswebrtcsession kmswebrtcsession.c:838:kms_webrtc_session_set_relay_info:<kmswebrtcsession0> TURN relay server not configured! NAT traversal requires STUN or TURN
0:00:07.511496205  4558 0x7f52ac00e860 INFO       KurentoWorkerPool WorkerPool.cpp:67:WorkerPool: Worker thread pool size: 6
0:00:07.738858033  4558 0x7f52b8002360 WARN       kmsbasertpsession kmsbasertpsession.c:774:kms_base_rtp_session_start_transport_send:<kmswebrtcsession0> Cannot configure connection for media (id=1)
0:00:07.788409801  4558 0x7f52a8016e40 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<'':sink_video_default> Add probe: DISCONT buffers and GAP events
0:00:07.888763396  4558 0x7f52a8016e40 INFO         basertpendpoint kmsbasertpendpoint.c:1956:kms_base_rtp_endpoint_jitterbuffer_set_latency:<rtpjitterbuffer0> Add probe: Set jitterbuffer latency
0:00:07.888828711  4558 0x7f52a8016e40 INFO         basertpendpoint kmsbasertpendpoint.c:2005:kms_base_rtp_endpoint_jitterbuffer_monitor_rtp_out:<rtpjitterbuffer0> Add probe: Adjust jitterbuffer PTS out
0:00:07.889512096  4558 0x7f52a8016e40 INFO         basertpendpoint kmsbasertpendpoint.c:2050:kms_base_rtp_endpoint_jitterbuffer_monitor_rtcp_in:<rtpjitterbuffer0> Add probe: Get jitterbuffer RTCP SR timing
0:00:07.889911970  4558 0x7f526c0060c0 INFO         basertpendpoint kmsbasertpendpoint.c:1941:kms_base_rtp_endpoint_jitterbuffer_set_latency_probe:<rtpjitterbuffer0> Setting latency to 500 ms
0:00:07.890574528  4558 0x7f526c0060c0 INFO         basertpendpoint kmsbasertpendpoint.c:1944:kms_base_rtp_endpoint_jitterbuffer_set_latency_probe:<rtpjitterbuffer0> Jitterbuffer latency set; remove probe
0:00:07.892342220  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:1189:kms_agnostic_bin2_set_property:<kmswebrtcendpoint0_kmsagnosticbin2-2> target-encoder-bitrate set: 3000000
0:00:07.892525593  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:1223:kms_agnostic_bin2_set_property:<kmswebrtcendpoint0_kmsagnosticbin2-2> max-encoder-bitrate set: 4500000
0:00:07.892795195  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:1204:kms_agnostic_bin2_set_property:<kmswebrtcendpoint0_kmsagnosticbin2-2> min-encoder-bitrate set: 1500000
0:00:07.895143322  4558 0x7f526c0060c0 INFO                kmsutils kmsutils.c:1511:kms_utils_depayloader_monitor_pts_out:<rtph264depay0> Add probe: Adjust depayloader PTS out
0:00:07.900837875  4558 0x7f526c0060c0 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<kmswebrtcendpoint0_kmsagnosticbin2-2:sink> Add probe: DISCONT buffers and GAP events
0:00:07.901580260  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:900:input_bin_src_caps_probe:<kmswebrtcendpoint0_kmsagnosticbin2-2> Set input caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c01fffe1000f6742c01f8c8d405017fcb00f08846a01000468ce3c80, level=(string)3.1, profile=(string)constrained-baseline, width=(int)640, height=(int)360, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:07.901732531  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:775:kms_agnostic_bin2_link_pad:<kmswebrtcendpoint0_kmsagnosticbin2-2> Upstream provided caps: ANY
0:00:07.902037200  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:784:kms_agnostic_bin2_link_pad:<kmswebrtcendpoint0_kmsagnosticbin2-2> Downstream wanted caps: video/x-raw, format=(string)BGRA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:07.930996508  4558 0x7f526c0060c0 INFO             agnosticbin kmsagnosticbin.c:730:kms_agnostic_bin2_find_or_create_bin_for_caps:<kmswebrtcendpoint0_kmsagnosticbin2-2> TRANSCODING ACTIVE for video
0:00:07.941487403  4558 0x7f52b400b640 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<kmsfilterelement0_kmsagnosticbin2-0:sink> Add probe: DISCONT buffers and GAP events
0:00:07.941540567  4558 0x7f52b400b640 DEBUG            agnosticbin kmsagnosticbin.c:900:input_bin_src_caps_probe:<kmsfilterelement0_kmsagnosticbin2-0> Set input caps: video/x-raw, width=(int)640, height=(int)360, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, format=(string)BGRA
0:00:21.934948069  4558 0x7f526c0060c0 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<kmswebrtcendpoint0_kmsagnosticbin2-2:sink> Add probe: DISCONT buffers and GAP events
0:00:21.935298217  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:900:input_bin_src_caps_probe:<kmswebrtcendpoint0_kmsagnosticbin2-2> Set input caps: video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c020ffe1000f6742c0208c8d407808bf700f08846a01000468ce3c80, level=(string)3.2, profile=(string)constrained-baseline, width=(int)960, height=(int)540, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
0:00:21.935720750  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:775:kms_agnostic_bin2_link_pad:<kmswebrtcendpoint0_kmsagnosticbin2-2> Upstream provided caps: ANY
0:00:21.935905418  4558 0x7f526c0060c0 DEBUG            agnosticbin kmsagnosticbin.c:784:kms_agnostic_bin2_link_pad:<kmswebrtcendpoint0_kmsagnosticbin2-2> Downstream wanted caps: video/x-raw, format=(string)BGRA, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:21.956383137  4558 0x7f52b400b640 INFO                kmsutils kmsutils.c:518:kms_utils_pad_monitor_gaps:<kmsfilterelement0_kmsagnosticbin2-0:sink> Add probe: DISCONT buffers and GAP events
neilyoung commented 1 year ago

Subscriber video quality terrible. WebRTC internals:

image
neilyoung commented 1 year ago

Wait I forgot the patch

neilyoung commented 1 year ago

Didn't compile. Changed it to GST_WARNING_OBJECT. Compiled.

/home/ubuntu/kurento/server/module-core/src/gst-plugins/commons/kmsenctreebin.c:288:3: error: implicit declaration of function ‘GST_WARN_OBJECT’; did you mean ‘GST_WARNING_OBJECT’? [-Werror=implicit-function-declaration] 288 | GST_WARN_OBJECT(self, "*** Final bitrate: %d", bitrate); | ^~~~~~~ | GST_WARNING_OBJECT