search

Kurento / kurento

Kurento WebRTC Media Server
Apache License 2.0
263 stars 55 forks source link

Can't recording 720P .mp4 (H264) video #49

Open TLexYuW opened 1 year ago

TLexYuW commented 1 year ago

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

Issue description

Context

I'm trying to accomplish this:

Assuming there are 10 participants in one room, including a host, only the .mp4 (H264) video of the host will be recorded, and audio of the host along with the audio of the other 9 participants mixed by hubport without their video, separate and store as two individual files.

General
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (mp42/mp41/isom/iso2)
File size                                : 121 MiB
Duration                                 : 10 min 4 s
Overall bit rate                         : 1 683 kb/s
Frame rate                               : 29.708 FPS

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Constrained Baseline@L3
Format settings                          : 1 Ref Frames
Format settings, CABAC                   : No
Format settings, Reference frames        : 1 frame
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 10 min 4 s
Bit rate                                 : 1 681 kb/s
Width                                    : 640 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Variable
Frame rate                               : 29.708 FPS
Minimum frame rate                       : 12.500 FPS
Maximum frame rate                       : 33.223 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.184
Stream size                              : 121 MiB (100%)
Codec configuration box                  : avcC

ERROR 26856 --- [EventExec-e2-t7] o.k.t.helloworld.HelloWorldRecHandler : [RecorderEndpoint::ErrorEvent] Error code 1: 'STREAM_ERROR_FAILED', source: d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint, timestamp: 1689673935353, tags: [], description: Error code 1: Internal data stream error., source: kmsrecorderendpoint2, element: kmsrecorderendpoint2, debug info: gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline4/GstAppSrc:videoSrc: streaming stopped, reason not-negotiated (-4)

    Media Server Error:
```bash
632 0x7fe1d8002240 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:689:MediaElementImpl: Configured target video bitrate for media transcoding: 6000000 bps
632 0x7fe1d8002240 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:694:MediaElementImpl: Configured minimum video bitrate for media transcoding: 2000000 bps
632 0x7fe1d8002240 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:699:MediaElementImpl: Configured maximum video bitrate for media transcoding: 6000000 bps
632 0x7fe1d8002240 DEBUG   KurentoBaseRtpEndpointImpl BaseRtpEndpointImpl.cpp:114:BaseRtpEndpointImpl: No predefined RTP MTU found in config; using default
632 0x7fe1d8002240 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:572:WebRtcEndpointImpl: No QOS-DSCP value set
632 0x7fe1d8002240 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:112:remove_not_supported_codecs_from_array:<kmswebrtcendpoint1> Removing not supported codec 'AMR/8000'
632 0x7fe1d8002240 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:599:WebRtcEndpointImpl: No predefined external IPv4 address found in config; you can set one or default to STUN automatic discovery
632 0x7fe1d8002240 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:610:WebRtcEndpointImpl: No predefined external IPv6 address found in config; you can set one or default to STUN automatic discovery
632 0x7fe1d8002240 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:621:WebRtcEndpointImpl: No predefined network interfaces found in config; you can set one or default to ICE automatic discovery
632 0x7fe1d8002240 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:631:WebRtcEndpointImpl: ICE-TCP option not found in config; you can set it or default to 1 (TRUE)
632 0x7fe1d8002240 INFO    KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:645:WebRtcEndpointImpl: Predefined STUN server: stun.l.google.com:19302:3478
632 0x7fe1d8002240 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:667:WebRtcEndpointImpl: TURN relay server not found in config
632 0x55c84ddb6640 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:689:MediaElementImpl: Configured target video bitrate for media transcoding: 6000000 bps
632 0x55c84ddb6640 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:694:MediaElementImpl: Configured minimum video bitrate for media transcoding: 2000000 bps
632 0x55c84ddb6640 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:699:MediaElementImpl: Configured maximum video bitrate for media transcoding: 6000000 bps
632 0x55c84ddb6640 INFO    KurentoRecorderEndpointImpl RecorderEndpointImpl.cpp:92:RecorderEndpointImpl: Set MP4 profile
632 0x7fe1dc001400 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:1004:connect: Connecting d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/28f7702f-e53f-4e8d-9fe3-00237a07ecca_kurento.WebRtcEndpoint -> d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint params AUDIO default default
632 0x7fe1e0002300 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:1004:connect: Connecting d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/28f7702f-e53f-4e8d-9fe3-00237a07ecca_kurento.WebRtcEndpoint -> d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint params VIDEO default default
632 0x7fe1d806ab60 DEBUG   KurentoWebRtcEndpointImpl WebRtcEndpointImpl.cpp:293:newSelectedPairFull:<kmswebrtcendpoint1> New candidate pair selected, local: 'candidate:4 1 TCP 935330047 192.168.1.191 3080 typ prflx raddr 172.17.0.2 rport 46075 tcptype active', remote: 'candidate:418784936 1 TCP 1518280447 192.168.1.155 63259 typ host tcptype passive', stream_id: '1', component_id: 1
632 0x7fe1b80088c0 DEBUG   KurentoBaseRtpEndpointImpl BaseRtpEndpointImpl.cpp:202:updateConnectionState:<kmswebrtcendpoint1> ConnectionState changed to 'CONNECTED'
632 0x7fe1b8008b60 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:526:mediaFlowOutStateChanged:<kmswebrtcendpoint1> MediaFlowOutStateChanged: FLOWING, pad: 'default', type: 'audio'
632 0x7fe1b8008b60 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:596:onMediaTranscodingStateChanged:<kmswebrtcendpoint1> MediaTranscodingStateChanged: TRANSCODING, bin: 'kmswebrtcendpoint1_kmsagnosticbin2-8', type: 'audio'
632 0x7fe1b8008b60 WARN                kmsutils kmsutils.c:1452:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay1> Fix PTS not strictly increasing, last: 0:00:00.259455791, current: 0:00:00.259455791, fixed = last + 1: 0:00:00.260455791
632 0x7fe1b8008b60 WARN                kmsutils kmsutils.c:1452:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay1> Fix PTS not strictly increasing, last: 0:00:00.260455791, current: 0:00:00.259455791, fixed = last + 1: 0:00:00.261455791
632 0x7fe1b8008b60 WARN                kmsutils kmsutils.c:1452:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay1> Fix PTS not strictly increasing, last: 0:00:00.261455791, current: 0:00:00.259455791, fixed = last + 1: 0:00:00.262455791
632 0x7fe1b8008b60 WARN                kmsutils kmsutils.c:1452:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay1> Fix PTS not strictly increasing, last: 0:00:00.262455791, current: 0:00:00.259455791, fixed = last + 1: 0:00:00.263455791
632 0x7fe1b8008b60 WARN                kmsutils kmsutils.c:1452:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay1> Fix PTS not strictly increasing, last: 0:00:00.263455791, current: 0:00:00.259455791, fixed = last + 1: 0:00:00.264455791
632 0x7fe1b8008b60 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
632 0x7fe11c004360 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:601:onMediaTranscodingStateChanged:<kmswebrtcendpoint1> MediaTranscodingStateChanged: NOT TRANSCODING, bin: 'kmswebrtcendpoint1_kmsagnosticbin2-9', type: 'video'
632 0x7fe11c004360 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:526:mediaFlowOutStateChanged:<kmswebrtcendpoint1> MediaFlowOutStateChanged: FLOWING, pad: 'default', type: 'video'
632 0x7fe17400ac60 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:561:mediaFlowInStateChanged:<kmsrecorderendpoint2> MediaFlowInStateChanged: FLOWING, pad: 'default', type: 'video'
632 0x7fe1a0010000 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:561:mediaFlowInStateChanged:<kmsrecorderendpoint2> MediaFlowInStateChanged: FLOWING, pad: 'default', type: 'audio'
632 0x7fe1e00088c0 WARN                   qtmux gstqtmux.c:2981:gst_qt_mux_start_file:<mp4mux2> Robust muxing requires reserved-moov-update-period to be set
632 0x7fe164005580 DEBUG   KurentoRecorderEndpointImpl RecorderEndpointImpl.cpp:199:onStateChanged:<kmsrecorderendpoint2> State changed to Recording
632 0x7fe1e0008920 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<mp4mux2> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c033ffe100196742c033959005005bb01100000303e80001ccf08da088464801000468cb8f20, level=(string)5.1, profile=(string)constrained-baseline, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
632 0x7fe1e0008920 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<mp4mux2> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c033ffe100196742c033959005005bb01100000303e80001ccf08da088464801000468cb8f20, level=(string)5.1, profile=(string)constrained-baseline, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
632 0x7fe1e0008920 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<videoSrc> error: Internal data stream error.
632 0x7fe1e0008920 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<videoSrc> error: streaming stopped, reason not-negotiated (-4)
632 0x7fe1e0008920 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<mp4mux2> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c033ffe100196742c033959005005bb01100000303e80001ccf08da088464801000468cb8f20, level=(string)5.1, profile=(string)constrained-baseline, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
632 0x7fe1e0008920 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<mp4mux2> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c033ffe100196742c033959005005bb01100000303e80001ccf08da088464801000468cb8f20, level=(string)5.1, profile=(string)constrained-baseline, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
632 0x55c84ddb6150 ERROR   KurentoMediaElementImpl MediaElementImpl.cpp:456:processBusMessage:<kmsrecorderendpoint2> Error code 1: Internal data stream error., source: kmsrecorderendpoint2, element: kmsrecorderendpoint2, debug info: gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline4/GstAppSrc:videoSrc:
streaming stopped, reason not-negotiated (-4)
632 0x7fe1e0008920 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<mp4mux2> pad video_0 refused renegotiation to video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0142c033ffe100196742c033959005005bb01100000303e80001ccf08da088464801000468cb8f20, level=(string)5.1, profile=(string)constrained-baseline, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
632 0x7fe1b80088c0 DEBUG   KurentoBaseRtpEndpointImpl BaseRtpEndpointImpl.cpp:165:updateMediaState:<kmswebrtcendpoint1> MediaState changed to 'CONNECTED'
632 0x7fe1dc05a800 DEBUG        KurentoMediaSet MediaSet.cpp:121:doGarbageCollection: Running garbage collector
632 0x7fe1640055e0 WARN        recorderendpoint kmsrecorderendpoint.c:1346:kms_recorder_endpoint_on_eos:<kmsrecorderendpoint2> Releasing pending pads
632 0x7fe1640055e0 DEBUG   KurentoRecorderEndpointImpl RecorderEndpointImpl.cpp:186:onStateChanged:<kmsrecorderendpoint2> State changed to Stopped
632 0x7fe138067210 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:1133:disconnect: Disconnecting d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/28f7702f-e53f-4e8d-9fe3-00237a07ecca_kurento.WebRtcEndpoint - d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint params AUDIO default default
632 0x7fe138067210 DEBUG   KurentoMediaElementImpl MediaElementImpl.cpp:1133:disconnect: Disconnecting d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/28f7702f-e53f-4e8d-9fe3-00237a07ecca_kurento.WebRtcEndpoint - d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint params VIDEO default default
632 0x7fe128061980 DEBUG        KurentoMediaSet MediaSet.cpp:463:async_delete: Destroying WebRtcEndpoint -> d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/28f7702f-e53f-4e8d-9fe3-00237a07ecca_kurento.WebRtcEndpoint
632 0x7fe15c00d590 DEBUG        KurentoMediaSet MediaSet.cpp:463:async_delete: Destroying RecorderEndpoint -> d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline/640aeaa9-a655-4066-b5c9-f6103d819010_kurento.RecorderEndpoint
632 0x7fe128061980 DEBUG        KurentoMediaSet MediaSet.cpp:463:async_delete: Destroying MediaPipeline -> d9431e68-3a83-46e3-b50b-ecfcde68d2e3_kurento.MediaPipeline

How to reproduce?

Expected & current behavior

(Optional) Possible solution

Info about your environment

configs

encoderBitrate=6000000

//...

minEncoderBitrate=2000000

//...

maxEncoderBitrate=6000000


### About Kurento Media Server

* Kurento version:  7.0.0
* Server OS: Ubuntu 20.04 (Focal)      
* Installation method:
    <!-- Fill with an 'x' in the boxes that apply. -->
  - [x] [apt-get]
  - [x] [Docker]
  - [] [AWS CloudFormation]
  - [] [Built from sources]

[apt-get]: https://doc-kurento.readthedocs.io/en/latest/user/installation.html#installation-local
[Docker]: https://doc-kurento.readthedocs.io/en/latest/user/installation.html#installation-docker
[AWS CloudFormation]: https://doc-kurento.readthedocs.io/en/latest/user/installation.html#installation-aws
[Built from sources]: https://doc-kurento.readthedocs.io/en/latest/dev/dev_guide.html#dev-sources

### About your Application Server

* Programming Language:  Java
* Kurento Client version:  7.0.0

### About end-user clients

* Device(s):  PC
* OS(es):  Windows 10
* Browser(s):  Chrome 114

### Run these commands

<!--
Run these commands in your Kurento machine, and paste the output
inside the ```triple backquotes``` to preserve formatting.
-->

```bash
gst-inspect-1.0 --gst-version
GStreamer Core Library version 1.16.3
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
kurento-media-server --version
0:00:00.017338554  4554 0x55f2eba16150 DEBUG   KurentoModuleManager ModuleManager.cpp:172:loadModules: Looking for modules in path: /usr/lib/x86_64-linux-gnu/kurento/modules
0:00:00.017453925  4554 0x55f2eba16150 DEBUG   KurentoModuleManager ModuleManager.cpp:182:loadModules: Found file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so
0:00:00.017879398  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:89:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so
0:00:00.017939089  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:136:loadModule: Loaded module: core, version: 7.0.0, date: Mar 21 2023 16:57:44
0:00:00.017997539  4554 0x55f2eba16150 DEBUG   KurentoModuleManager ModuleManager.cpp:182:loadModules: Found file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so
0:00:00.030635033  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:89:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so
0:00:00.030701494  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:136:loadModule: Loaded module: elements, version: 7.0.0, date: Mar 21 2023 17:03:47
0:00:00.030737904  4554 0x55f2eba16150 DEBUG   KurentoModuleManager ModuleManager.cpp:182:loadModules: Found file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so
0:00:00.031636361  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:89:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so
0:00:00.031695762  4554 0x55f2eba16150 INFO    KurentoModuleManager ModuleManager.cpp:136:loadModule: Loaded module: filters, version: 7.0.0, date: Mar 21 2023 17:09:51
Kurento Media Server version: 7.0.0
Found modules:
        'core' version 7.0.0
        'elements' version 7.0.0
        'filters' version 7.0.0
dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.0-nice:amd64              0.1.18-0kurento1                  amd64        ICE library (GStreamer plugin)
ii  gstreamer1.0-plugins-good:amd64      1:1.16.3-0ubuntu1.1kurento1       amd64        GStreamer plugins from the "good" set
ii  gstreamer1.0-plugins-good-kurento    1.0.0-0kurento1ubuntu1            amd64        Dummy package that depends on Kurento patched gstreamer1.0-plugins-good (shared library)
ii  kurento-jsonrpc                      7.0.0-1kurento1                   amd64        Kurento JSON-RPC library
ii  kurento-media-server                 7.0.0-1kurento1                   amd64        Kurento Media Server
ii  kurento-module-core                  7.0.0-1kurento1                   amd64        Core module for use by Kurento Media Server
ii  kurento-module-elements              7.0.0-1kurento1                   amd64        Kurento Elements module
ii  kurento-module-filters               7.0.0-1kurento1                   amd64        Kurento Filters module
ii  libgstreamer-plugins-good1.0-0:amd64 1:1.16.3-0ubuntu1.1kurento1       amd64        GStreamer development files for libraries from the "good" set
ii  libnice10:amd64                      0.1.18-0kurento1                  amd64        ICE library (shared library)
ii  libsrtp-kurento                      1.0.0-0kurento1ubuntu1            amd64        Dummy package that depends on Kurento patched libsrtp (shared library)
ii  libsrtp2-1:amd64                     1:2.3.0-2kurento1                 amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  openh264                             2.3.0-0kurento1                   amd64        OpenH264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugin                  1.0.0-0kurento1ubuntu1            amd64        OpenH264 plugin for GStreamer
Suaro commented 5 months ago

Hi,

Any update with this issue? I know Kurento is in low mantainance mode, but I think there are a possible compatibility issue in Kurento with new H264/X264 releases of 2024.

I reproduce same behaviour in iOS 17.4 devices in Safari. It happens 80% of connections. When everything works fine, there are no WARN/ERROR logs related to this, but sometimes, Kurento in latest version (7.0.1) prints:

(enabling MKV):

ERROR   KurentoMediaElementImpl MediaElementImpl.cpp:456:processBusMessage:<kmsrecorderendpoint43> Error code 10: Could not multiplex stream., source: kmsrecorderendpoint43, element: kmsrecorderendpoint43, debug info: matroska-mux.c(991): gst_matroska_mux_can_renegotiate_caps (): /GstPipeline:pipeline59/GstMatroskaMux:avmuxer_matroskamux37:

Most caps changes are not supported by Matroska

Current: video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)0164001effe100132764001eac131470781469b81010103c22119e01000428ee3cb0, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, framerate=(fraction)0/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

New: video/x-h264, codec_data=(buffer)0164001effe1001d6764001eacb40b077c44f80b706010540000030005dcd650003c58ba8001000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)338, height=(int)452, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

Enabling MP4:


0:12:09.842125763    12 0x7f21dde0f2a0 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<avmuxer_mp4mux12> pad video_0 refused renegotiation to video/x-h264, codec_data=(buffer)0164001effe1001f6764001eacb40f028dff80a900a9b706010540000003005dcd650003c58ba801000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, pixel-aspect-ratio=(fraction)338/339, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

0:12:09.843460714    12 0x7f21dde0f2a0 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<avmuxer_mp4mux12> pad video_0 refused renegotiation to video/x-h264, codec_data=(buffer)0164001effe1001f6764001eacb40f028dff80a900a9b706010540000003005dcd650003c58ba801000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, pixel-aspect-ratio=(fraction)338/339, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

0:12:09.843517636    12 0x7f21dde0f2a0 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<avMuxerVideoSrc> error: Internal data stream error.

0:12:09.843527036    12 0x7f21dde0f2a0 WARN                 basesrc gstbasesrc.c:3072:gst_base_src_loop:<avMuxerVideoSrc> error: streaming stopped, reason not-negotiated (-4)

0:12:09.844702412    12 0x7f21dde0f2a0 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<avmuxer_mp4mux12> pad video_0 refused renegotiation to video/x-h264, codec_data=(buffer)0164001effe1001f6764001eacb40f028dff80a900a9b706010540000003005dcd650003c58ba801000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, pixel-aspect-ratio=(fraction)338/339, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

0:12:09.844930737    12 0x7f22132bdb50 ERROR   KurentoMediaElementImpl MediaElementImpl.cpp:456:processBusMessage:<kmsrecorderendpoint12> Error code 1: Internal data stream error., source: kmsrecorderendpoint12, element: kmsrecorderendpoint12, debug info: gstbasesrc.c(3072): gst_base_src_loop (): /GstPipeline:pipeline15/GstAppSrc:avMuxerVideoSrc:
streaming stopped, reason not-negotiated (-4)

0:12:09.846331946    12 0x7f21dde0f2a0 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<avmuxer_mp4mux12> pad video_0 refused renegotiation to video/x-h264, codec_data=(buffer)0164001effe1001f6764001eacb40f028dff80a900a9b706010540000003005dcd650003c58ba801000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, pixel-aspect-ratio=(fraction)338/339, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

0:12:09.847353331    12 0x7f21dde0f2a0 WARN                   qtmux gstqtmux.c:5118:gst_qt_mux_can_renegotiate:<avmuxer_mp4mux12> pad video_0 refused renegotiation to video/x-h264, codec_data=(buffer)0164001effe1001f6764001eacb40f028dff80a900a9b706010540000003005dcd650003c58ba801000468ef3cb0, stream-format=(string)avc, alignment=(string)au, level=(string)3, profile=(string)high, width=(int)480, height=(int)640, pixel-aspect-ratio=(fraction)338/339, framerate=(fraction)0/1, interlace-mode=(string)progressive, colorimetry=(string)1:4:5:4, chroma-site=(string)mpeg2, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono

Curiously, we receive correctly images in our custom OpenCV filter (we're saving all frames Kurento receive for testing purposes) but final video is pixelated, trimmed, partially black... Could be related to transcoding or something like that?

According to GStreamer errors, it seems an incompatibility issue. Any light on that? By the way, I reproduce same behaviour in OpenVidu demo with Kurento backend.

Regards.

Suaro commented 5 months ago

Could it be related to old issue https://github.com/Kurento/bugtracker/issues/535? As far as I saw in @j1elo comments, no fix was implemented (because matroska and VP8). I don't know if at least, you have a workaround.

Regards.

j1elo commented 5 months ago

The problem is exactly issue https://github.com/Kurento/bugtracker/issues/535, as mentioned above, and it stems from this:

Current: video/x-h264, width=(int)480, height=(int)640
New: video/x-h264, width=(int)338, height=(int)452

Problem was that Kurento got upgraded to use the GStreamer dependency that comes with Ubuntu 20.04 (GStreamer 1.16), but it turns out that this came with breaking changes in behaviors (I guess at some point they considered it a bug but then changed their mind and considered it a regression, but in the meantime a release was cut, published, and ended up being part of Ubuntu).

Solution is not small as it would consist on researching a bit about that GStreamer issue, see which commits fixed the problem, then locate which releases already contain the commit(s) to be clear on what GStreamer version needs to be used. And then, the major part, upgrading Kurento to make it work with that version. It's not a trivial thing to do because GStreamer sometimes changes internals such that old code needs to be changed in order to use newer versions of the library.

I'd like to get some of my own time at some point to do the upgrade. For now, may I suggest you try using the latest available version of Kurento 6.x?

j1elo commented 5 months ago

Seems that I had done the initial part that I mentioned:

https://github.com/Kurento/bugtracker/issues/535#issuecomment-1280626392

So GStreamer >= 1.20 seems to contain the fixes already. But as the next comment in that issue points out, there seem to be some other additional issues when using those newer versions of GStreamer!

No doubt, those corruption issues come probably from the fact that the user who tested with GStteamer 1.21 tried to use Kurento 7 as-is, without the work to adapt it to possible breaking changes. As expected, some work needs to be done explicitly to support newer versions of the library, it cannot be used as-is like nothing bad would happen.

Suaro commented 5 months ago

Hi @j1elo

Thanks for your quick response. I think I 've understood properly root cause of problem.

I got your advice to rollback to previos Kurento versión (6.18.0) but I've been looking for a quick solution and I found (already deprecated) degradationPreference parameter (https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/setParameters#degradationpreference) and I think it could be a mitigation and possible cause of behaviour we're experiencing since iOS 17.4.

According to iOS 17.4 release notes, they "fixed" degradationPreference's "mantain-framerate" option (https://developer.apple.com/documentation/safari-release-notes/safari-17_4-release-notes#WebRTC) and It seems they fixed "balanced" option too (which is default value), so probable it's related to our recent problem.

When I have a chance to test It, I'll come back to share my results.

Regards.

Suaro commented 5 months ago

Hi all,

In case it helps to anyone, issue for me is solved setting "degradationPreference" parameter to "mantain-resolution". Sample code:

  private addMediaStreamTracks(stream: MediaStream, webRtcPeer: RTCPeerConnection) {
    const tracksResponse = stream.getTracks();
    tracksResponse.forEach((track: MediaStreamTrack) => {
      if (webRtcPeer) {
        const sender = webRtcPeer.addTrack(track, stream);

        if(track.kind == 'video') {
          track.contentHint = 'text';
          const parameters = sender.getParameters();
          parameters.degradationPreference = "maintain-resolution";
          sender.setParameters(parameters);
        }
      }
    });
  }

Anyway, this is a mitigation because Kurento shouldn't crash. Do you have plans to upgrade it to latest version of Gstreamer? I think I could help in case is needed.

Regards.