Kurento / bugtracker

[ARCHIVED] Contents migrated to monorepo: https://github.com/Kurento/kurento
46 stars 10 forks source link

'MediaFlowInStateChange' event is not triggered for Node js #569

Open codercoder12 opened 3 years ago

codercoder12 commented 3 years ago

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

Issue description

I am getting 0 size recording file once in 10 times, i have read the trouble shooting guide and followed all the points but -

As stated in documentation, that recording should be started when -

RecorderEndpoint has notified a MediaFlowInStateChange for ALL streams (so, if you record AUDIO+VIDEO, your application must receive a MediaFlowInStateChange event for audio, and another MediaFlowInStateChange event for video).

But 'MediaFlowInStateChange' is not triggered for recorder endpoint. But 'MediaFlowOutStateChange' is triggered for webrtc endpoint.

Webrtc endpoint is connected with Recorder endpoint.

Sample code -

let recorderVideo = false;
let recorderAudio = false;

recorderEndpoint.on('MediaFlowInStateChange', function(event){
    console.log('recorderEndpoint MediaFlowInStateChange ', event);

    if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
        recorderAudio=true;
    }

    if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
        recorderVideo=true;
    }

    if(recorderAudio && recorderVideo){
        recorderEndpoint.record((err) => {
        if (err) {
            console.error('error when trying to record');
            pipeline.release();
            callback(err);
            }
        });
    }
});

Recording never starts as 'MediaFlowInStateChange' is not triggered for recorder Endpoint

Context

Need to record video from browser in Safari, as Media Recorder API is not yet supported by default in Safari, So using webrtc for the same

How to reproduce?

I changed Magic-Mirror-Node Tutorial to support recording by replacing -

faceOverlayFilter pipeline to RecorderEndpoint pipeline

And to start recording I added below code to check if 'MediaFlowInStateChange' is flowing for both Audio and Video for Recorder Endpoint, and only then start recording but this event is not fired.

let recorderVideo = false;
let recorderAudio = false;

recorderEndpoint.on('MediaFlowInStateChange', function(event){
    console.log('recorderEndpoint MediaFlowInStateChange ', event);

    if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
        recorderAudio=true;
    }

    if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
        recorderVideo=true;
    }

    if(recorderAudio && recorderVideo){
        recorderEndpoint.record((err) => {
        if (err) {
            console.error('error when trying to record');
            pipeline.release();
            callback(err);
            }
        });
    }
});

Expected & current behavior

Expected behavior - 'MediaFlowInStateChange' should have been fired in recorder endpoint as 'MediaFlowOutStateChange' for webrtc endpoint was fired for both audio and video.

Current behavior - 'MediaFlowInStateChange' is not fired in recorder endpoint even when 'MediaFlowOutStateChange' for webrtc endpoint was fired for both audio and video.

(Optional) Possible solution

Info about your environment

About the application -

  1. I am using Webrtc Endpoint which is connected to Recorder Endpoint
  2. Coturn server is installed and configured with KMS

About Kurento Media Server

About your Application Server

About end-user clients

Run these commands

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
Kurento Media Server version: 6.15.0
Found modules:
    'core' version 6.15.0
    'elements' version 6.15.0
    'filters' version 6.15.0
ii  gstreamer1.5-libav:amd64             1.8.1-1kurento1.18.04               amd64        libav plugin for GStreamer
ii  gstreamer1.5-nice:amd64              0.1.18-0kurento1.18.04              amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-plugins-bad:amd64       1.8.1-1kurento5.18.04               amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-base:amd64      1.8.1-1kurento2.18.04               amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64      1.8.1-1kurento5.18.04               amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64      1.8.1-1kurento1.18.04               amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-pulseaudio:amd64        1.8.1-1kurento5.18.04               amd64        GStreamer plugin for PulseAudio
ii  kms-core                             6.15.0-0kurento1.18.04              amd64        Kurento Core module
ii  kms-elements                         6.15.0-0kurento1.18.04              amd64        Kurento Elements module
ii  kms-filters                          6.15.0-0kurento1.18.04              amd64        Kurento Filters module
ii  kms-jsonrpc                          6.15.0-0kurento1.18.04              amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                           1.6.3-1kurento1.18.04               amd64        Kurento jsoncpp library
ii  kurento-media-server                 6.15.0-0kurento1.18.04              amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64  1.8.1-1kurento5.18.04               amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.18.04               amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64              1.8.1-1kurento2.18.04               amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                      0.1.18-0kurento1.18.04              amd64        ICE library (shared library)
ii  libsrtp0:amd64                       1.6.0-0kurento1.18.04               amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                           0.9.2-1kurento1.18.04               amd64        sctp-refimpl library
ii  openh264                             1.5.0-0kurento1.18.04               amd64        H.264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugins-bad-1.5:amd64   1.8.1-1kurento5.18.04               amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins               0.10.0-1kurento1.18.04              amd64        OpenWebRTC specific GStreamer plugins
github-actions[bot] commented 3 years ago

Hello @codercoder12! :wave: we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

j1elo commented 3 years ago

Hi, does the Kurento logs show any suspicious Warning or Error message?

does this issue happen randomly? You said it fails 1 in each 10 times... is there any difference between that failed time and the successful ones? (differences in devices, browsers, number of peers connected to the same Pipeline, different Pipeline, different recording format... anything you can think of)

is the WebRtcEndpoint connected directly to the Recorder? Or are there any other intermediate elements in the Pipeline?

Are you getting the Error event from the Pipeline itself, or any of the Elements/Endpoints inside?

codercoder12 commented 3 years ago

-- does the Kurento logs show any suspicious Warning or Error message? --> log says that Error: Already in state stop

--does this issue happen randomly? You said it fails 1 in each 10 times. --> Yes it fails(by failing I meant i get 0 size recorded files) randomly 1 in 10 times meant 10-20 % of the time I get 0 size recorded files, but sometimes it is 30-40%

--is there any difference between that failed time and the successful ones? (differences in devices, browsers, number of peers connected to the same Pipeline, different Pipeline, different recording format... anything you can think of). --> No, no difference at all, same safari browser(14.0) in same macbook pro, only one peer connected to the piepline (which is me, as i am testing it),same recording format which is MKV(as our requirement is to use H264 and AAC).

Only webrtc endpoint is connected with recorder endpoint(As i want to record video in browser from safari, as media recorder api is not suported by default in safari)

Everything is same, it is just that I was getting 0 size recorded files, then I read the trouble shooting guide to remove 0 size recorded files. In it, it is mentioned to wait for MediaFlowInStateChange for ALL streams (so, if you record AUDIO+VIDEO, your application must receive a MediaFlowInStateChange event for audio, and another MediaFlowInStateChange event for video) for recorder Endpoint, only then start recording.

But MediaFlowInStateChange event is never triggered in node, so recorder endpoint never starts recording with below code--

let recorderVideo = false;
let recorderAudio = false;

recorderEndpoint.on('MediaFlowInStateChange', function(event){
   console.log('recorderEndpoint MediaFlowInStateChange ', event);

   if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
       recorderAudio=true;
   }

   if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
       recorderVideo=true;
   }

   if(recorderAudio && recorderVideo){
       recorderEndpoint.record((err) => {
       if (err) {
           console.error('error when trying to record');
           pipeline.release();
           callback(err);
           }
       });
   }
});

I was using MediaFlowInStateChange event to remove 0 size recorded files, but as it is not fired, recording dosent starts.

But webrtc Endpoint fires MediaFlowOutStateChange event for both Audio and Video

j1elo commented 3 years ago

Hi, thanks for the info.

Please let me see your SDP Offer and Answers. For this, first configure Kurento debug logs to this value:

GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,KurentoMediaElementImpl:5,agnosticbin*:5,kmssdpsession:5,webrtcendpoint:5,kmswebrtcsession:5,kmsiceniceagent:5"

Then restart Kurento, repeat your test until it fails, and share the logs with us (you can post it in pastebin or similar and share the link here, or send it to the email in my Github profile).

j1elo commented 3 years ago

I'm not sure if the previous message got a wrong formatting. Use this value for the debug verbosity:

GST_DEBUG="3,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,KurentoMediaElementImpl:5,agnosticbin*:5,kmssdpsession:5,webrtcendpoint:5,kmswebrtcsession:5,kmsiceniceagent:5"
codercoder12 commented 3 years ago

Hi, I have shared the logs and error with you on your email mentioned on github profile. Please let me know if anything else is required.

j1elo commented 3 years ago

I see this in the logs:

2021-01-15T06:36:29 Kurento Media Server started

2021-01-15T06:41:35 Process SDP Offer 1

MediaFlowOutStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowOutStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' 2021-01-15T06:41:48 Process SDP Offer 2 MediaFlowOutStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowOutStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' 2021-01-15T06:42:00 Process SDP Offer 3 MediaFlowOutStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowInStateChange: FLOWING, type: 'audio' MediaFlowOutStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' MediaFlowInStateChange: FLOWING, type: 'video' 2021-01-15T06:42:17 Process SDP Offer 4 NO FLOW AT ALL Is it correct that the problem happened on attempt 4? Or the attempt 4 was just aborted (you hit Ctrl+C or similar to early-finish execution)?
codercoder12 commented 3 years ago

Yes, it failed on 4th attempt(I clicked stop- to stop the recording and it created 0 size file) and I did not click Ctrl+C or similar to stop early-finish execution

And this was my attempt without using below code(checking for 'MediaFlowInStateChange' events for audio and video)

let recorderVideo = false;
let recorderAudio = false;

recorderEndpoint.on('MediaFlowInStateChange', function(event){
   console.log('recorderEndpoint MediaFlowInStateChange ', event);

   if(event.mediaType=='AUDIO' && event.state == "FLOWING") {
       recorderAudio=true;
   }

   if(event.mediaType=='VIDEO'&& event.state == "FLOWING") {
       recorderVideo=true;
   }

   if(recorderAudio && recorderVideo){
       recorderEndpoint.record((err) => {
       if (err) {
           console.error('error when trying to record');
           pipeline.release();
           callback(err);
           }
       });
   }
});

I was just using the below code to start the recording without checking the 'MediaFlowInStateChange' in recorderEndpoint, if I use the above code(checking for 'MediaFlowInStateChange' events for audio and video) then recording dosent start, as event is not fired for 'MediaFlowInStateChange' for recorder endpoint.

recorderEndpoint.record((err) => {
       if (err) {
           console.error('error when trying to record');
           pipeline.release();
           callback(err);
           }
       });

What I noticed is that 'MediaFlowInStateChange' events for audio and video for recorder endpoint are fired only when I call recorderEndpoint.record

XingkaiAcquire commented 2 years ago

Hi, I am encountering the same issues, any updates?