Cant see others video feed if im not feeding video

[gettyhub]

Describe the bug

If I don't have a video camera device I'm not seeing any video obviously, but I can't see other people's video feeds that are either.

To Reproduce

Steps to reproduce the behavior:

Have participants that doesn't have video feed although there are others who are.
People with video see others with video but guy without video doesn't see others with video.

Expected behavior

Should be able to see others videos, especially the main feed even if I don't have, or not doing a video feed myself.

Screenshots

If applicable, add screenshots to help explain your problem.

Desktop - Mobile

Please complete the following information:

• Device or OS: [e.g. iPhone6, Android, Linux, Mac, Windows]
• Browser [e.g. Chrome, Safari]
• Version [e.g. 22]

Additional context

Add any other context about the problem here.

[miroslavpejic85]

Hello @gettyhub, I tried now seems ok

Did you have this issue on my Live demo or where? Thank you.

[miroslavpejic85]

Ok, the issue seems to be if the first joining in the room is a device that has a webcam, and then one that doesn't. Vice versa it seems to work, everything is solved with a reload of the page on the device that has a webcam, but it's not good. Thanks you for the report.

[gettyhub]

On own VPS.

Can get a video feed on non video guys if non video guys are connected first and video guys connect after that.

[gettyhub]

A non video guy will show up to another non video guy but a video guy that was on before the non video will not even have a window show up for a video guy on a non video guy.

This seems to be a problem for a person who has no video device and it doesn't ask for permission to use video if the video device is not present.

If a video device is present, but muted, turning video on will of course show up. However it still doesn't show up if the video guy came before the nonvideo guy.

[gettyhub]

No audio comes in either on a non video guy from a video guy if video is before the non video guy.

[miroslavpejic85]

OK, Do you want to try to fix the bug yourself contributing with a PR? :)

[gettyhub]

OK, Do you want to try to fix the bug yourself contributing with a PR? :)

Depends on whether I can find the bug or not.

[gettyhub]

In client.js here's where we need to look

async function handleOnTrack(peer_id, peers) {
    console.log('[ON TRACK] - peer_id', { peer_id: peer_id });
    peerConnections[peer_id].ontrack = (event) => {
        console.log('[ON TRACK] - kind', { peer_id: peer_id, kind: event.track.kind });
        if (event.streams && event.streams[0]) {
            console.log('[ON TRACK] - peers', peers);
            let peer_video = peers[peer_id]['peer_video'];
            let peer_audio = peers[peer_id]['peer_audio'];
            let loadRemoteMedia = false;
            if (
                // conditions to load remote media
                (peer_audio && peer_video && event.track.kind === 'video') ||
                (!peer_audio && peer_video && event.track.kind === 'video') ||
                (peer_audio && !peer_video && event.track.kind === 'audio')
            ) {
                loadRemoteMedia = true;
            }
            if (loadRemoteMedia) loadRemoteMediaStream(event.streams[0], peers, peer_id);
        } else {
            console.log('[ON TRACK] - SCREEN SHARING', { peer_id: peer_id, kind: event.track.kind });
            let remoteVideoStream = getId(peer_id + '_video');
            // attach newStream with screen share video and audio already existing
            let inboundStream = new MediaStream([remoteVideoStream.srcObject.getAudioTracks()[0]]);
            inboundStream.addTrack(event.track);
            attachMediaStream(remoteVideoStream, inboundStream);
        }
    };
}

[gettyhub]

OK if I comment out the whole part from let loadRemoteMedia = false down to if loadRemoteMedia, there are two video streams on refresh from video guy being created so that's why the nonvideo guy picks it up after the refresh from a video guy. It doesn't pick up the first one, but it picks up the second one or vice versa. But if video guy comes first, the window shows up, but still no video even though peer_video is true for a video guy. So the conditions to load remote media needs to be reworked to take out the duplicate video stream. Or figure out why two video streams are being created in the first place.

[miroslavpejic85]

You are on the right direction. They shouldn't be dual video streams (event.track.kind), probably one is video and the other is audio ... That event is triggered by this method.

async function handleAddTracks(peer_id) {
    localMediaStream.getTracks().forEach((track) => {
        peerConnections[peer_id].addTrack(track, localMediaStream);
    });
}

This could be a partial fix, just need to understand why a video guy sent first audio stream and not video here...

async function handleOnTrack(peer_id, peers) {
    console.log('[ON TRACK] - peer_id', { peer_id: peer_id });
    peerConnections[peer_id].ontrack = (event) => {
        let remoteVideoStream = getId(peer_id + '_video');
        let peer_name = peers[peer_id]['peer_name'];
        let kind = event.track.kind;
        userLog('info', '[ON TRACK] - peer_name: ' + peer_name + ' kind: ' + kind);
        console.log('[ON TRACK] - info', { peer_id: peer_id, peer_name: peer_name, kind: kind, track: event.track });
        if (event.streams && event.streams[0]) {
            console.log('[ON TRACK] - peers', peers);
            remoteVideoStream
                ? attachMediaStream(remoteVideoStream, event.streams[0])
                : loadRemoteMediaStream(event.streams[0], peers, peer_id);
        } else {
            console.log('[ON TRACK] - SCREEN SHARING', { peer_id: peer_id, peer_name: peer_name, kind: kind });
            // attach newStream with screen share video and audio already existing
            let inboundStream = new MediaStream([remoteVideoStream.srcObject.getAudioTracks()[0]]);
            inboundStream.addTrack(event.track);
            attachMediaStream(remoteVideoStream, inboundStream);
        }
    };
}

I'm sure you will find the right solution :)

[gettyhub]

Sometimes the audio is tied to the same device as the video (embedded built in webcam) and other times the audio comes from a plugged in USB or traditional microphone separately. The event.track.kind seems to distinguish which device is which.

Seems like something called nested or completely outside of OnTrack is deciding the way the peers are being handled.

[gettyhub]

Okay so its calling the first video guy an audio target (before the nonvideo guy). But then the next video guy that comes along is a video target (after the nonvideo guy). Almost like I know you are but what am I? I'm a nonvideo, so that makes the first target audio. A nonvideo guy and a nonvideo guy show up just fine.

[gettyhub]

Addtracks is making separate streams for audio and video on a video guy and only audio on a nonvideo guy.

[gettyhub]

Whatever you sent me is not posted here for some reason, but it threw an unhandled exception.

Type error: failed to execute addtrack on rtcpeerconnection parameter 1 is not of type mediastreamtrack

Maybe instead of null it should be MediaStream, but that doesn't work either.

Addtracks is also calling the first video guy an audio track.

[gettyhub]

Plugging in another camera and switching to it is enough to call refreshmedia and allows the videostream to come through to the nonvideo guy.

[miroslavpejic85]

Excellent detective work, I'm sure you will find the definitive solution, keep it up and, Thank you so much for your time and contribution to solve it!

[gettyhub]

If I go to above the peers tile and hover over the video icon it correctly shows the status of whether video is being used or if it has been turned on/off. But of course this is since we have changed the code so that the video guy even shows up in a tile to the nonvideo guy whereas before it didn't even show up.

Most likely most mobile devices today have an embedded camera, so the most likely people to suffer with this problem is the ones who have desktops which are going to be more comfortable for those working from home. The desktop guys will most likely have to use a USB webcam. But how the function of whether the nonvideo guy doesn't see the video guy seems to somehow be based on whether I have a camera. But another interesting point, is if I am the first guy on, in peer_count, shouldn't that be 0? There are no peers, I am just doing a selfie cam until someone else arrives.

[gettyhub]

I can't get it to refresh my stream even if I call refreshLocalMedia or refreshMyStreamToPeers unless I am trying to put it in the wrong place to make it activate.

[gettyhub]

I don't know if I really changed anything significant, but the next video guy that comes along can get all the video streams to show up that were previously not seen by nonvideo guys. So any video guy can pop it up not just the first one that didn't show or was before a nonvideo guy. Maybe something on the server side is wrong. Was able to change peer_video to false in join server which reflects correctly on a nonvideo guy and toggling shows up correctly but still no video feed. This has no effect to set this to false if somebody does actually have a video feed. The video feed still comes through.

I had to put both refreshLocalMedia and refreshMyStreamToPeers in loadRemoteMediaStream to get that to work that way. I need a better trigger to do that with.

[miroslavpejic85]

Hi, As a P2P, The count equal on how many peers you are connected to.

refreshMyLocalStream - refreshMyStreamToPeers is ok, if your video-audio stream changes, for example if you change webcam from the settings or if you swap camera on mobile and so on.

PS: I pushed a partial fix and added some additional logs in the console, so you can continue to indagate as well for a definitive fix. Thank you again for your time!

[gettyhub]

Haven't pulled new commit yet, but stopping systemctl, so we can npm start so we can read console log from the server. It shows that peer_video on both is false. Even on refresh, when the video does show up to the nonvideo guy, it still shows peer_video is false.

Also emitpeerstatus shows only an audio element as false initially followed by another emitpeerstatus to audio being true. No mention of video.

[gettyhub]

Now on the newest commit, server console correctly showing peer_video result. On either commit, video guy shows white text for video guy, while nonvideo guy shows single quotes and green text just like the room name or peer_id. This may just be a type in server.js that causes this difference.

If nonvideo guy is on first, a still clip on the video guy shows up on the nonvideo guy. Still no video as usual on the other way.

[gettyhub]

OnTrack still shows audio for a video guy on a nonvideoguy.

[gettyhub]

Just stopping the server with clients still connected, drops everyone of course. On reconnect,, the video guy shows up, but the nonvideo guy doesn't. Local feed is still showing on video guys screen.

[miroslavpejic85]

Should everything be fixed with the latest commit: https://github.com/miroslavpejic85/mirotalk/commit/a0bad8e91dab59abf3a64b8c6bad611f150eed5e Can you check please :)

[gettyhub]

Should everything be fixed with the latest commit: a0bad8e Can you check please :)

Holy cow batman! It looks like it might accidently be working.

The video is fine on the computer with the cam, but backwards remotely (horizontal flip). Maybe that's the way it's supposed to be?

Haven't tried the mirotalksfu or mirotalkp2p, but maybe has similar code/bugs?

[miroslavpejic85]

Hi @gettyhub,

Not a bug, it has to be, you must see yourself as if you were in front the mirror, while the participants see you as if you were in front of them. If you put a text file in front of the camera, you will see that the participants will read it correctly :)

So we can close this task, do you do it?

PS: You can try it in google-meet, you will always see yourself as if you were facing the mirror. Thank you.

[gettyhub]

Hi @gettyhub,

Not a bug, it has to be, you must see yourself as if you were in front the mirror, while the participants see you as if you were in front of them. If you put a text file in front of the camera, you will see that the participants will read it correctly :)

So we can close this task, do you do it?

PS: You can try it in google-meet, you will always see yourself as if you were facing the mirror. Thank you.

Looking through the commit, I probably wouldn't have gotten it fixed or it would've taken a long time.

[miroslavpejic85]

It wasn't easy. It also took me a while to get there with a lot of debugging :) Thanks anyway for your contribution. I wish you all the best! BR. Miroslav