philnash / twilio-video-react-hooks

A video chat application built with Twilio Video and React Hooks
MIT License
110 stars 61 forks source link

Help needed - Connecting meeting without audio and Video. Later I create audio track and video track for the local participant. It won't render the audio/video through the audio/video element in the participant component. #35

Closed bijmon closed 4 years ago

bijmon commented 4 years ago

I have a requirement not to connect audio and video while connecting to the room. So I passed them as false while connecting to the room. Later when I create audio or video track and publish them to the local participant the local participant's component will render nut all those useeffects won't get fired that will end in not associating the video/audio tracks with audio/video html elements. Please help to fix it. My Prod live date is getting closer.

philnash commented 4 years ago

Hey @bijmon,

You're right, the useEffect for tracks won't run because the event for a local participant publishing a new track isn't fired (and localparticpants don't fire the trackSubscribed event for their own tracks).

Try adding listeners for trackPublished events to the Participant component, something like this:

import { LocalParticipant } from "twilio-video";
import React, { useState, useEffect, useRef } from "react";

const Participant = ({ participant }) => {
  // other stuff

  useEffect(() => {
    setVideoTracks(trackpubsToTracks(participant.videoTracks));
    setAudioTracks(trackpubsToTracks(participant.audioTracks));

    const trackSubscribed = (track) => {
      if (track.kind === "video") {
        setVideoTracks((videoTracks) => [...videoTracks, track]);
      } else if (track.kind === "audio") {
        setAudioTracks((audioTracks) => [...audioTracks, track]);
      }
    };

    const trackUnsubscribed = (track) => {
      if (track.kind === "video") {
        setVideoTracks((videoTracks) => videoTracks.filter((v) => v !== track));
      } else if (track.kind === "audio") {
        setAudioTracks((audioTracks) => audioTracks.filter((a) => a !== track));
      }
    };

    participant.on("trackSubscribed", trackSubscribed);
    participant.on("trackUnsubscribed", trackUnsubscribed);

    if (participant instanceof LocalParticipant) {
      const trackPublished = (trackPublication) => {
        if (trackPublication.kind === "video") {
          setVideoTracks((videoTracks) => [...videoTracks, trackPublication.track]);
        } else if (trackPublication.kind === "audio") {
          setAudioTracks((audioTracks) => [...audioTracks, trackPublication.track]);
        }
      }
      participant.on("trackPublished", trackPublished)
      // and do the same for trackUnpublished
    }

    return () => {
      setVideoTracks([]);
      setAudioTracks([]);
      participant.removeAllListeners();
    };
  }, [participant]);

That way, the local participant will listen for events and add the new video and audio tracks to their state, triggering the other useEffects.

Let me know how you get on with that.

bijmon commented 4 years ago

@philnash Thank you so much for your quick response. What I noticed is that Initially when I connect to a room and render the local participant it will go through all the use effects. But there won't be any tracks. So Video/Audio html elements won't get associated with them. Later as per my need when I create video track as below and attach it to local participant it will go inside participant component but it will not go under any useffects. May be because the local participant a nested object or something. Do you know the cause?

  if(room.localParticipant.videoTracks.size === 0){
    createLocalVideoTrack().then(localVideoTrack => {
      return room.localParticipant.publishTrack(localVideoTrack);
    }).then(() => {
      console.log('Video Track added since Track was not available');
    });
  }

I will try your trick. It should work.

philnash commented 4 years ago

The useEffects don't run again because nothing changes that would cause them to.

In the largest useEffect in the Participant component, the only thing that would make the effect change is a change of participant because the second argument to the useEffect is [participant]. When a participant publishes a new track, that doesn't change the participant object itself.

So, what I suggest is to listen for those trackPublished events within that useEffect and they will trigger the other useEffects that depend on the videoTracks or audioTracks arrays.

Let me know how you get on with it.

bijmon commented 4 years ago

@philnash That worked fine. Thanks for your help. The condition I had to tweak little bit as (participant.constructor.name === 'LocalParticipant'). Because LocalParticipant won't be available in the participant component.

But I always thought after publishing tracks to room.localParticipant the localParticipant object changes. Because I see the video tracks and audio tracks in the localparticipant nested object. Somehow it won't invoke the useEffects in the participant component even there is change in it.

philnash commented 4 years ago

@philnash That worked fine. Thanks for your help. The condition I had to tweak little bit as (participant.constructor.name === 'LocalParticipant'). Because LocalParticipant won't be available in the participant component.

Ah, yes, I did throw in:

import { LocalParticipant } from "twilio-video";

at the top, but that was probably not obvious, sorry.

But I always thought after publishing tracks to room.localParticipant the localParticipant object changes. Because I see the video tracks and audio tracks in the localparticipant nested object. Somehow it won't invoke the useEffects in the participant component even there is change in it.

I think that's the point, React doesn't do a deep object comparison for useEffect dependencies. Here's an article with some other suggestions to check an object like that, but I would avoid doing that kind of comparison as it would likely be slow.

Instead, listening for the events the participant emits and updating state based on the changes will work better. And it's how the Video SDK is designed to work. Check out the events list in the JS docs (in the top navigation) to see all of the events the various objects emit and how to use them.

bijmon commented 4 years ago

Got it. Thanks for all your help. I am using this repo for one of my projects. So far so good. Thanks for it.

philnash commented 4 years ago

Awesome, glad it's working out! I'll close this issue now, but let me know if you have any other questions.

anagharm commented 4 years ago

Hi, I am also trying to implement the same. But as I am new to react not able to get it. Can you provide the updated file.

Thanks in advance

philnash commented 4 years ago

I haven't written this all out myself. Did you try following the instructions from this conversation? Is there something you are trying that isn't working?

anagharm commented 4 years ago

Wow.. Very quick reply...

Thanks it worked !!! Thank a lot.

philnash commented 4 years ago

I happened to be looking at my email when your comment came in! Just happened again!

Glad you got it working. Thanks for letting me know 🙂