"Dummy" video stops audio

[horvbalint]

Hello guys,

I am creating an app, with video call functionalities for a while now and I want to remove the need of renegotiation when video/desktop sharing is toggled. So decided to use replaceTrack() to achive it. It works beautifully when I get audio and video from user, and then replace the videotrack with a screenCapture. But. I do not want to ask for camera permission if the user makes an audio call, to immediatly disable it and only use it to have something to replace. So I began to use dummy tracks, tried it with canvas capture track and with new PeerConnection.addTransceiver().sender.track to add it to the local stream (with only one audio track) before making the call, but if I dont use a real cam video track, neither the audio nor the video arrives to the other side. Better said it arrives but contains nothing, can't display it in a video tag.

I really don't know what can be wrong here, can you please help me with some tipps or have you came across this problem before?

Thank you in advance!

( Or is there a way, to add a video transceiver to the connection before peer.call() creates the offer? )

[theevann]

@balintx99 Hello, I would be interested in knowing how you were able to use the replaceTrack function ?

---

Update:

I manage to do what you described using a canvas. I had to draw on the canvas before getting the canvas stream and after setting the call to get it working.

Example:

draw();  // Draw before canvas.captureStream()

var canvas = document.querySelector('canvas');
var canvas_stream = canvas.captureStream(25);
var canvas_track = stream.getTracks()[0]

let local_stream = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: false
});
local_stream.addTrack(canvas_track)

let my_call = peer.call(remote_peer_key, local_stream);

my_call.on('stream', function (remote_stream) {
    let video_remote = document.querySelector("#videoRemote");
    video_remote.srcObject = remote_stream;
    draw(); // Draw after stream was sent
    setup_audio(my_call);
});

async function setup_audio(call) {
    let senders = call.peerConnection.getSenders();
    let audio_stream = await navigator.mediaDevices.getUserMedia({
        audio: true
    });
    let audio_track = audio_stream.getAudioTracks()[0];
    senders[0].replaceTrack(audio_track);
}

async function setup_video(call) {
    let senders = call.peerConnection.getSenders();
    let video_stream = await navigator.mediaDevices.getUserMedia({
        video: true
    });
    let video_track = video_stream.getVideoTracks()[0];
    senders[1].replaceTrack(video_track);
}

async function setup_screen(call) {
    let senders = call.peerConnection.getSenders();
    let screen_stream = await navigator.mediaDevices.getDisplayMedia({
        video: true
    });
    let video_track = screen_stream.getVideoTracks()[0];
    senders[1].replaceTrack(video_track);
}

function draw() {
    var canvas = document.getElementById('canvas');
    if (canvas.getContext) {
        var ctx = canvas.getContext('2d');
        ctx.fillRect(10, 10, 0, 0);
    }
}

[vkatsar]

@theevann your example didn't work for me.

This is my draw function that works. draw () { let c = document.getElementById('canvas'), ctx = c.getContext('2d'), cw = c.width = 400, ch = c.height = 300, circle = { x: (cw / 2) + 5, y: (ch / 2) + 22, radius: 90, speed: 2, rotation: 0, angleStart: 270, angleEnd: 90, hue: 220, thickness: 18, blur: 25 }, updateCircle = function(){ if(circle.rotation < 360){ circle.rotation += circle.speed; } else { circle.rotation = 0; } }, clear = function(){ ctx.globalCompositeOperation = 'destination-out'; ctx.fillStyle = 'rgba(0, 0, 0, .1)'; ctx.fillRect(0, 0, cw, ch); ctx.globalCompositeOperation = 'lighter'; }, loop = function(){ clear(); updateCircle(); } /* Loop It, Loop It Good */ setInterval(loop, 1000); }

[theevann]

@vkatsar It indeed is not working all the time. In my case, I had to call draw two or three times. This was also varying depending on the browser.

I now use a slightly different draw function:

function draw() {
    var canvas = document.getElementById('canvas');
    if (canvas.getContext) {
        var ctx = canvas.getContext('2d');
        ctx.clearRect(0, 0, canvas.width, canvas.height);
        ctx.fillRect(0, 0, 0, 0);
    }
}

And I call it twice in the on("stream") with :

setTimeout(draw, 500);
setTimeout(draw, 1000);

That is working for me in chrome and firefox.

[horvbalint]

Thank you very much for your answers!

Drawing multiple times on the canvas before sending the stream is something I have never thougt it would help.

I will try it as soon as I can, and update this answer with the results.

[noblessem]

Thank you very much for your answers!

Drawing multiple times on the canvas before sending the stream is something I have never thougt it would help.

I will try it as soon as I can, and update this answer with the results.

Where is ur update my friend?

[horvbalint]

@noblessem Oh well, I stopped that side project soon after this comment :/ As far as I remember drawing multiple times did work, but it did sound like a very hacky solution, so when I started a very similar project ~ a year ago I went with SimplePeer instead of Peer.js which seemed to be more actively developed at the time.

I hope someone can help you if you are facing this problem.

[noblessem]

@noblessem

Oh well, I stopped that side project soon after this comment :/ As far as I remember drawing multiple times did work, but it did sound like a very hacky solution, so when I started a very similar project ~ a year ago I went with SimplePeer instead of Peer.js which seemed to be more actively developed at the time.

I hope someone can help you if you are facing this problem.

Thx for ur answer, will try it out

[afrokick]

I think it should be added to FAQ, but now lets provide our boilerplate(TypeScript):

type Size = { width: number; height: number };

export const createEmptyAudioTrack = () => {
  const ctx = new AudioContext();
  const oscillator = ctx.createOscillator();

  const dst = oscillator.connect(ctx.createMediaStreamDestination());

  const track = dst.stream.getAudioTracks()[0];

  return Object.assign(track, { enabled: false });
};

export const createEmptyVideoTrack = ({ width, height }: Size) => {
  const canvas = Object.assign(document.createElement('canvas'), { width, height });
  canvas.getContext('2d')!.fillRect(0, 0, width, height);

  const stream = canvas.captureStream();
  const track = stream.getVideoTracks()[0];

  return Object.assign(track, { enabled: false });
};

//we can reuse audio track
let cachedAudioTrack: AudioTrack | undefined;

export const createEmptyMediaStream = ({
  videoSize,
}: {
  videoSize: Size;
}): MediaStream => {
  if (!cachedAudioTrack) {
    cachedAudioTrack = createEmptyAudioTrack();
  }

  const audioTrack = cachedAudioTrack;
  const videoTrack = createEmptyVideoTrack(videoSize);

  return new MediaStream([audioTrack, videoTrack]);
};

And usage:

const videoSize = { width: 640, height: 480 };

const localStream = createEmptyMediaStream({ videoSize });

const call = peer.call(id, localStream);

[laxyharaxy]

Try this solution it works : #147