[1.x] Add background with janus.plugin.videoroom

[alivadjid]

What version of Janus is this happening on? Put the version and the commit identifier (available in version.c or from an info request) here version : 1102 Have you tested a more recent version of Janus too? Yes/no. If yes, which one. If not, why.

Was this working before? If you have information on a version/commit where the issue wasn't there (e.g., the result of a git bisect), please provide it here.

Is there a gdb or libasan trace of the issue? If you have a trace related to the issue (e.g., after a segfault), provide it as a gist/pastebin/other link here, or as a collapsible section. Do NOT paste logs inline. For more information on the kind of traces we may find useful, please refer to the documentation.

Additional context Add any other context about the problem here.

---

Thank's a lot for usefull library. Greatfull! I need information.

I'm usingjanus.plugin.videoroom and adding background with @mediapipe/selfie_segmentation

function initRTC() {
  Janus.init({
    dependencies: Janus.useDefaultDependencies({ adapter }),
    callback: () => {
      janusInstance = new Janus({
        server: url,
        success: () => {
          janusInstance.attach?.({
            plugin: 'janus.plugin.videoroom',
            opaqueId: `user_${user.id}`,
            success: (plugin) => createRoom(),
            onmessage: (message, jsep) => handleMessage(message, jsep),
            onlocaltrack: (track) => handleLocalTrack(track),
            error: (error) => handleError(error),
          });
        },
        error: (error: string) => handleError(error),
      });
    },
  });
}

after join to room, on handlemessage make createOffer in publishOwnFeed function. But I modified this callbacks with adding custom function addBackgound.

function handleMessage(message) {
  if (message.videoroom) handleEvent(message);
}

function handleEvent(message) {
  if (message.videoroom === 'joined') {
      publishOwnFeed();
    }
}

function publishOwnFeed() {
  pluginHandle?.createOffer({
    tracks: [
      { type: 'audio', capture: true, recv: false },
      { type: 'video', capture: true, recv: false },
    ],

    success: (jsep) => {
      pluginHandle?.send({
        message: {
          request: 'configure',
          audio: true,
          video: true,
        },
        jsep,
      });
    },

    // get client stream, modify and return it
    addBackground: (stream) => {
      const videoElement = document.createElement('video');
      const newStream = new MediaStream(stream).clone();
      const audioTracks = newStream.getAudioTracks();

      if (audioTracks[0]) {
        audioTracks[0].enabled = false;
      }

      videoElement.autoplay = true;
      videoElement.playsInline = true;
      videoElement.srcObject = newStream;

      let lastTime = new Date().getTime();

      async function getFrames() {
        const now = videoElement?.currentTime;

        if (now > lastTime) await selfieSegmentation.send({ image: videoElement });
        lastTime = now;
        requestAnimationFrame(getFrames);
      }
      getFrames();

      const canvasStream = canvas.captureStream();

      return canvasStream;
    },
  });
}

const selfieSegmentation = new SelfieSegmentation({
  locateFile: (file) => {
    if (file.endsWith('.tflite')) {
      return tflite;
    } else if (file.endsWith('wasm_bin.js')) {
      return binJs;
    } else if (file.endsWith('.binarypb')) {
      return binarypb;
    } else if (file.endsWith('.wasm')) {
      return selfieWasm;
    }

    return ``;
  },
});

selfieSegmentation.setOptions({
  modelSelection: 1,
});
selfieSegmentation.onResults(handleSegmentationResults);

And this addBackground function I used in janus.js in here: https://github.com/meetecho/janus-gateway/blob/15ef4d6cde38ea8d441f7e27eba298265c2a7358/html/janus.js#L2454C5-L2454C5

I rewrite stream with results from addBacgkround

else if (track.capture) {
        if (track.gumGroup && groups[track.gumGroup] && groups[track.gumGroup].stream) {
          // We did a getUserMedia before already
          let stream = groups[track.gumGroup].stream;

          /**
           * UPDATES: Get user stream and redefine stream with background
           *
           */
          if (track.type === 'video') {
            let canvasStream = callbacks.addBackground?.(stream);

            canvasStream?.getTracks().forEach((track) => {
              const updateStream = new MediaStream([track]);
              stream = updateStream;
            });
          }

          /**
           * UPDATES: Get user stream and redefine stream with background
           *
           */

          nt = track.type === 'audio' ? stream.getAudioTracks()[0] : stream.getVideoTracks()[0];
          delete groups[track.gumGroup].stream;
          delete groups[track.gumGroup];
          delete track.gumGroup;
        } else if (track.capture instanceof MediaStreamTrack) {

And this solution is work. I redefine stream and add background here. But the key question is: Is this solution correct? I mean is this a correct place to modify stream. Maybe I missed something and in janus I can make it with more correctfull way.

P.S. One minus I have, which I'm trying to solve. After using SelfieSegmenation function and import .wasm, .tflite, .binarypb, wasm_bin.js web browser memory usage increases. And It can be explained with calculating on browser. But without this normal memory usage - 130Mb, with this - jumps from 200Mb to 1Gb(2Gb) and down.

[lminiero]

Please use the group for questions like this. We only use github for code issues, and demos are not part of the Janus codebase.