WASM decoder can get stuck on didDecode loop, never drawing anything

[humphd]

Sometimes the WASM decoder seems to get stuck on startup. I'm streaming via a WebSocket, and I can see the data pouring in, however, the canvas never gets updated. In the debugger I can see that this.functions._mpeg1_decoder_decode always returns false, so it bails on every call to decode(). I can't debug it further, as the rest is WASM.

One or two reloads will often get it to work again, so it's not the content of the network stream itself.

MPEG1WASM.prototype.decode = function() {
    var startTime = JSMpeg.Now();

    if (!this.decoder) {
        return false;
    }

    var didDecode = this.functions._mpeg1_decoder_decode(this.decoder);
    if (!didDecode) {
        return false;          // <--- when it gets stuck, it never gets beyond this...
    }

    // Invoke decode callbacks
    if (this.destination) {
        var ptrY = this.functions._mpeg1_decoder_get_y_ptr(this.decoder),
            ptrCr = this.functions._mpeg1_decoder_get_cr_ptr(this.decoder),
            ptrCb = this.functions._mpeg1_decoder_get_cb_ptr(this.decoder);

        var dy = this.instance.heapU8.subarray(ptrY, ptrY + this.codedSize);
        var dcr = this.instance.heapU8.subarray(ptrCr, ptrCr + (this.codedSize >> 2));
        var dcb = this.instance.heapU8.subarray(ptrCb, ptrCb + (this.codedSize >> 2));

        this.destination.render(dy, dcr, dcb, false);
    }

    this.advanceDecodedTime(1/this.frameRate);

    var elapsedTime = JSMpeg.Now() - startTime;
    if (this.onDecodeCallback) {
        this.onDecodeCallback(this, elapsedTime);
    }
    return true;
};

[humphd]

Disabling WASM seems to fix it as well.

[phoboslab]

Hard to say what's going on. My hunch is, that it's aborting here already. Can you check if loadSequenceHeader() is called and we get the right width/height and framerate?

I would have assumed that it is a problem with the demuxer, not finding packet starts and therefore not feeding the WASM video decoder, but then the JS video decoder would fail in the same way...

I can dig into this early next week. Can you share some details about how you send the WebSocket messages, so I can try to reproduce it?

[humphd]

Thanks for your help. I did some more debugging, and here is what I'm seeing.

I can make this happen 1 in every ~15 reloads in both Safari or Chrome on macOS 12.2.1 with an M1 Pro (seems to happen more often in Chrome).

The project isn't open source, but I've got two mjpeg cameras (704 x 480) that I'm converting to MPEG1 streams like so:

$ ffmpeg -r 6 -f mjpeg -i http://192.168.2.139/mjpg/1/video.mjpg -an \
  -f mpegts -c:v mpeg1video -bf 0 -q 4 -r 24 pipe:1

I pipe these into a node app and do something similar to your WebSocket relay, writing the chunks to each connected socket:

function broadcast(data) {
  wss.clients.forEach((client) => {
    if (client.readyState === WebSocket.OPEN) {
      client.send(data, { binary: true });
    }
  });
}

ffmpeg.stdout.on('data', broadcast);

In the browser, I'm wrapping jsmpeg with React, and showing two Camera components like this:

export default function Camera({ src }) {
  const canvasElemRef = useRef();

  useEffect(() => {
    const canvasElem = canvasElemRef.current;
    if (!canvasElem) {
      return;
    }

    let player = new window.JSMpeg.Player(src, {
      canvas: canvasElem,
      audio: false,
      // jsmpeg doesn't like when the gl context gets destroyed by React
      // https://github.com/phoboslab/jsmpeg/issues/392
      disableGl: true,
    });

    return () => {
      if (player) {
        player.destroy();
        player = null;
      }
    };
  }, [src, canvasElemRef]);

  return (
    <CardMedia
      component="canvas"
      ref={canvasElemRef}
      width={704}
      height={480}
    />
  );
}

I can always see the data coming over the two socket connections, so I don't think it's the network (I also note that a browser refresh can solve it, implying that the stream is still fine).

When it works, loadSequnceHeader() (typo in the code, missing an 'e') is called, and the right frameRate, w, and h are parsed.

When it fails, this function isn't called. As you suggested, it dies here:

  MPEG1WASM.prototype.write = function (pts, buffers) {
    JSMpeg.Decoder.Base.prototype.write.call(this, pts, buffers);
    if (
      // this.hasSequenceHeader=false
      !this.hasSequenceHeader &&
      // _mpeg1_decoder_has_sequence_header(this.decoder)=0
      this.functions._mpeg1_decoder_has_sequence_header(this.decoder) 
    ) {
      this.loadSequnceHeader();
    }
  };

So MPEG1WASM.prototype.write() is being called over and over, but it never goes into this.loadSequnceHeader().

When I disable WASM, it always goes through MPEG1.prototype.decodeSequenceHeader for the same network stream (I don't restart it, so nothing has changed), and gets the correct frameRate, width, and height.

Originally I suspected it was some stupidity with React tearing down the connecting or something, but I'm able to reproduce it with:

<canvas id="video-canvas-1" width="704" height="480"></canvas>
<canvas id="video-canvas-2" width="704" height="480"></canvas>
<script type="text/javascript" src="jsmpeg.min.js"></script>
<script type="text/javascript">
  function camera(num) {
    const canvas = document.getElementById(`video-canvas-${num}`);
    const url = `ws://${document.location.hostname}:3012/api/v3/ws/camera${num}`;
    return new JSMpeg.Player(url, { canvas, audio: false });
  }

  camera(1);
  camera(2);
</script>

Let me know if there's more I can try or provide you.

[phoboslab]

That took me a while to find!

When creating multiple JSMpeg.Player instances, JSMpeg tries to re-use the same WASM Module instance, so it doesn't have to be re-compiled.

However, there was a race condition that created the Module instance twice (or more times). All buffer writes would then go into the memory of the wrong instance. The WASM mpeg1 functions in turn would read from a buffer that was never written to.

Thanks for reporting!

[humphd]

Fantastic, thank you for spending the time to figure it out!