Smooth FPS

[matijagaspar]

@waclaw66 Suggested a enhancement for smooth rendering of frames following the predefined "frames per second" configuration.

Rough example bellow. It works quite fluently. Precise rendering could be optional and fps could be passed as a parameter together with init message payload.

var wsavc = new WSAvcPlayer.default({useWorker: false});
document.getElementById('webcam_box').appendChild(wsavc.AvcPlayer.canvas);
wsavc.AvcPlayer.render = false;
var frames = [];
var fps = 20;
var renderStarted = false;
var nexttime = null;

function render(now) {
  if (!nexttime) {
      nexttime = now;
  }
  if (now >= nexttime)
  {
      frame = frames.shift();
      if (frame) {
          wsavc.AvcPlayer.renderFrame(frame);
          nexttime += 1000 / fps;
      }
  }
  if (renderStarted) {
      requestAnimationFrame(render);
  }
}

wsavc.AvcPlayer.onPictureDecoded = function (buffer, width, height, infos) {
  frames.push({
      canvasObj: this.canvasObj,
      data: Uint8Array.from(buffer),
      width: width,
      height: height,
  });
  if (!renderStarted) {
      renderStarted = true;
      nexttime = null;
      requestAnimationFrame(render);
  }
}

wsavc.connect("wss://example.com");

Originally posted by @waclaw66 in https://github.com/matijagaspar/ws-avc-player/issues/9#issuecomment-560337583

This could be added to the player provided it is optional. The playback will be smoother but it can induce slight delay.

[waclaw66]

I have tested it on some less stable networks and added a little enhancement with automatic buffer. Playback could be delayed up to maxbuffertime, but the smoothness worth for it in some scenarios.

var frametime = 1000 / fps;
var buffertime = 0;
var maxbuffertime = 1000;

function render(now) {
    if (!nexttime) {
        nexttime = now + buffertime;
    }
    if (now >= nexttime)
    {
        frame = frames.shift();
        if (frame) {
            wsavc.AvcPlayer.renderFrame(frame);
            nexttime += frametime;
        } else {
            if (buffertime < maxbuffertime) {
                buffertime += frametime;
                nexttime += frametime;
            }
        }
    }
    if (renderStarted) {
        requestAnimationFrame(render);
    }
}

[matijagaspar]

Great work, however after thinking some more, about it. I think it should be modified a bit, so instead of having a fixed framerate defined on the client, server would collect a timestamp for each frame and add it to the packet. Than use that timestamp for accurate timing. There are 2 reasons for this approach:

• The timing is embeded in the stream, so it will be simpler to handle different frame rates on the client
• It will be a good starting point for implementing audio-video sync like is done on WebRTC's Lip sync

Enabling/disabling the feature could be done on ws connect, since some sort of handshake will be inevitable anyway.