Strange duration when inside web worker

[KaliaJS]

Hi,

The muxer works great outside the webworker but when I put it inside a webworker the video duration is really weird.

Once the page is loaded, if I wait 10s before starting recording, the video duration will be 10s + recorded video time. The second strange thing is that the video in the player will start at 10s (and not 0s) but impossible to return before 10s.

And if I record a new video after the other video, say after 60s, the video when finished recording will be 64s, etc.

When I reload the page the "bug" starts from zero but increases according to the time I stay on the page.

After trying for days and days, reading all the documents on the subject and trying all possible examples, believing I was doing something wrong, I tried the webm-writer library modified by the WebCodecs team [example](https://github.com/w3c/webcodecs/tree/704c167b81876f48d448a38fe47a3de4bad8bae1/ samples/capture-to-file) and everything works normally.

Do you have any idea what the problem is or am I doing something wrong?

Some exemple code

  function start() {
    const [ track ] = stream.value.getTracks()
    const trackSettings = track.getSettings()
    const processor = new MediaStreamTrackProcessor(track)
    inputStream = processor.readable

    worker.postMessage({
      type: 'start',
      config: {
        trackSettings,
        codec,
        framerate,
        bitrate,
      },
      stream: inputStream
    }, [ inputStream ])

    isRecording.value = true

    stopped = new Promise((resolve, reject) => {
      worker.onmessage = ({data: buffer}) => {
        const blob = new Blob([buffer], { type: mimeType })
        worker.terminate()
        resolve(blob)
      }
    })
  }

Worker.js

import '@workers/webm-writer'

let muxer
let frameReader

self.onmessage = ({data}) => {
  switch (data.type) {
    case 'start': start(data); break;
    case 'stop': stop(); break;
  }
}

async function start({ stream, config }) {
  let encoder
  let frameCounter = 0

  muxer = new WebMWriter({
    codec: 'VP9',
    width: config.trackSettings.width,
    height: config.trackSettings.height
  })

  frameReader = stream.getReader()

  encoder = new VideoEncoder({
    output: chunk => muxer.addFrame(chunk),
    error: ({message}) => stop()
  })

  const encoderConfig = {
    codec: config.codec.encoder,
    width: config.trackSettings.width,
    height: config.trackSettings.height,
    bitrate: config.bitrate,
    avc: { format: "annexb" },
    framerate: config.framerate,
    latencyMode: 'quality',
    bitrateMode: 'constant',
  }

  const encoderSupport = await VideoEncoder.isConfigSupported(encoderConfig)
  if (encoderSupport.supported) {
    console.log('Encoder successfully configured:', encoderSupport.config)
    encoder.configure(encoderSupport.config)
  } else {
    console.log('Config not supported:', encoderSupport.config)
  }

  frameReader.read().then(async function processFrame({ done, value }) {
    let frame = value

    if (done) {
      await encoder.flush()
      const buffer = muxer.complete()
      postMessage(buffer)
      encoder.close()
      return
    }

    if (encoder.encodeQueueSize <= config.framerate) {
      if (++frameCounter % 20 == 0) {
        console.log(frameCounter + ' frames processed');
      }
      const insert_keyframe = (frameCounter % 150) == 0
      encoder.encode(frame, { keyFrame: insert_keyframe })
    }

    frame.close()
    frameReader.read().then(processFrame)
  })
}

async function stop() {
  await frameReader.cancel()
  const buffer = await muxer.complete()
  postMessage(buffer)
  frameReader = null
}

Screenshot

Video sample 1

Video sample 2

[Vanilagy]

Hi!

What's likely happening is that you're writing the unmodified EncodedVideoChunks spit out by your VideoEncoder into the muxer. This is usually fine for cases where you self-construct your VideoFrames, but here, they're coming from a media source. These frames have attached to themselves a timestamp which is usually relative to the age of your document. WebMMuxer will take these timestamps and write them into the file, verbatim, which is not what you want in this case.

What you'll want to do is offset these timestamps so that the first EncodedVideoChunk's timestamp is 0, and then rises naturally from there. I have example logic for this in my demo:

audioEncoder = new AudioEncoder({
    output(chunk, meta) {
        if (firstAudioTimestamp === null) firstAudioTimestamp = chunk.timestamp; // <-- THIS PART !!
        muxer.addAudioChunk(chunk, meta, chunk.timestamp - firstAudioTimestamp); // 3rd param is timestamp
    },
    error: e => console.error(e)
});     

You simply need to use the first chunk's timestamp as the offset for all subsequent chunks - this way, your video will start at 0 but then continue on in the same rate in which you recorded it.

Hope this helps! I agree, if you haven't read through my demo thoroughly, this behavior might be unexpected and leave you in the dark. I'll probably add an error/warning when the first chunk that you pass does not start at 0, because it's rarely what you want.

[Vanilagy]

Dropped a new version which forces explicit handling of this behavior. Please try it out (v2.0.0), and tell me if it addresses / fixes your issue.

[KaliaJS]

@Vanilagy It actually makes so much sense now! It works perfectly with . I feel silly for not watching the audio part of your demo (that I've watched so many times...). I said to myself "it has nothing to do with it, you can avoid reading this part" 🤭

Thank you for being quick !