VideoFrame.timestamp and EncodedVideoChunk.timestamp

[aboba]

In Section 9.4.3, the definition of VideoFrame.timestamp says:

"The presentation timestamp, given in microseconds. The timestamp is copied from the EncodedVideoChunk corresponding to this VideoFrame."

This implies that when decoding an EncodedVideoChunk, we should expect EncodedVideoChunk.timestamp to be copied into VideoFrame.timestamp, and indeed this does happen.

Is there is a corresponding expectation when encoding?

When the the Webcodecs encode/decode sample code was written, the value of VideoFrame.timestamp was copied into EncodedVideoChunk.timestamp when encoding. So the sample used the timestamp attribute to link a VideoFrame provided to the encoder to the corresponding output EncodedVideochunk, in order to estimate the encoding time.

However, in recent Chromium builds, the behavior has changed. If timestamp no longer links a VideoFrame to the EncodedVideoChunk(s) produced by an encoder, is there some other metadata that serves the same purpose?

[padenot]

I think there's an expectation that authors can match those. If it's not explicit, it should be.

While implementing, this is generally true for the underlying platform API, except some times on Windows. We've had to implement our own mapping in the Windows backend of encoders (using WMF) so that the guarantee the timestamp can be matched by authors still exists.

Do you see this on a particular configuration? (OS / codec / what encoder, etc.)?

[aboba]

I agree that the handling of timestamp should be explicit for both the encoder and decoder.

Am seeing this on Chrome across codecs, platforms (Windows or Mac OS X) and devices.

Even assuming that VideoFrame.timestamp and EncodedVideoChunk.timestamp have independent offsets but run at the same clockrate, they still don't match up.

[dalecurtis]

@Djuffin @sandersdan

[Djuffin]

I agree that timestamps are expected to match, barring corner cases like dropped frames and one chunk containing two frames.

However, in recent Chromium builds, the behavior has changed

@aboba can you point out what you mean? CL, chrome build, or at least a repro page

[aboba]

Question: Are you seeing timestamp values matching between VideoFrame and EncodedVideoChunk on encode?

Repro is here. Just ran this on Chrome 126 on Mac OS and could repro regardless of the codec or other parameter settings.

Click on the start button, let it capture video for a few seconds, then click stop button. You'll then see a graph of decode times with data points, and an empty graph of encode times that used to include values (not sure of the build when it last functioned correctly).

In the text box, you'll see a dump of VideoFrame.timestamp (output: 0) and EncodedVideoChunk.timestamp (output: 1). None of the EncodedVideoChunk.timestamp values match VideoFrame.timestamp values. The timestamp values for VideoFrame and EncodedVideoChunk have very different offsets, so they aren't even close. Even adjusting for the distinct offsets (e.g. assuming that the two timestamp sequences came from different clocks), I couldn't find matches.

[Djuffin]

Thanks for reporting. it's a Chromium bug