Passing a user data pointer along with a frame when encoding video

[gizahNL]

I want to pass a pointer to metadata belonging to my application along with a frame and retrieve the associated pointer. Unfortunately I haven't seen any straightforward way to do that. Is passing a user pointer along with a frame when queuing for encoding and retrieving along with bitstream supported?

[jeffreymcallister]

Timestamps may provide what you are looking for. You can see examples of using timestamps to hold application metadata here:

• Media SDK low latency transcode tutorial
• Media SDK low latency encode tutorial

Timestamps are designed to carry container data through the pipeline

• decode: bitstream in->frame out
• vpp:frame in->frame out
• encode: frame in->bitstream out

This timestamp data doesn't affect decode, vpp, or frame operations. If the data attached to each frame is reinterpreted as a pointer, array index, hash, etc. you can work around the size limitation of the timestamp.

[gizahNL]

I'm not sure that would work with encode operations, as that could leak into HRD VPB calculations and other places and would lead to incorrect values in the encoded bitstream when it comes to timing.

[jeffreymcallister]

I understand the concern. Fortunately with this approach the "timestamp" is just external data that is carried through the pipeline for developer convenience. Whether timestamp is set or not doesn't change anything about what gets encoded or how the encode algorithm runs. Whatever 'timestamp' is set to gets copied through the pipeline in any case. You can use the code from one of the links above to prove that the output bitstream is not affected.