Support adding frame-level metadata

[tvercaut]

A few video containers allow the storage of frame-level metadata. FFmpeg allows to store such data in the metadata field of AVFrame as a pointer to a AVDictionary: https://www.ffmpeg.org/doxygen/trunk/structAVFrame.html#a5bde87fd101f66d6263bb451056dba13

An arbitrary string can then be stored using av_dict_set: https://www.ffmpeg.org/doxygen/trunk/group__lavu__dict.html#ga8d9c2de72b310cef8e6a28c9cd3acbbe

Ideally, we would use this to store a serialised protobuf message.

Note that av_frame_set_metadata has been deprecated only because the metadata field is directly accessible: https://github.com/FFmpeg/FFmpeg/commit/7df37dd319f2d9d3e1becd5d433884e3ccfa1ee2

[dzhoshkun]

• [ ] Does the HEVC specification have provision for frame-level metadata? (possibly this is the wrong question to ask, as this is more likely to be done at the container (MP4) level)
• [x] A quick way to check this is to call av_frame_set_metadata, and then inspecting the resulting file to see how the metadata is stored. Though, inspection may not be straightforward, after all the file will be HEVC-encoded. Maybe can call av_frame_get_metadata on the file and see whether metadata has indeed be stored.
• [ ] Another way is to trace the implementation of av_frame_set_metadata in FFmpeg

[dzhoshkun]

Also the Fraunhofer page on HEVC contains potentially useful links

[dzhoshkun]

• [ ] consider documenting above option

[dzhoshkun]

• [x] branch has an example application to start experimenting with

[dzhoshkun]

• [ ] no metadata seems to be encoded as of the above commit (using both HEVC and VP9, both of which use FFmpeg)

[dzhoshkun]

• [x] there is also an opaque field in AVFrame, which is

for some private data of the user

[dzhoshkun]

• [ ] when and who to free opaque after above commit?

[tvercaut]

For the record, a potentially relevant post I found: https://stackoverflow.com/questions/39853810/opaque-pointer-in-ffmpeg-avframe?rq=1

Note that there is also a AVFrameSideData field... https://www.ffmpeg.org/doxygen/trunk/structAVFrame.html#a44d40e03fe22a0511c9157dab22143ee

Might be worth asking on the libav-user mailing list.

[dzhoshkun]

Thanks Tom. AVFrameSideData seems to be quite specific, as its type field (of type AVFrameSideDataType does not seem to provide an option for free-form data.

[dzhoshkun]

• [ ] check whether there's an FFmpeg flag that needs to be set to activate per-frame metadata

[dzhoshkun]

The opaque field does not seem to be respected by FFmpeg any more, AVBuffer should be used instead

[dzhoshkun]

opaque_ref does not seem to work either, on top of this, it doesn't seem to be available in all FFmpeg versions (see above failing CI build). But maybe there's a missing step in the code, such as https://github.com/gift-surg/GIFT-Grab/issues/66#issuecomment-477509746

[tvercaut]

For the record, it seems like metadata in this context is also referred to as SEI (Supplemental Enhancement Information).

x265 mentions dynamic user SEI (https://x265.readthedocs.io/en/default/releasenotes.html):

Dynamic metadata may be either supplied as a bitstream via the userSEI field of x265_picture, or as a json jile that can be parsed by x265 and inserted into the bitstream; use --dhdr10-info to specify json file name, and --dhdr10-opt to enable optimization of inserting tone-map information only at IDR frames, or when the tone map information changes.

Potentially useful discussion threads:

• https://ffmpeg.org/pipermail/libav-user/2019-February/011614.html
• https://stackoverflow.com/questions/31528348/read-sei-data-from-a-hevc-video-with-ffmpeg

[tvercaut]

Also of interest: Overview of HEVC High-Level Syntax and Reference Picture Management https://doi.org/10.1109/TCSVT.2012.2223052

[dzhoshkun]

In https://doi.org/10.1109/TCSVT.2012.2223052 a "User data registered / User data unregistered" SEI message type is listed, which was inherited from the H.264/AVC standard. This might be what we are looking for:

• https://www.quora.com/What-is-the-simplest-algorithm-to-embed-data-into-h-264-video
• https://code.videolan.org/libav/libav/commit/271ce76d317c5432e151216cf23f12b77ed6cb7e

[dzhoshkun]

Further potentially useful links:

• Decoding Unregistered User Data (SEI) in FFMPEG
• decode_unregistered_user_data source code
• https://github.com/FFmpeg/FFmpeg/blob/3f2103ec73c96d530b0516869363ad9ea730c909/libavcodec/h264_sei.c#L441

[dzhoshkun]

• [ ] only, it looks like "Unregistered User Data" is a one-byte field

[joubs]

Closing as out of date.