Looking for help producing `CodecPrivate` data for MKV Stream

[dadleyy]

Hello! Thanks for all the work put into this crate. Using this for an application that encodes MJPEG data coming off a V4L video device has been super straightforward; the openh264::encoder::EncodedBitStream I get out, when written to some file (e.g video.h264) are able to be manipulated with every tool I've tried (ffprobe, ffmpeg, h264bitstream); I can take that file and produce .mkv/.mp4 files easily.

However, I'd like the application I am working on to programmatically produce the.mkv data; I will be sending it up to AWS KVS (amazon web services kinesis video streams). For this, their API requires that the data be ingested in the Matroska/MKV container format, and that format requires that we send a CodecPrivate "element" (the underlying format is an xml-like node tree) early on in our stream. I am a bit out of my element here, but I have no idea what should go in that element. Ultimately, it is a vector of bytes, but I'm just not sure how to translate from openh264::encoder::EncodedBitStream to some theoretical CodecPrivate(Vec<u8>), and was wondering if maybe someone already has experience doing this?

I've dug pretty deep, and I think the data is "there", if I:

let bitstream: openh264::encoder::EncodedBitStream<'_> = // ...;

if bitstream.frame_type() == openh264::encoder::FrameType::IDR {
  // do things with `bitstream.num_layers()`  + `bitstream.layer()` -> `layer.nal_unit()`
}

I believe this is what ffmpeg does when it produces .mkv files here in avc.c which is called by mkv_assemble_native_codecprivate in matroskaenc.c.

any help would be really appreciated, thank you!!

[ralfbiedert]

Unfortunately I have never worked with MKV directly. From what you describe it mostly sounds like an embedding problem on the container side though. In rough terms I would expect the flow to look like this:

• Initialize OpenH264
• Encode first frame (YUV -> H264)
• Segment resulting bitstream into NALs, possibly you even have to parse some NALs depending on MKV requirements
• Initialize MKV with video data from your stream / SPS / PPS info
• Add special initialization XML header you describe (details depend on MKV API and format)
• Continue encoding frame and append them to MKV

About the code you posted:

if bitstream.frame_type() == openh264::encoder::FrameType::IDR {
  // do things with `bitstream.num_layers()`  + `bitstream.layer()` -> `layer.nal_unit()`
}

I was about to recommend just using bitstream.write() or bitstream.write_vec() for most regular IDR frames if MKV accepts them as such. But then again, as you pointed out, you might need to access individual layers with the functions you mentioned.

If CodecPrivate really means something private to the H264 codec (I haven't read the spec), this could mean the NAL units, in which case it might be worthwhile just trying to dump things via bitstream.write_vec().

Edit: And if you get this to work, it would be fantastic to file a small PR under examples/mkv to show how it's done.

[dadleyy]

alright I'm going to close this issue out - I was able to get this (putting h264 encoded video inside an MKV container) working and created a "rough draft" of something that works (the success criteria being - "I can view streamed video data in the AWS KVS web interface") in this repository: https://github.com/dadleyy/rs-kvs-streamer.

If it'd be valuable here, I can follow up with a more minimal example (dropping the kvs shenanigans) here in the examples/mkv directory, but I figured I'd at least comment with a link for posterity.