Question: how to use with mp4 crate

[DCNick3]

Trying to directly pass the packets from Mp4Sample to openh264 fails with some error.

After some digging I found two functions in the utils module: to_bitstream_with_001_be and to_bitstream_with_001_le. The docs for them mention that they are useful when dealing with mp4 crate. But the question is: which one of them should I use?

How do I know which kind of bitstream mp4 crate spits out: big-endian or little-endian length prefixed?

[ralfbiedert]

I'm not entirely sure myself how to determine which one is the right function and why. In my current project where I only deal with videos from exactly one camera system I have

for sample in 1..num_samples {
    let sample_data = mp4reader
        .read_sample(video_track_id, sample)?
        .ok_or(Error::InvalidSample)?;

    let buffer = &sample_data.bytes[..];

    openh264::to_bitstream_with_001_be::<u32>(buffer, &mut nal_conversion_buffer);
    ...
}

But that said, I'm like 51% sure I had to do _le once as well with some other files. If anyone has some better insight into this we should improve the documentation.

[DCNick3]

This transformation is called h264_mp4toannexb in ffmpeg and seems to have quite a lot of logic in it... I'll try to figure out what it does exactly and maybe submit a PR implementing it

[ralfbiedert]

That would be great!

[DCNick3]

Sooo... Actually properly converting an mp4 h264 stream to an Annex B stream requires some additional information extracted from the mp4. To be more exact, AVCDecoderConfigurationRecord stored in an avcC box is needed (see here)

You need two things there:

1. lengthSizeMinusOne containing the size of the length field (it's actually not always 4 bytes, but always in big endian!)
2. Codec parameters in terms of SPS (Sequence parameter set) and PPS (Picture parameter set) NALs

Apparently SPS and PPS is sometimes (always?) omitted from the mp4 bitstream and stored out-of-band in the headers, but openh264 expects them to be passed inline. h264_mp4toannexb inserts those (I think?) before each group of IDR_SLICE NALs. It also omits those if the bitstream already has SPS or PPS NALs before the IDR_SLICE

There's also some logic with regards to what size of start codes to use (00 00 01 vs 00 00 00 01). I think it opts into the length-4 markers at the start of each packet and when emitting SPS or an PPS. length-3 is used otherwise. I don't know if this is actually of any importance...

[ralfbiedert]

Thanks for the summary!

• The _le can be deleted then it seems?
• Yes, writing / converting the SPS / PPS from out-of-band to in-band I've seen before. In our scenario we insert them only once at the beginning, but for our hardware decoder we inserted them ad-hoc when replaying mid-stream. I think there is no harm repeating them.
• About 00 00 01 vs 00 00 00 01, those also didn't matter IIRC, as the decoder actually looks for 00 00 01, and any additional 0 before that is just more noise.

How should this knowledge best be stored / reflected in the project? Have a FAQ entry about mp4 interop?

[DCNick3]

I think the function could be changed to take a reference of a AvcCBox from the mp4 crate (under a feature). It can then do length parsing adhering to the length size field and inserting SPS and PPS as needed

Also, an example would be nice

[DCNick3]

Filed #30 covering this

[ralfbiedert]

This can be closed, right?

[DCNick3]

Sure!

[ilyas-taouaou]

I tried following this example, but it doesn't work for me with this error:

[OpenH264] this = 0x0x559e5955c550, Info:CWelsDecoder::SetOption for ERROR_CON_IDC = 0.
[OpenH264] this = 0x0x559e5955c550, Warning:invalid syntax max_bits_per_mb_denom 39
[OpenH264] this = 0x0x559e5955c550, Info:decode failed, failure type:16 

[OpenH264] this = 0x0x559e5955c550, Info:CWelsDecoder::UninitDecoderCtx(), openh264 codec version = openh264 default: 1.4.
[OpenH264] this = 0x0x559e5955c550, Info:CWelsDecoder::UninitDecoder(), verify memory usage (0 bytes) after free..
[OpenH264] this = 0x0x559e5955c550, Info:CWelsDecoder::~CWelsDecoder()
Error: OpenH264 encountered an error. Native:16. Decoding State:0. User Message:None

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/codotaku/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.75/src/error.rs:551:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/result.rs:1959:27
   2: video_test::main
             at ./src/main.rs:38:30
   3: core::ops::function::FnOnce::call_once
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/function.rs:250:5
   4: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/sys_common/backtrace.rs:154:18
   5: std::rt::lang_start::{{closure}}
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/rt.rs:167:18
   6: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/core/src/ops/function.rs:284:13
   7: std::panicking::try::do_call
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:552:40
   8: std::panicking::try
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:516:19
   9: std::panic::catch_unwind
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panic.rs:142:14
  10: std::rt::lang_start_internal::{{closure}}
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/rt.rs:148:48
  11: std::panicking::try::do_call
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:552:40
  12: std::panicking::try
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panicking.rs:516:19
  13: std::panic::catch_unwind
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/panic.rs:142:14
  14: std::rt::lang_start_internal
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/rt.rs:148:20
  15: std::rt::lang_start
             at /rustc/4cb3beec86178baff601529389306a4949b077ce/library/std/src/rt.rs:166:17
  16: main
  17: <unknown>
  18: __libc_start_main
  19: _start