MP4 mux not correctly recording H264 when frames sliced over multiple NALUs

[elwongest]

Hello,

Not sure if I'm doing something incorrectly, or if this is a feature that is not implemented in the MP4 Muxer. I can't get it to properly record H264 streams when each frame/picture is sliced into multiple NAL units.

For some context, I am working on a Directshow-based application that renders and records up to 24 video and audio feeds, coming from IP cameras, hardware encoders/streamers and/or desktop screen captures encoded using x264. I have the following constraints:

• All feeds have to be synchronized, with minimal latency
• Due to large number of feeds, I cannot transcode them (too much CPU), so I have to mux the signals as they arrive

When I use RTSP/H264, or x264 encoding with single NALU for each frame, the app and muxer works great. However, as I lower the fps on the x264 encoding (say 5 fps), this introduces a 4-5 seconds latency. To get an acceptable latency, I use the "superfast" preset, then I try the tune zerolatency (which includes the sliced-threads option). With sliced-threads, frames are broken down into multiple NALUs (1 per CPU core). The LAV Video Decoder correctly decodes the sliced frames, but the muxer only saves the first slice (bottom macroblock of the picture). The rest of the picture flashes gray/green (in NV12 pixel format), and when I try seeking this recorded video, it crashes my GPU.

I have looked into a few options: 1) Concatenate the NALUs into a single NALU per frame, leaving start codes in between. This works with LAV decoder, but not the muxer. 2) Concatenate the NALUs by removing the start codes and nal_unit_type byte. Doesn't work. 3) Tried multiple options on x264 to mimic the zerolatency tune, but without slicing the frames, but I can't get below a few seconds of latency (at 5 fps)

I'd like to get some help, ideally by having the mp4muxer work with sliced frames, or if you can point me to code/sample on how to concatenate sliced frame NALUs, or if someone knows x264 options that can achieve the lowest latency without slicing frames. I am using very recent versions of ffmpeg and Live555 (from december 2018/january 2019).

Thank you

[roman380]

I am not 100% sure here (and I have a short term task ahead of me to also try recording with partial frames, from another signal though) but I am under impression that the format itself does not offer the flexibility to record slices separately.

I would say that item 1 above has to work, the problem you are running into should be solvable.

[elwongest]

Thanks for quick response. When you say format does not offer flexibility to record slices separately, you mean the MP4 container/AVCC might not include a standard for handling partial frames? Do you know where I can look to confirm that?

To give you a fuller picture, I read the item you referenced and recognized some RAM issues I met already. I reduced the size of the internal 200MB buffer down to 10MB. I figured it was this large only because the muxer can handle raw video, which have very large footprint. Since I only use it for H264, I don't need it to buffer so much data. At 10 MB, it works fine.

Also, in 32-bit with LARGEADRESSAWARE, I could not get more than 20x Full HD video signals, so I migrated the application to 64-bit. Again, it works great as long as frames are into single NALUs. I have not tried resolutions above Full HD (not a requirement for my application... yet).

I read some more about NALU format, I will make some more attempts to concatenate.

[roman380]

My point is that if you have a frame split into parts and you want to transfer them using separate DirectShow media samples then MP4 multiplexer would need to write (unless it implements certain internal behavior to do otherwise) the parts in separate MP4 boxes and the file will have 2+ boxes corresponding to the same effectively frame, with time stamp collision and other artifacts. I anticipate it is the real problem and this would confuse players which do not expect, as I suppose, such layout. To clear this up I think you would need to check MPEG-4 specifiocation (parts 12, 14) and generate some content with slices written separately, then attempt to play this on various devices. My guess is that such content will be problematic to play.

Irrelevant to everything else mentioned above a concatenation of data so that one media sample (on the input of MP4 multiplexer) corresponds to one video frame, with correct time stamps attached, should be a good solution to the problem here.