Unable to write H264 bytestream and get valid H264 data back: ffmpeg reports "Invalid data found when processing input"

[tachang]

Environment

This is an Ingenic T20 MIPS processor with custom kernel.

v4l2loopback driver version 0.12.5 loaded
Linux openmiko 3.10.14 #1 PREEMPT Sun Aug 30 20:54:47 UTC 2020 mips GNU/Linux

Reproduction

modprobe v4l2loopback devices=1

This gives me /dev/video3 (this is because 0,1, and 2 are being used by Ingenic's custom kernel code and aren't really working v4l2 devices).

Send some data to the /dev/video3 device.

I am using the code here: https://gist.github.com/tachang/e0cdf9d72a4dd9db2050ff8daf3b9892

It retrieves H264 NAL units and writes it out to /dev/video3.

I then read from /dev/video3 and dump it out to a file. I do dd if=/dev/video3 of=temp_video.264. For some reason when I attempt

ffmpeg -i temp_video.264 -c copy -bsf:v trace_headers -f null -

I get

temp_video.264: Invalid data found when processing input

However when I write the to a file instead of /dev/video3 I can see the frames: https://gist.github.com/tachang/51fec92dfdd65228e65bd32502bdc894

This is why I suspect it is an issue in the v4l2loopback driver.

I have the /dev/video3 raw dump here using dd: https://github.com/tachang/attachments/blob/master/dev_video_3_dump.bin

It contains as the first few bytes 00 00 00 01 which is what you'd expect:

$ hexdump -C -n 100 dev_video_3_dump.bin 
00000000  00 00 00 01 65 88 80 40  c7 f4 be 5c ca 80 a5 ec  |....e..@...\....|
00000010  c6 01 e2 ca d0 5f dc 96  71 d9 fa a6 39 8c 72 87  |....._..q...9.r.|
00000020  fb 14 7e eb cd a7 ec 00  f4 ba af 63 af 91 e5 66  |..~........c...f|
00000030  18 74 51 21 ed ab 3f ad  f5 e7 d1 f0 ac 5e e0 37  |.tQ!..?......^.7|
00000040  f1 0c 13 f6 e2 fb a4 c1  b1 82 c9 96 aa 3e b9 4a  |.............>.J|
00000050  ab c5 36 ab 7f f0 04 fa  bc 02 b4 30 8b 0c 65 d5  |..6........0..e.|
00000060  56 ee 89 9d                                       |V...|
00000064

So what I am observing is that the output from /dev/video3 when doing dd doesn't seem to parse in ffmpeg but when I dump it direct to a file and use ffmpeg it does parse it. So I think v4l2loopback is doing something with the frames.

[tachang]

When I insert the module with debug=3 I do see attempts to write:

[23832.465770] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:2058[allocate_buffers]
[23832.483278] v4l2-loopback[2080]: allocating 16588800 = 8294400x2
[23832.494110] v4l2-loopback[2088]: vmallocated 16588800 bytes
[23832.500736] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:2089[allocate_buffers]
[23832.516724] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:2101[init_buffers]
[23832.532728] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:2127[init_buffers]

[23833.569650] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:2058[allocate_buffers]
[23833.584180] v4l2-loopback[2065]: allocating buffers again: 16588800 16588800
[23833.592454] v4l2-loopback[2003]: v4l2_loopback_write() trying to write 26 bytes
[23833.601426] v4l2-loopback[2022]: leave v4l2_loopback_write()
[23833.607619] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:1990[v4l2_loopback_write]
[23833.622401] v4l2-loopback[2003]: v4l2_loopback_write() trying to write 8 bytes
[23833.630222] v4l2-loopback[2022]: leave v4l2_loopback_write()
[23833.640049] /openmiko/build/buildroot-2016.02/output/build/v4l2loopback-aba3067f81b343f4e80c588de895c1aeb0da4b76/./v4l2loopback.c:1990[v4l2_loopback_write]
[23833.654928] v4l2-loopback[2003]: v4l2_loopback_write() trying to write 1060 bytes
[23833.662967] v4l2-loopback[2022]: leave v4l2_loopback_write()

Is it possible that between the time I startup the writer and when I start v4l2rtspserver I am losing bytes? Is there a setting to indicate that unless both ends are connected not to start the stream and block?

[idreamerhx]

got same issue

[mtstickney]

I know this is an old issue by now, but for posterity:

The issue here is that read(), like write(), consumes exactly one buffer entry for the video device. If you request a read of less than the buffer size, the driver will return that data and recycle the buffer, discarding any additional data.

dd reads in 512-byte blocks by default, and if you examine the output dump you'll see there's another start code at offset 0x200. Essentially, dd gets the first 512 bytes of each buffer entry (usually one frame). dd bs=1024 would get (up to) the first 1024 bytes of each buffer, etc.

[tachang]

Are you saying that dd is asking the v4l2 device for at most 512 bytes at the start and then the v4l2 device is providing it but then throwing the rest of the frame away? So the next read of 512 bytes is infact a whole new frame? That's a lot of data to throw away isn't it since a frame is usually a lot more than 512 bytes right?

[mtstickney]

Yep, you got it. I don't think this is intentional behavior, exactly, but the driver code to support the read()/write() interface assumes clients will be reading and writing a complete frame (or at least reading and writing in the same sized chunks, since it doesn't actually know or care where real frame boundaries are).

Support for partial buffer reads is dicey, because a fast producer can and will overwrite the partially-read buffer if the consumer is too slow reading it, and then the client is stuck with part of a frame. An H.264 decoder could resync on the next start code, but not all formats support that sort of thing.

I think the upshot here is that as long as you're dealing with a format with variable-length frames like H.264, the only way to avoid trouble is to read whole buffers at a time, which means using VIDIOC_QBUF/VIDIOC_DQBUF, or doing reads that are always >= the buffer size (dd won't do this, because it will issue smaller reads to fill up its buffer if the first read wasn't exactly that size).

[mtstickney]

It's also worth noting that there is apparently some work going on to move to a different buffer-management framework (#406), which might change the situation somewhat.