Issues serving NAL H264 units

tachang commented 3 years ago

So I am copying an H264 byte stream via hardware encoding to /dev/video3 using v4l2loopback.

[ 5329.401628] set sensor gpio as PA-low-10bit
[ 5329.560204] jxf23 0-0040: jxf23 chip found @ 0x40 (i2c0)
[ 5329.565714] tx_isp: Registered sensor subdevice jxf23 0-0040
[ 5329.873126] ###### image_tuning_v4l2_open 4329 #######
ImpSystemInit success
i264e[info]: profile Constrained Baseline, level 4.0
[ 5329.885934] &&& chan1  scaler.max_width = 1920 max_height = 1080  min_width = 128 min_height = 128 &&&
[chn0] scaler->outwidth = 1920 scaler->outheight = 1080, sscaler.outwidth = 0 sscaler.outheight = 0
Open Stream file /dev/video3    vid_format->type                =2
    vid_format->fmt.pix.width       =1920
    vid_format->fmt.pix.height      =1080
    vid_format->fmt.pix.pixelformat =875967048
    vid_format->fmt.pix.sizeimage   =0
    vid_format->fmt.pix.field       =1
    vid_format->fmt.pix.bytesperline=0
    vid_format->fmt.pix.colorspace  =842093913
OK

Starting v4l2rtpserver I am not able to connect with VLC. It says

Codec not supported
VLC could not decode the format "h264" (H264 - MPEG-4 AVC (part 10))

Not sure what could be causing this.

# v4l2rtspserver -v /dev/video3 
log level:600

[NOTICE] main.cpp:385
    Version: 1 live555 version:2016.01.29
[NOTICE] main.cpp:426
    Create V4L2 Source.../dev/video3
[NOTICE] src/V4l2Device.cpp:133
    driver:v4l2 loopback capabilities:85008003 mandatory:4000001
[NOTICE] src/V4l2Device.cpp:135
    /dev/video3 support output
[NOTICE] src/V4l2Device.cpp:136
    /dev/video3 support capture
[NOTICE] src/V4l2Device.cpp:138
    /dev/video3 support read/write
[NOTICE] src/V4l2Device.cpp:139
    /dev/video3 support streaming
[NOTICE] src/V4l2Device.cpp:225
    /dev/video3:H264 size:1920x1080 bufferSize:8294400
[NOTICE] src/V4l2Device.cpp:246
    fps:1/25
[NOTICE] src/V4l2Device.cpp:247
    nbBuffer:2
[NOTICE] src/V4l2MmapDevice.cpp:49
    Device /dev/video3
[NOTICE] src/V4l2MmapDevice.cpp:73
    Device /dev/video3 nb buffer:2
[INFO] src/V4l2MmapDevice.cpp:92
    Device /dev/video3 buffer idx:0 size:8294400 offset:0
[INFO] src/V4l2MmapDevice.cpp:92
    Device /dev/video3 buffer idx:1 size:8294400 offset:8294400
[NOTICE] main.cpp:449
    Create Source .../dev/video3
[NOTICE] inc/V4l2RTSPServer.h:82
    Play this stream using the URL "rtsp://192.168.0.174:8554/unicast"
[NOTICE] src/DeviceSource.cpp:93
    begin thread
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738862 fps:1 bandwidth:64800kbps
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:80 bufSize:5255801
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:24 bufSize:5255773
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:76 bufSize:5255693
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:293 bufSize:5255368
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:57 bufSize:5219268
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:212 bufSize:5216976
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:92 bufSize:5215088
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:160 bufSize:5214856
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:664 bufSize:5214188
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:57 bufSize:5210768
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:5160 bufSize:4867899
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:2196 bufSize:4863903
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:508 bufSize:4863375
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:2228 bufSize:4861143
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:508 bufSize:4860579
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:252 bufSize:4860323
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:628 bufSize:4848727
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:80 bufSize:4847587
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:268 bufSize:4847303
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:628 bufSize:4845935
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:210 bufSize:4830014
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:9884 bufSize:4132311
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:220 bufSize:3152454
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:84 bufSize:2900932
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:8 bufSize:2900920
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:84 bufSize:2900832
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:8 bufSize:2900820
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:76 bufSize:2900740
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:8 bufSize:2900728
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:68 bufSize:2900656
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:8 bufSize:2900644
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:68 bufSize:2900572
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:8 bufSize:2900560
[INFO] src/H264_V4l2DeviceSource.cpp:39
    SPS size:68 bufSize:2900488
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:241 bufSize:2900243
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:1276 bufSize:2383970
[INFO] src/H264_V4l2DeviceSource.cpp:40
    PPS size:76 bufSize:1716579
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:9748 bufSize:1540491
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738863 fps:1 bandwidth:18kbps
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738864 fps:19 bandwidth:263kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:28813 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738865 fps:25 bandwidth:359kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:29068 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738866 fps:25 bandwidth:359kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:29261 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738867 fps:25 bandwidth:424kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:29442 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738868 fps:25 bandwidth:344kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:29671 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738869 fps:25 bandwidth:535kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30253 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738870 fps:25 bandwidth:442kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30082 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738871 fps:25 bandwidth:378kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30722 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738872 fps:25 bandwidth:363kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30658 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738873 fps:25 bandwidth:360kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30527 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738874 fps:25 bandwidth:366kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30923 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738875 fps:25 bandwidth:304kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30773 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738876 fps:25 bandwidth:370kbps
[INFO] src/H264_V4l2DeviceSource.cpp:41
    IDR size:30264 bufSize:0
[INFO] src/DeviceSource.cpp:29
    intv_sec:1598738877 fps:25 bandwidth:368kbps

mpromonet commented 3 years ago

Hi,

Thé frame spliting is messing up. Do you set h264 marker (0 0 0 1) writing frame to v4l2loopback device ?

Best Regards Michel

tachang commented 3 years ago

I am just writing the elementary h264 bytestream to the /dev/video3 v4l2loopback device. When I dump it into a file and run this:

ffmpeg -i video3.264 -c copy -bsf:v trace_headers -f null -

I get the following: https://gist.github.com/tachang/51fec92dfdd65228e65bd32502bdc894

My take on this is that there are no Access Unit Delimiter (AUD) units.

Basing this on https://doc-kurento.readthedocs.io/en/6.11.0/knowledge/h264.html#nal-units-nalu:

The NAL units can contain either actual data video (VCL units) or codec metadata (non-VCL units). Some of the most important types of NAL units are:

Sequence Parameter Set (SPS): This non-VCL NALU contains information required to configure the decoder such as profile, level, resolution, frame rate.
Picture Parameter Set (PPS): Similar to the SPS, this non-VCL NALU contains information on entropy coding mode, slice groups, motion prediction, quantization parameters (QP), and deblocking filters.
Instantaneous Decoder Refresh (IDR): This VCL NALU is a self contained image slice. That is, an IDR can be decoded and displayed without referencing any other NALU save SPS and PPS.
Access Unit Delimiter (AUD): An AUD is an optional NALU that can be use to delimit frames in an elementary stream. It is not required (unless otherwise stated by the container/protocol, like TS), and is often not included in order to save space, but it can be useful to finds the start of a frame without having to fully parse each NALU.

So my bytestream is missing the last one.

My guess is that v4l2loopback does not support bytestream. I am setting:

    vid_format.fmt.pix.pixelformat = V4L2_PIX_FMT_H264;

which by this documentation: https://dri.freedesktop.org/docs/drm/media/uapi/v4l/pixfmt-compressed.html

Says

H264 Access Unit. The decoder expects one Access Unit per buffer. The encoder generates one Access Unit per buffer. If ioctl VIDIOC_ENUM_FMT reports V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM then the decoder has no requirements since it can parse all the information from the raw bytestream.

I am on kernel 3.10.14 which does not have any references to V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM.

So my guess is I need to write out a AUD NALU to "frame" the bytestream?

mpromonet commented 3 years ago

Hi,

As I told you, I guess you are missing the start code prefix, the code of v4l2rtspServer looks for a 4 byte or a 3 byte ones, to split frame SPS, PPS, IDR,.... A standard V4L2 device get in one read (SPS, PPS, IDR) then the code split that. If you don't have any start code prefix and one H264 frame per read, you can easily modify the code to process such data using directly the base V4L2DeviceSource instead of H264_V4L2DeviceSource.

Best Regards, Michel.

P.S. recent v4l2loopback support H264, you may have a look to https://github.com/mpromonet/v4l2tools that can compress H264 to a v4l2loopback device.

tachang commented 3 years ago

Wouldn't the code error out if the start code prefix of {0, 0, 0, 1} wasn't present?

I see it here: https://github.com/mpromonet/v4l2rtspserver/blob/master/src/H264_V4l2DeviceSource.cpp#L134

I am using the most v4l2loopback that supports H264.

tachang commented 3 years ago

I'm not seeing the log message No marker found which I would expect.

tachang commented 3 years ago

It seems like what is happening is that I am not seeing SPS or PPS units in time. I managed to somehow get it to work by letting it feed frames for a while but haven't been able to replicate again. Is there a "hook" into the server to begin streaming?

mpromonet commented 3 years ago

Hi,

SPS & PPS are decoded from the capture frame and updated, this fill the auxilary codec information that will be given to RTSP DESCRIBE. If the parameter -c is not given the SPS & PPS frames are added in front of each IDR frame.

Best Regards, Michel.

mpromonet / v4l2rtspserver

Issues serving NAL H264 units #200