High latency rtspsrc playback with imxvpudec

[Talkless]

Our software on Linux Desktop uses this GSTreamer v1.22.1 to achieve VA-API-based hardware-accelerated IP camera video decoding with just about 220ms delay from reality using this pipeline:

rtspsrc location=rtsp://... protocols=tcp latency=100 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! h264parse ! vah264dec compliance=3 ! glupload ! glcolorconvert ! qmlglsink

Meanwhile, on IMX8MM device (some Chinese-made "panel pc") with this pipeline:

rtspsrc location=rtsp://... protocols=udp latency=200 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! h264parse ! queue max-size-buffers=0 ! imxvpudec_h264 ! queue max-size-buffers=0 ! glupload ! qmlglsink sync=1

I get ~400ms "lag" from the reality. I had to increace latency=200 for rtspsrc to make video framerate stable.

I see same latency without qmlglsink, by using glimagesink in gst-launch-1.0, and if I enable tracing like this:

env GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=trace.log  ./gst-launch-1.0 rtspsrc "location=rtsp://..." protocols=tcp latency=200 buffer-mode=slave ! queue max-size-buffers=0 ! rtph264depay ! queue max-size-buffers=0 ! h264parse ! queue max-size-buffers=0 ! imxvpudec_h264 ! queue max-size-buffers=0 ! glimagesink

I get these statistics:

# gst-stats-1.0 trace.log 

Overall Statistics:
Number of Threads: 0
Number of Elements: 0
Number of Bins: 0
Number of Pads: 0
Number of GhostPads: 0
Number of Buffers passed: 0
Number of Events sent: 0
Number of Message sent: 0
Number of Queries sent: 0
Time: 0:00:04.940553089

Element Latency Statistics:
        0xffff6c0281c0.rtpsession0.sync_src: mean=0:00:00.000523234 min=0:00:00.000523234 max=0:00:00.000523234
        0xffff6c028480.rtpsession1.sync_src: mean=0:00:00.000295741 min=0:00:00.000295741 max=0:00:00.000295741
        0xffff6c0281c0.rtpsession0.recv_rtp_src: mean=0:00:00.000125820 min=0:00:00.000090997 max=0:00:00.004230490
        0xaaaafd285120.rtpstorage0.src: mean=0:00:00.000142735 min=0:00:00.000093622 max=0:00:00.004509857
        0xffff6c033000.rtpssrcdemux0.src_284028243: mean=0:00:00.000119028 min=0:00:00.000080247 max=0:00:00.004701101
        0xffff6c028480.rtpsession1.recv_rtp_src: mean=0:00:00.000194770 min=0:00:00.000106997 max=0:00:00.000421737
        0xaaaafd285240.rtpstorage1.src: mean=0:00:00.000185261 min=0:00:00.000100372 max=0:00:00.000362863
        0xffff6c033140.rtpssrcdemux1.src_1916206052: mean=0:00:00.000113312 min=0:00:00.000082372 max=0:00:00.000432111
        0xffff6c053db0.rtpjitterbuffer1.src: mean=0:00:00.000168245 min=0:00:00.000168245 max=0:00:00.000168245
        0xffff6c04f180.rtpjitterbuffer0.src: mean=0:00:00.017574938 min=0:00:00.000082248 max=0:00:00.187682900
        0xffff6c033280.rtpptdemux0.src_96: mean=0:00:00.000164264 min=0:00:00.000100372 max=0:00:00.005838189
        0xaaaafd20e010.queue0.src: mean=0:00:00.001034534 min=0:00:00.000102621 max=0:00:00.012492227
        0xaaaafd2201e0.rtph264depay0.src: mean=0:00:00.000341606 min=0:00:00.000163619 max=0:00:00.001578200
        0xaaaafd20e310.queue1.src: mean=0:00:00.000347693 min=0:00:00.000072498 max=0:00:00.009754690
        0xaaaafd228ea0.h264parse0.src: mean=0:00:00.002500049 min=0:00:00.000737351 max=0:00:00.013598817
        0xaaaafd20e610.queue2.src: mean=0:00:00.005426299 min=0:00:00.000081497 max=0:00:00.078790367
        0xaaaafd2398a0.imxvpudech264-0.src: mean=0:00:00.163509955 min=0:00:00.074685122 max=0:00:00.242467406
        0xaaaafd20e910.queue3.src: mean=0:00:00.015722577 min=0:00:00.002107933 max=0:00:00.093848762
        0xaaaafd25c140.gluploadelement0.src: mean=0:00:00.001066231 min=0:00:00.000573232 max=0:00:00.003934374
        0xaaaafd25c4d0.glcolorconvertelement0.src: mean=0:00:00.000224491 min=0:00:00.000121996 max=0:00:00.000473360
        0xaaaafd2641d0.glcolorbalance0.src: mean=0:00:00.000126172 min=0:00:00.000094496 max=0:00:00.000264491

Highlight:

0xaaaafd2398a0.imxvpudech264-0.src: mean=0:00:00.163509955 min=0:00:00.074685122 max=0:00:00.242467406

So only imxvpudech264 element itself results basically half of the whole latency.

Is this inherently unavoidable with IMX devices, or could it be reduced?

[Talkless]

We can try playing h264 from file:

wget https://archive.org/download/big_buck_bunny_1080p_CLHD/big_buck_bunny_1080p_CLHD.mp4
env GST_DEBUG_NO_COLOR=1 GST_DEBUG="GST_TRACER:7" GST_TRACERS="latency(flags=pipeline+element)" GST_DEBUG_FILE=trace.log timeout 5s ./gst-launch-1.0 filesrc location=$(pwd)/big_buck_bunny_1080p_CLHD.mp4 ! queue max-size-buffers=0 !  qtdemux ! queue max-size-buffers=0 ! h264parse ! queue max-size-buffers=0 ! imxvpudec_h264 ! queue max-size-buffers=0 ! glimagesink

Now gst-stats-1.0 shows smaller latency when we use file source instead of rtsp:

0xaaaafac0ca10.imxvpudech264-0.src: mean=0:00:00.006786511 min=0:00:00.004894840 max=0:00:00.022958374

[dv1]

Finally revisiting this. I did a glass-to-glass measurement. First, I measured the latency I get when recording something with a USB camera and showing the recording on the same PC the camera is connected to. Then, I set up an RTSP server that encodes the camera signal and sends it to the board. There, I ran your pipeline.

Important: The camera was set up to record 640x480 at 30 fps.

My glass-to-glass measurements showed similar latency figures. However, interestingly, when I replaced imxvpudec_h264 with avdec_h264, I got almost identical latency. This tells me that imxvpudec is not the reason for this latency. It is unclear at this point what is.

You can try it yourself. Replace imxvpudec_h264 with avdec_h264 and do a glass-to-glass measurement.

[Talkless]

Thanks for your test @dv1 , but after getting latest image from device manufacturers I can get very low latency, about 200ms:

Not sure why thought, but I don't complain :) .

https://github.com/Freescale/gstreamer-imx/assets/292191/508ae026-2526-41b2-86d4-61cae6a51b24