Streaming to different devices extremely slow [How to boost the streaming on ZED Box and Nvidia Jetson]

[keli95566]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
• [X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

I am trying to get the HD720 rectified image streams from a linux device to a Windows device in real time. However, the framerate is extremely low and latency very large.

Steps to Reproduce

1. On the ZED device, run the server command:

gst-launch-1.0 zedsrc camera-resolution=2 camera-fps=30 stream-type=2 ! queue ! zeddemux is-depth=false name=demux demux.src_left ! queue ! videoconvert ! x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=500000 ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5001 max-bitrate=3000000 sync=false async=false demux.src_aux ! queue ! videoconvert ! x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=500000 ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5000 max-bitrate=3000000 sync=false async=false

3. On another device, run the client command: gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,clock-rate=90000,payload=96 ! queue ! rtph264depay ! h264parse ! avdec_h264 ! queue ! autovideoconvert ! fpsdisplaysink
4. Even though I set the frame rate to 30 fps, the fps displayed is on average only 5.2 fps, on average 2.93. I have not measured the latency, but the lag is more than 200 ms.

Expected Result

Gstreamer should be able to stream 2K HD at 30 fps. However, even with 720HD, it is having problem.

Actual Result

Slow streaming at HD 720.

ZED Camera model

ZED Mini

Environment

OS: ZED Edge AI Box Jetson TX2
ZED SDK: 3.7

Anything else?

Please anyone knows a tip for optimization?

[Myzhar]

Hi @keli95566 are you streaming over a cabled network or WiFi?

[keli95566]

Hi ! Thank you for getting back to me so quickly. I am trying to stream over a 5G wifi router. Turning the birate down from 500000 to 5000 does give me a better frame rate at 15fps at HD720, and 30 fps at VGA. However, I can't find any other parameters that give better streaming performance..

[Myzhar]

You could try a cabled connection before, in this way you can understand if it's something related to the reliability of the wireless connection. 5G WiFi has a large bandwidth, but it's short-range, this can affect the quality of the transmission and the number of correct frames received causing a low frame rate.

[keli95566]

Our application needs to have a wireless mobile streaming option. Nonetheless, I gave it a try by connecting both devices to the ethernet network, unfortunately, the performance didn't change much. The following HD720 only streams at around 15fps..

gst-launch-1.0 zedsrc camera-resolution=2 camera-fps=30 stream-type=2 enable-positional-tracking=false depth-mode=0 depth-stabilization=false ! zeddemux is-depth=false name=demux demux.src_left ! videoconvert ! x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=6000 ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5001 max-bitrate=3000000 sync=false async=false demux.src_aux ! videoconvert ! x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=6000 ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5000 max-bitrate=3000000 sync=false async=false

[keli95566]

I am wondering how does VR streaming such as virtual desktop or oculus air link achieve great frame via 5G network? VR streaming works perfectly with the current 5G router at high resolution, framerate, and re-fresh rate..

[Myzhar]

Is the ZED Box the receiver or the sender?

[keli95566]

Is the ZED Box the receiver or the sender?

The ZED Box is the sender..

[Myzhar]

Have you tried to use the GPU accelerated H264 encoder? https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/accelerated_gstreamer.html#wwpID0E06Z0HA

omxh264enc instead of x264enc can boost the performances.

[keli95566]

Have you tried to use the GPU accelerated H264 encoder? https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/accelerated_gstreamer.html#wwpID0E06Z0HA

omxh264enc instead of x264enc can boost the performances.

Great suggestion! Just gave this a try, and it gains a huge performance boost ( from 20fps- to 35fps) Thanks a lot!! The following pipeline works at around 35-40 fps.

Is there any other performance tuning I could perform with the following pipeline?

gst-launch-1.0 zedsrc camera-resolution=2 camera-fps=60 stream-type=2 enable-positional-tracking=false depth-stabilization=false ! zeddemux is-depth=false name=demux demux.src_left ! queue ! autovideoconvert ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5001 max-bitrate=3000000 sync=false async=false demux.src_aux ! queue ! autovideoconvert ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP]:5000 max-bitrate=3000000 sync=false async=false

[Myzhar]

There are many accelerated plugins provided by Nvidia. Here you can find the full list: https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/accelerated_gstreamer.html#wwpID0E0R40HA

I see for example nvvidconv to be used instead autovideoconvert (but not always)

[Myzhar]

There is also a not saying to use nvv4l2h264enc instead of the obsolete omxh264enc

[keli95566]

There are many accelerated plugins provided by Nvidia. Here you can find the full list: https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/accelerated_gstreamer.html#wwpID0E0R40HA

I see for example nvvidconv to be used instead autovideoconvert (but not always)

Thanks a lot for all the great suggestions. I will give them a try !

[Myzhar]

Not a problem, it's always a pleasure to help our customers. Please keep the thread updated so it can be useful for the other users. Thank you.

[keli95566]

Yes, I will further update the optimized pipelines :)

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[oriaj3]

Hi keli95566, with which version of jetpack? Which command is the one that works for you to have no latency?