search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
2.13k stars 440 forks source link

opencv-python Gstreamer nvvidconv error #201

Open TimSong412 opened 1 year ago

TimSong412 commented 1 year ago

When I use gstreamer to decode rtsp stream by opencv-python in docker image foxy-pytorch-l4t-r35.1.0 , I can only use ordinary decoders rather than GPU-based decoder such as nvvidconv. But I can find such element in the cmd.

source="rtspsrc location=rtsp://192.168.144.108:554/stream=0 latency=1 buffer-mode=synced ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink max-buffers=1 drop=true" self.vid = cv2.VideoCapture(source, cv2.CAP_GSTREAMER)

image image

Wonder how to properly use opencv-python & gstreamer & GPU to decode rtsp streams?

TimSong412 commented 1 year ago

image this is the error when using ordinary decoders

dusty-nv commented 1 year ago

Hi @TimSong412, can you check that the NVIDIA GStreamer elements are found in the container? I see them in ros:foxy-pytorch-l4t-r35.1.0 when I check with gst-inspect-1.0:

gst-inspect-1.0 | grep nv
nvdrmvideosink:  nvdrmvideosink: Nvidia Drm Video Sink
nveglstreamsrc:  nveglstreamsrc: nVidia EGL Stream
nvvideosinks:  nv3dsink: Nvidia 3D sink
nvjpeg:  nvjpegdec: JPEG image decoder
nvjpeg:  nvjpegenc: JPEG image encoder
nvtee:  nvtee: NvTee
nvvideosink:  nvvideosink: nVidia Video Sink
nvarguscamerasrc:  nvarguscamerasrc: NvArgusCameraSrc
nveglglessink:  nveglglessink: EGL/GLES vout Sink
nvvidconv:  nvvidconv: NvVidConv Plugin
nvegltransform:  nvegltransform: NvEGLTransform
nvivafilter:  nvivafilter: NvIVAFilter Plugin
nvv4l2camerasrc:  nvv4l2camerasrc: NvV4l2CameraSrc
nvcompositor:  nvcompositor: NvCompositor
nvvideo4linux2:  nvv4l2av1enc: V4L2 AV1 Encoder
nvvideo4linux2:  nvv4l2vp9enc: V4L2 VP9 Encoder
nvvideo4linux2:  nvv4l2vp8enc: V4L2 VP8 Encoder
nvvideo4linux2:  nvv4l2h265enc: V4L2 H.265 Encoder
nvvideo4linux2:  nvv4l2h264enc: V4L2 H.264 Encoder
nvvideo4linux2:  nvv4l2decoder: NVIDIA v4l2 video decoder
TimSong412 commented 1 year ago

@dusty-nv Thanks. Yes they are available in the container. I found that there may be some conflicts with torch. After the torch is imported, such error occurs. image

nvidia@ubuntu:~/matrix-autopilot-uav$ docker run -it --runtime nvidia --rm --net=host --privileged=true  -p 9001:9001/tcp -p 9004:9004/udp -p 12345:12345/udp -p 12777:12777/udp -v $PWD:/workdir/code -v /dev:/dev -v /tmp:/tmp -e GST_DEBUG=WARN -w /workdir/code dustynv/ros:foxy-pytorch-l4t-r35.1.0 bash
WARNING: Published ports are discarded when using host network mode
sourcing   /opt/ros/foxy/install/setup.bash
ROS_ROOT   /opt/ros/foxy
ROS_DISTRO foxy
root@ubuntu:/workdir/code# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import torch
>>> source="rtspsrc location=rtsp://192.168.144.108:554/stream=0 latency=1 buffer-mode=synced ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true"
>>> vid = cv2.VideoCapture(source, cv2.CAP_GSTREAMER)
0:00:00.856294102    73 0xaaab11c88780 WARN                  ladspa gstladspa.c:507:plugin_init:<plugin131> no LADSPA plugins found, check LADSPA_PATH
0:00:00.882079765    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.882158293    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.882200949    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.882247508    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.882287508    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.882354260    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.882406035    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.882473395    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.882547410    73 0xaaab11c88780 WARN                 default gstsf.c:97:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
0:00:02.560610806    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block

(python3:62): GStreamer-WARNING **: 03:30:25.055: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so': /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block
0:00:02.560921844    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstplugin.c:1330:gst_plugin_load_by_name: load_plugin error: Opening module failed: /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block
0:00:02.560966836    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstpluginfeature.c:132:gst_plugin_feature_load: Failed to load plugin containing feature 'nvv4l2decoder'.
0:00:02.561063059    62     0x2c14a980 WARN     GST_ELEMENT_FACTORY gstelementfactory.c:398:gst_element_factory_create:<nvv4l2decoder> loading plugin containing feature (null) returned NULL!
0:00:02.561124819    62     0x2c14a980 ERROR           GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "nvv4l2decoder"
0:00:02.561185746    62     0x2c14a980 ERROR           GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no sink [source=@0x2c16f280]
0:00:02.574551135    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block

(python3:62): GStreamer-WARNING **: 03:30:25.069: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so': /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block
0:00:02.574794494    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstplugin.c:1330:gst_plugin_load_by_name: load_plugin error: Opening module failed: /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block
0:00:02.574833630    62     0x2c14a980 WARN      GST_PLUGIN_LOADING gstpluginfeature.c:132:gst_plugin_feature_load: Failed to load plugin containing feature 'nvvidconv'.
0:00:02.574877053    62     0x2c14a980 WARN     GST_ELEMENT_FACTORY gstelementfactory.c:398:gst_element_factory_create:<nvvidconv> loading plugin containing feature (null) returned NULL!
0:00:02.574943581    62     0x2c14a980 ERROR           GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "nvvidconv"
0:00:02.574995229    62     0x2c14a980 ERROR           GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@(nil)]
0:00:02.577797387    62     0x2c14a980 ERROR           GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@0x2c12bed0]
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (713) open OpenCV | GStreamer warning: Error opening bin: no element "nvv4l2decoder"
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
>>>

However, if I create the videocapture first and then import torch, everything seems fine.

root@ubuntu:/workdir/code# python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> source="rtspsrc location=rtsp://192.168.144.108:554/stream=0 latency=1 buffer-mode=synced ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink max-buffers=1 drop=true"
>>> vid = cv2.VideoCapture(source, cv2.CAP_GSTREAMER)
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Stream format not found, dropping the frame
......
Stream format not found, dropping the frame
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (898) open OpenCV | GStreamer warning: unable to query duration of stream
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (935) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=0, duration=-1
>>> ret, frame=vid.read()
>>> frame.shape
(1080, 1920, 3)
>>> import torch
>>> torch.cuda.is_available()
True
>>> ret, frame=vid.read()
>>> frame.shape
(1080, 1920, 3)

Any help would be appreciated!

rajusm commented 1 year ago

@dusty-nv Hi, I am also faced the same issue when using nvcr.io/nvidia/l4t-pytorch:r35.2.1-pth2.0-py3

The script I tried and working:

import cv2
import time

cap = "rtspsrc location=rtsp://xxxx:xxxx@192.168.0.130:554 latency=0 ! application/x-rtp, media=video ! queue max-size-buffers=100 leaky=downstream ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx, width=1280, height=720 ! videoconvert ! video/x-raw, format=BGR ! appsink max-buffers=1 drop=true sync=false"

cap = cv2.VideoCapture(cap, cv2.CAP_GSTREAMER)
frame_num = 0
while True:
    frame_num += 1
    ret, frame = cap.read()
    if ret:
        print(ret)
        pass
    else:
        break

cap.release()

The script I tried with importing torch and throws error:

import cv2
import time
import torch

cap = "rtspsrc location=rtsp://xxxx:xxxx@192.168.0.130:554 latency=0 ! application/x-rtp, media=video ! queue max-size-buffers=100 leaky=downstream ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx, width=1280, height=720 ! videoconvert ! video/x-raw, format=BGR ! appsink max-buffers=1 drop=true sync=false"

cap = cv2.VideoCapture(cap, cv2.CAP_GSTREAMER)
frame_num = 0
while True:
    frame_num += 1
    ret, frame = cap.read()
    if ret:
        print(ret)
        pass
    else:
        break

cap.release()

eeror:

(python3:154): GStreamer-WARNING **: 11:13:31.127: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so': /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block

(python3:154): GStreamer-WARNING **: 11:13:31.132: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so': /lib/aarch64-linux-gnu/libGLdispatch.so.0: cannot allocate memory in static TLS block
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (713) open OpenCV | GStreamer warning: Error opening bin: no element "nvv4l2decoder"
[ WARN:0] global /opt/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

Is there any solution for this?

Note: nvv4l2decoder plugin is already available

dusty-nv commented 1 year ago

@rajusm try running export LD_PREOAD=/lib/aarch64-linux-gnu/libGLdispatch.so.0 in the container first, before you run your Python scripts.

you can also set this in your own Dockerfile with ENV LD_PRELOAD=${LD_PRELOAD}:/lib/aarch64-linux-gnu/libGLdispatch.so.0 , or when you start the container with --env LD_PREOAD=/lib/aarch64-linux-gnu/libGLdispatch.so.0 argument to docker run.