Why use FFMEPG instead of Cuda::videoreader

[djmv]

I notice that you use videocapture with ffmpeg instead of use cuda::videoreader. If cuda videoreader is a function of opencv and also do the decoding why is better use ffmpeg and after do an upload from Mat to GpuMat was surprising for me. I have this question or doubt. On the other hand, I'm working in a little project and want to use my video card of nvidia, but I have some problems of decoding video image and using at the same script different functions of opencv that use cuda. I feel that the use of different functions that use gpu in the same script overload the gpu and this produce that when decoding in some frames looks at corrupted. But if I run the project decoding with gpu this corrupted frames disappear.

I open this new issue but I don't know if it is needed here. It is more a discussion that issue. Thanks :)

[dapicard]

cuda::videoreader isn't supported anymore and it works only with CUDA <= 6.5. See https://github.com/opencv/opencv/issues/11220#issuecomment-378582885 This is the reason why I use ffmpeg with cuvid support.

Following the merge request https://github.com/opencv/opencv/pull/11481 you can force the video codec ffmpeg uses with the folowing env var : OPENCV_FFMPEG_CAPTURE_OPTIONS="video_codec;h264_cuvid".

For your specific problem, without taking a look at your code, I can't say what happens.

[djmv]

@dapicard Applying this patch https://gist.github.com/sberryman/936c3aa7918b94daf07bd4923533e9f6 I could use cuda::videoreader with cuda 8.0 and opencv 3.4.0. I think that opencv 3.4.+ with this patch is compatible.

I'm using this code for decoding a rtsp cam

#include <iostream>
#include "opencv2/opencv_modules.hpp"
#if defined(HAVE_OPENCV_CUDACODEC)

#include <string>
#include <vector>
#include <algorithm>
#include <numeric>

#include <opencv2/core.hpp>
#include <opencv2/core/opengl.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/cudacodec.hpp>

using namespace std;
using namespace cv;

int main(int argc, const char* argv[])
{
    if (argc != 2)
        return -1;

    const std::string fname(argv[1]);

    cv::cuda::printCudaDeviceInfo(0);

    cv::cuda::GpuMat d_frame;
    cv::Ptr<cv::cudacodec::VideoReader> d_reader = cv::cudacodec::createVideoReader(fname);

    Mat frame;
    cv::namedWindow("GPU",0);
    d_reader->format();
    for (;;)
    {
        if (!d_reader->nextFrame(d_frame))
            break;
        d_frame.download(frame);

        cv::imshow("GPU", frame);

        if (cv::waitKey(3) > 0)
            break;

    }
    return 0;
}
#endif

I can decode perfectly with this code and really use gpu, but sometimes frames come corrupted, but this bug happen if I only use another functions of opencv that use the gpu. Although, If I only have a script that decode the rtsp, the frames are good and never are corrupted.

[dapicard]

I didn't hear about this patch before, but it seems that you encounter some side effects with it. I can't help you here, but I think that using ffmpeg+cuvid and non-patched OpenCV version would be more stable/maintainable.

(with ffmpeg you also can actually use gpu video decoder, but you have to compile it with cuda support)

[djmv]

Thanks, I'll follow your advice. I also compiled ffmpeg with cuda support, but opencv not take my ffmpeg installation. If run opencv cmake, still not take my compilation. I don't know if some flag is missed.
I installed ffmpeg in this way.

mkdir ~/av_sources
cd ~/av_sources
wget https://www.nasm.us/pub/nasm/releasebuilds/2.13.03/nasm-2.13.03.tar.xz
tar xzvf nasm-2.13.03.tar.xz
cd nasm-2.13.03
./configure --prefix="$HOME/av_build" --bindir="$HOME/bin"
make -j$(nproc) VERBOSE=1
make -j$(nproc) install
make -j$(nproc) distclean

cd ~/av_sources
wget https://github.com/FFmpeg/FFmpeg/archive/n3.2.13.zip
unzip n3.2.13.zip
cd FFmpeg-n3.2.13

sudo apt-get install yasm libvpx. libx264.

PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/av_build/lib/pkgconfig" ./configure \
  --prefix="$HOME/av_build" \
  --pkg-config-flags="--static" \
  --extra-cflags="-I$HOME/av_build/include" \
  --extra-ldflags="-L$HOME/av_build/lib" \
  --bindir="$HOME/bin" \
  --enable-cuda \
  --enable-cuvid \
  --enable-libnpp \
  --extra-cflags="-I/usr/local/cuda/include/" \
  --extra-ldflags=-L/usr/local/cuda/lib64/ \
  --enable-gpl \
  --enable-libx264 \
  --extra-libs=-lpthread \
  --enable-nvenc \
  --enable-nonfree

make -j$(nproc) VERBOSE=1
make -j$(nproc) install
make -j$(nproc) distclean

The system recognize this installation and also I can convert videos using cuda, but opencv doesn't find the ffmpeg version.

Thanks, so much @dapicard

[dapicard]

Do you configure OpenCV with -DWITH_FFMPEG=ON ?

[djmv]

Yes, I set the flag but still not find ffmpeg. look at

- 
--   GUI: 
--     QT:                          YES (ver 5.5.1)
--       QT OpenGL support:         YES (Qt5::OpenGL 5.5.1)
--     GTK+:                        NO
--     OpenGL support:              YES (/usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so)
--     VTK support:                 NO
-- 
--   Media I/O: 
--     ZLib:                        /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.8)
--     JPEG:                        /usr/lib/x86_64-linux-gnu/libjpeg.so (ver )
--     WEBP:                        build (ver encoder: 0x020e)
--     PNG:                         /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.2.54)
--     TIFF:                        build (ver 42 - 4.0.9)
--     JPEG 2000:                   /usr/lib/x86_64-linux-gnu/libjasper.so (ver 1.900.1)
--     OpenEXR:                     build (ver 1.7.1)
--     GDAL:                        NO
-- 
--   Video I/O:
--     DC1394:                      YES (ver 2.2.4)
--     FFMPEG:                      NO
--       avcodec:                   YES (ver 57.64.101)
--       avformat:                  YES (ver 57.56.101)
--       avutil:                    YES (ver 55.34.101)
--       swscale:                   YES (ver 4.2.100)
--       avresample:                NO
--     GStreamer:                   
--       base:                      YES (ver 1.8.3)
--       video:                     YES (ver 1.8.3)
--       app:                       YES (ver 1.8.3)
--       riff:                      YES (ver 1.8.3)
--       pbutils:                   YES (ver 1.8.3)
--     libv4l/libv4l2:              NO
--     v4l/v4l2:                    linux/videodev2.h
--     Xine:                        NO
--     Intel Media SDK:             NO
--     gPhoto2:                     NO
-- 
--   Parallel framework:            TBB (ver 4.4 interface 9002)
-- 
--   Trace:                         YES (with Intel ITT)
-- 
--   Other third-party libraries:
--     Intel IPP:                   2017.0.3 [2017.0.3]
--            at:                   /opencv-3.4.0/build/3rdparty/ippicv/ippicv_lnx
--     Intel IPP IW:                sources (2017.0.3)
--               at:                /opencv-3.4.0/build/3rdparty/ippicv/ippiw_lnx
--     VA:                          YES
--     Intel VA-API/OpenCL:         NO
--     Lapack:                      NO
--     Custom HAL:                  NO
-- 
--   NVIDIA CUDA:                   YES (ver 8.0, CUFFT CUBLAS NVCUVID)
--     NVIDIA GPU arch:             20 30 35 37 50 52 60 61
--     NVIDIA PTX archs:
-- 
--   OpenCL:                        YES (no extra features)
--     Include path:                /opencv-3.4.0/3rdparty/include/opencl/1.2
--     Link libraries:              Dynamic load
-- 
--   Python (for build):            /usr/bin/python2.7
-- 
--   Java:
--     ant:                         NO
--     JNI:                         NO
--     Java wrappers:               NO
--     Java tests:                  NO
-- 
--   Matlab:                        NO
-- 
--   Install to:                    /usr/local
-- -----------------------------------------------------------------
-- 
-- Configuring done
-- Generating done

When run ffmpeg command, I get this

ffmpeg version 3.2.13 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
  configuration: --prefix=/home/tse/av_build --pkg-config-flags=--static --extra-cflags=-I/home/tse/av_build/include --extra-ldflags=-L/home/tse/av_build/lib --bindir=/home/tse/bin --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-I/usr/local/cuda/include/ --extra-ldflags=-L/usr/local/cuda/lib64/ --enable-gpl --enable-libx264 --extra-libs=-lpthread --enable-nvenc --enable-nonfree
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

[djmv]

I don't know if I'm missing some step. You have some guide of installation or compilation of ffmepg with cuda support, that after opencv read well ? .

[dapicard]

I use ffmpeg v3.4.2, configured this way : ./configure --prefix=/var/lib/jenkins/workspace/ffmpeg-cuda-release/install/usr/local --enable-gpl --enable-libx264 --enable-cuda --enable-nvenc --enable-cuvid --enable-nonfree --enable-libnpp '--extra-cflags=-I/usr/local/cuda/include -I/usr/local/src/Video_Codec_SDK_8.1.24/Samples/NvCodec/NvEncoder' --extra-ldflags=-L/usr/local/cuda/lib64 --enable-shared --disable-static

Maybe you miss --enable-shared / --disable-static ?

[djmv]

Thanks, good I used this config and opencv now find ffmpeg

./configure \
  --enable-cuda \
  --enable-cuvid \
  --enable-libnpp \
  --extra-cflags="-I/usr/local/cuda/include/" \
  --extra-ldflags=-L/usr/local/cuda/lib64/ \
  --enable-gpl \
  --enable-libx264 \
  --extra-libs=-lpthread \
  --enable-nvenc \
  --enable-nonfree \
  --enable-shared \ 
  --disable-static

I'm compiling opencv. I'll test ffmepg with cuda in opencv. I'll set ffmepg flags and test it.

[djmv]

With this project is unnecessary export the var for ffmpeg use cuda by default?
OPENCV_FFMPEG_CAPTURE_OPTIONS="video_codec;h264_cuvid|rtsp_transport;tcp"

[dapicard]

You have to set this env var if you want to use the gpu decoder.

[mohand150]

hello @dapicard I want to use the hardware-accelerated of video encoding and decoding, I saw that you used the videocapture with ffmpeg as you explained at the top.

hadware => jetson TX2 with cuda 9.0 and opencv 3.4 + opencv_contrib 3.4

as you advised to use ffmpeg i try to install it as follows:

git clone https://github.com/ffmpeg/ffmpeg -b master
cd ffmpeg
./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp 
--extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64

unfortunately I still get the same error ERROR: cuda requested, but not all dependencies are satisfied: ffnvcodec

I saw the first answer of the question : ERROR: cuvid requested, but not all dependencies are satisfied: cuda/ffnvcodec , problem is The NVIDIA headers were moved out of the FFmpeg codebase to a standalone repository in commit.

I compile the FFmpeg NVIDIA headers ("ffnvcodec"):

git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.git
cd nv-codec-headers
make
sudo make install

i tried to recompile the ffmpeg as first, but i get the same error. ERROR: cuda requested, but not all dependencies are satisfied: ffnvcodec

since I have already installed cuda 9.0, I found another folder /cuda9.0/ and /cuda/ in the directory /usr/local so I have to try again with :

./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --extra-cflags=-I/usr/local/cuda-9.0/include --extra-ldflags=-L/usr/local/cuda-9.0/lib64

config.log

but it still does not work, I'm stuck for 2 days your help is precious

[dapicard]

I never tried to compile this version, but maybe you should add the path to the headers you clone in the compile flags :

./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp '--extra-cflags=-I/usr/local/cuda/include -I/path/to/ffnvcodec/include' --extra-ldflags=-L/usr/local/cuda/lib64

/path/to/ffnvcpodec/include is the folder you cloned from videolan which contains nvEncodeApi.h etc.

[mohand150]

hello @dapicard thank you for your return trying your suggestion, I get the same error and being still stuck on this problem of library (3 days), I prefer passed to a less recent version to advance my project Is it possible to provide me the cloning links of your ffmpeg version? Best regards

[dapicard]

I use ffmpeg v3.4.2 on ubuntu 18.04 LTS for a while now. I just try to compile the v3.4.5 and it works like a charm ; furthermore, I've been able to compile it without the -I/usr/local/src/Video_Codec_SDK_8.1.24/Samples/NvCodec/NvEncoder flag as it is now available in the ffmpeg repo (here for 3.4.5) : https://git.ffmpeg.org/gitweb/ffmpeg.git/tree/cb74b3ff27d510c62b6a78fef5a416674bbb4e51:/compat/nvenc

But I compile ffmpeg using the Nvidia instructions :

• https://developer.nvidia.com/ffmpeg

[mohand150]

hello @dapicard after trying to install it on the jetson TX2 with ubuntu 16.04 several times I have not managed to install it, the problem is that the FFMpeg support planned for desktop GPUs, and it is not supported by the TX2 jetson

so I just compile OpenCV with the parameter with_ffmpeg

General configuration for OpenCV 3.4.1-dev =====================================
  Version control:               3.4.1-9-gec0bb66-dirty

  Extra modules:
    Location (extra):            /home/nvidia/opencv_contrib/modules
    Version control (extra):     3.4.1

  Platform:
    Timestamp:                   2019-03-22T15:04:56Z
    Host:                        Linux 4.4.38-tegra aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      
    Linker flags (Debug):        
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          dl m pthread rt /usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
    3rdparty dependencies:       tbb

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy hdf hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab viz xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    js world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj dnn_modern java matlab ovis sfm
    Applications:                tests perf_tests examples apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI: 
    QT:                          YES (ver 5.5.1)
      QT OpenGL support:         YES (Qt5::OpenGL 5.5.1)
    GTK+:                        NO
    OpenGL support:              YES (/usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so)
    VTK support:                 YES (ver 5.10.1)

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
    JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
    GStreamer:                   
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    libv4l/libv4l2:              1.10.0 / 1.10.0
    v4l/v4l2:                    linux/videodev2.h
    gPhoto2:                     NO

  Parallel framework:            TBB (ver 2018.0 interface 10001)

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.2.92)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 9.0, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             62
    NVIDIA PTX archs:

  OpenCL:                        YES (no extra features)
    Include path:                /home/nvidia/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
    packages path:               lib/python2.7/dist-packages

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.5.2)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.5m.so (ver 3.5.2)
    numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.11.0)
    packages path:               lib/python3.5/dist-packages

  Python (for build):            /usr/bin/python2.7

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        NO

  Install to:                    /usr/local
-----------------------------------------------------------------

i created a makefile for the project i ran it. it works very well.

result of execution

Register signals handler
Stream Ready
Input : rtsp://192.168.1.111:80/live/0/mjpeg.sdp
rtsp://192.168.1.111:80/live/0/mjpeg.sdp CAP_PROP_FPS : 90000
FPS at 53 : 10.2218
FPS at 85 : 6.19675
FPS at 116 : 6.17653
FPS at 148 : 6.35047
FPS at 181 : 6.44531
FPS at 212 : 6.13254
FPS at 243 : 6.13133
FPS at 275 : 6.35551
FPS at 306 : 6.16915
FPS at 338 : 6.35173
FPS at 370 : 6.33538
FPS at 401 : 6.05824
FPS at 432 : 6.1753
FPS at 464 : 6.36309
FPS at 497 : 6.43902
FPS at 528 : 6.18516
FPS at 560 : 6.39616

I just have a question in relation to the number of frames per second (my IP camera can provide 30 FPS) but with the tratements of this algo, on the jetson TX2 I get 6 FPS is this normal (because of program processing)? or this reduced number of frames is due to the non-use of a GPU for decoding video streams? (Do you get the same number of Frames / s?)

[dapicard]

I don't know. This could be due to a lot of things. Computing capacity is one of them. Try to remove some processing steps to see if the framerate increases. It could also be due to the camera : sometimes, when the light is low, cameras decrease the fps to increase the exposure time... And you can use the gpu video decoder by using GStreamer as the backend on Jetson platform. I've done it a long time ago on Jetson TK1.