search

osai-ai / tensor-stream

A library for real-time video stream decoding to CUDA memory
GNU Lesser General Public License v2.1
377 stars 44 forks source link

Segmentation fault (core dumped) #16

Closed stalestar closed 3 years ago

stalestar commented 3 years ago

hello，i have encountered the problem when i run the python script simple.py，it crash with：

TID: 139899863648000 Initializing()  +
TID: 139899863648000 Chosen GPU: 0
Segmentation fault (core dumped)

i have try the Binaries install with pytorch1.5 and pytorch1.4, problem is same.

i also try to install with source, but it always has a crash in compiling:

include/VideoProcessor.h:83:108: error: converting to ‘std::tuple<int, int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = int; _U2 = int; <template-parameter-2-3> = void; _T1 = int; _T2 = int]’
  leftTopCorner = { 0, 0 }, std::tuple<int, int> rightBottomCorner = { 0, 0 }) {
                                                                           ^
include/VideoProcessor.h:96:125: error: converting to ‘std::tuple<int, int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = int; _U2 = int; <template-parameter-2-3> = void; _T1 = int; _T2 = int]’
 ons(), ColorOptions color = ColorOptions(), CropOptions crop = CropOptions()) {
                                                                           ^
include/VideoProcessor.h:96:125: error: converting to ‘std::tuple<int, int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = int; _U2 = int; <template-parameter-2-3> = void; _T1 = int; _T2 = int]’
include/VideoProcessor.h: In constructor ‘FrameParameters::FrameParameters(ResizeOptions, ColorOptions, CropOptions)’:
include/VideoProcessor.h:96:128: error: converting to ‘std::tuple<int, int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = int; _U2 = int; <template-parameter-2-3> = void; _T1 = int; _T2 = int]’
 ons(), ColorOptions color = ColorOptions(), CropOptions crop = CropOptions()) {
                                                                              ^
include/VideoProcessor.h:96:128: error: converting to ‘std::tuple<int, int>’ from initializer list would use explicit constructor ‘constexpr std::tuple<_T1, _T2>::tuple(_U1&&, _U2&&) [with _U1 = int; _U2 = int; <template-parameter-2-3> = void; _T1 = int; _T2 = int]’
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
BykadorovR commented 3 years ago

Hi @hubeichenxing, What's your target OS? Windows or Linux? What C++ compiler do you use? You can try to re-install dependencies (like FFmpeg, Pytorch, etc) with command lines from Dockerfile

stalestar commented 3 years ago

hi @BykadorovR ，thanks for your reply. my os is ubuntu16.04, used gcc and g++ version is 5.4.0 20160609.

BykadorovR commented 3 years ago

In such case, please try to update your gcc/g++ and other tools (like FFmpeg). You can find tested environment in Dockerfile.

stalestar commented 3 years ago

yeah, i am testing the shell of the Dockerfile, and i will feedback result later.

stalestar commented 3 years ago

unfortunately, after update gcc/g++ to 6.5.0 and 8.4.0 to install ffmpeg, i can install with source successfully, but the Segmentation fault still exist. the crash is on reader.initialize() function.

BykadorovR commented 3 years ago

Can you try to build and execute c_examples? Also try to pass "-v HIGH" to your execution line (python simple.py -v HIGH) to get more detailed log file. Please also check other dependencies, like GPU driver, Python version and so on.

stalestar commented 3 years ago

  1. what version is needed to build c++ library and c++ example？when i run cmake, it crash with( cmake3.5.1 ):

    
CMake Error: Could not find cmake module file: CMakeDetermineCUDACompiler.cmake
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CUDA_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_CUDA_COMPILER
CMake Error: Could not find cmake module file: /root/chenxing/superresolution/RRN/RRN/argus-tensor-stream/build/CMakeFiles/3.5.1/CMakeCUDACompiler.cmake
CMake Error at CMakeLists.txt:2 (project):
No CMAKE_CUDA_COMPILER could be found.

Tell CMake where to find the compiler by setting the CMake cache entry
CMAKE_CUDA_COMPILER to the full path to the compiler, or to the compiler
name if it is in the PATH.

CMake Error: Could not find cmake module file: CMakeCUDAInformation.cmake CMake Error: CMAKE_CUDA_COMPILER not set, after EnableLanguage -- Configuring incomplete, errors occurred!

2. i change a os with cuda10.1(10.2 before), the crash log is not Segmentation fault but following:

TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end TID: 140483301480192 Initializing() + TID: 140483301480192 Chosen GPU: 0 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Parser.cpp Init 314 TID: 140483301480192 Error status != 0, status: -1330794744 TID: 140483301480192 src/Wrappers/WrapperPython.cpp initPipeline 45 TID: 140483301480192 End processing async part TID: 140483301480192 End processing sync part start TID: 140483301480192 End processing sync part end Traceback (most recent call last): File "simple.py", line 100, in reader.initialize(repeat_number=20) File "/opt/conda/lib/python3.6/site-packages/tensor_stream-0.4.0-py3.6-linux-x86_64.egg/tensor_stream/tensor_stream.py", line 200, in initialize RuntimeError: Can't initialize TensorStream

BykadorovR commented 3 years ago

Do you use Dockerfile or install all dependencies locally?

  1. Please check correctness of your GFX driver installation, looks like you have some issues with CUDA.
  2. Double check version of installed FFmpeg module, TensorStream tested with FFmpeg release/4.2 branch

Just checked gcc version (gcc -v) and nvcc (CUDA compiler) version (nvcc --version):

stalestar commented 3 years ago

Thanks for your valuable time and patient. I am not used Dockerfile, but run all shells in the Dockerfile to install dependencies. I have two environment： environment A:

gcc version 8.4.0 (Ubuntu 8.4.0-1ubuntu1~16.04.1)
Cuda compilation tools, release 10.2, V10.2.89
NVIDIA-SMI 460.32.03    Driver Version: 460.32.03
cmake version 3.13.0

ffmpeg version c9f3835 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 8 (Ubuntu 8.4.0-1ubuntu1~16.04.1)
configuration: --enable-cuda --enable-cuvid --enable-shared --disable-static --disable-doc --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --extra-libs=-lpthread --nvccflags='-gencode arch=compute_75,code=sm_75'
libavutil      56. 31.100 / 56. 63.101
libavcodec     58. 54.100 / 58.119.100
libavformat    58. 29.100 / 58. 65.101
libavdevice    58.  8.100 / 58. 11.103
libavfilter     7. 57.100 /  7. 98.100
libswscale      5.  5.100 /  5.  8.100
libswresample   3.  5.100 /  3.  8.100

C++ and python example have same error log:
TID: 140040202721728 Initializing()  +
TID: 140040202721728 Chosen GPU: 0
Segmentation fault (core dumped)

environment B:(C++ example can run successfully!!!)

Cuda compilation tools, release 10.1, V10.1.243
NVIDIA-SMI 418.56       Driver Version: 418.56

Other dependencies are same as environment A.

C++ example can run successfully, log like:
TID: 140320138327808 Processing() 51 frame +
TID: 140320502172096 End processing async part
TID: 140320138327808 Processing() 51 frame -
Function time: 42ms
TID: 140320138327808 Processing was interrupted or stream has ended
TID: 140320502172096 End processing sync part start
TID: 140320138327808 All consumers were notified about processing end
TID: 140320502172096 End processing sync part end

But, python example still fail with:
TID: 140025743394560 Initializing()  +
TID: 140025743394560 Chosen GPU: 0
TID: 140025743394560 Error status != 0, status: -1330794744
TID: 140025743394560 src/Parser.cpp Init 314
TID: 140025743394560 Error status != 0, status: -1330794744
TID: 140025743394560 src/Wrappers/WrapperPython.cpp initPipeline 45
TID: 140025743394560 End processing async part
TID: 140025743394560 End processing sync part start
TID: 140025743394560 End processing sync part end
Traceback (most recent call last):
  File "simple.py", line 100, in <module>
    reader.initialize(repeat_number=20)
  File "/opt/conda/lib/python3.6/site-packages/tensor_stream-0.4.0-py3.6-linux-x86_64.egg/tensor_stream/tensor_stream.py", line 200, in initialize
RuntimeError: Can't initialize TensorStream
BykadorovR commented 3 years ago

Can you double check that you have only 1 available Python/FFmpeg/etc in your environment? So just to avoid situations when you install all extensions to one Python (like from conda package), but try to call another Python (like pure one, installed separately) Try to check closed issues like: 3, 6, in the last issue the same error code from FFmpeg "-1330794744" appeared. BTW, if you just want to use TensorStream as it is, you can install it via pip

stalestar commented 3 years ago

Aha, this issue can solve my problem in environment B. Best wishes for you~