search

NVIDIA / VideoProcessingFramework

Set of Python bindings to C++ libraries which provides full HW acceleration for video decoding, encoding and GPU-accelerated color space and pixel format conversions
Apache License 2.0
1.31k stars 233 forks source link

Segmentation fault when starting multithread transcoding #219

Open Deformer opened 3 years ago

Deformer commented 3 years ago

Describe the bug Starting multithread transcoding

To Reproduce After running command: python SampleTranscodeMiltiThread.py 0 20 printing in console:

Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Decoding on GPU 0
Encoding on GPU 0
Segmentation fault (core dumped)

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context SampleTranscodeMiltiThread.py code:

import sys
import os

if os.name == 'nt':
    # Add CUDA_PATH env variable
    cuda_path = os.environ["CUDA_PATH"]
    if cuda_path:
        os.add_dll_directory(cuda_path)
    else:
        print("CUDA_PATH environment variable is not set.", file = sys.stderr)
        print("Can't set CUDA DLLs search path.", file = sys.stderr)
        exit(1)

    # Add PATH as well for minor CUDA releases
    sys_path = os.environ["PATH"]
    if sys_path:
        paths = sys_path.split(';')
        for path in paths:
            if os.path.isdir(path):
                os.add_dll_directory(path)
    else:
        print("PATH environment variable is not set.", file = sys.stderr)
        exit(1)

import PyNvCodec as nvc
from enum import Enum
import time
import numpy as np
from threading import Thread

class Transcoder(Thread):
    def __init__(self, n, gpuID):
        Thread.__init__(self)
        self.demuxer = nvc.PyFFmpegDemuxer('./%d.mp4' % n)
        self.decoder = nvc.PyNvDecoder(self.demuxer.Width(), self.demuxer.Height(), self.demuxer.Format(), self.demuxer.Codec(), gpuID)
        self.encoder = nvc.PyNvEncoder(
            {'preset': 'll_hp', 'codec': 'h264', 'vsync': '0', 'profile': 'high',
             's': '720x1280', 'bitrate': '2700K'}, gpuID)

    def run(self):
        inputPacket = np.ndarray(shape=(0), dtype=np.uint8)
        encodedFrame = np.ndarray(shape=(0), dtype=np.uint8)
        while True:
            if not self.demuxer.DemuxSinglePacket(inputPacket):
                break
            surface = self.decoder.DecodeSurfaceFromPacket(inputPacket)
            if surface.Empty():
                continue
            if not self.encoder.EncodeSingleSurface(surface, encodedFrame, sync=True):
                break
        print("Transcoding finished")

if __name__ == "__main__":
    if(len(sys.argv) < 2):
        print("Provide gpuID an N of threads")
        exit(1)

    gpuID = int(sys.argv[1])
    n = 0
    if sys.argv[2]:
        n = int(sys.argv[2])

    threads = []
    for i in range(n):
        th = Transcoder(n, gpuID)
        th.start()
        threads.append(th)
    for th in threads:
        th.join()

    exit(0)
rarzumanyan commented 3 years ago

Hi @Deformer

What's your GPU? Also, please run this script under debugger of choice to get more information on where does segfault happen. I wasn't able to reproduce the issue on my machine, it runs just fine.

Deformer commented 3 years ago

NVIDIA Tesla T4

Thread 13 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff9b63c700 (LWP 131651)]
0x00007ffff58bfd48 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
(gdb) bt
#0  0x00007ffff58bfd48 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#1  0x00007ffff575a747 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ffff59853e0 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ffff56e1099 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ffff570177c in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#5  0x00007ffff57ca075 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#6  0x00007ffff4d19804 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#7  0x00007ffff4cdf197 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#8  0x00007ffff4ce117e in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#9  0x00007ffff4ccb55e in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#10 0x00007ffff4cc7126 in cuvidMapVideoFrame64 () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#11 0x00007ffff750993e in NvDecoder::HandlePictureDisplay(_CUVIDPARSERDISPINFO*) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#12 0x00007ffff750af15 in NvDecoder::HandlePictureDisplayProc(void*, _CUVIDPARSERDISPINFO*) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#13 0x00007ffff4cc16ec in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#14 0x00007ffff4cc1c76 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#15 0x00007ffff4cc40a0 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#16 0x00007ffff4d170a3 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#17 0x00007ffff4d0047c in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#18 0x00007ffff4d17938 in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#19 0x00007ffff4cc180b in ?? () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#20 0x00007ffff750a9e9 in NvDecoder::DecodeLockSurface(VPF::Buffer const*, unsigned long const&, DecodedFrameContext&) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#21 0x00007ffff74f92da in VPF::NvdecDecodeFrame::Run() () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#22 0x00007ffff7fbf929 in VPF::Task::Execute() () from /home/sbelan/VideoProcessingFramework/install/bin/libTC_CORE.so
#23 0x00007ffff758aa6a in PyNvDecoder::getDecodedSurfaceFromPacket(pybind11::array_t<unsigned char, 16>*) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#24 0x00007ffff758b5a3 in PyNvDecoder::DecodeSurface(DecodeContext&) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#25 0x00007ffff758cac0 in PyNvDecoder::DecodeSurfaceFromPacket(pybind11::array_t<unsigned char, 16>&) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#26 0x00007ffff75b646f in pybind11::cpp_function::cpp_function<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}::operator()(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&) const ()
   from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#27 0x00007ffff75fd61f in std::shared_ptr<VPF::Surface> pybind11::detail::argument_loader<PyNvDecoder*, pybind11::array_t<unsigned char, 16>&>::call_impl<std::shared_ptr<VPF::Surface>, pybind11::cpp_function::cpp_function<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&, 0ul, 1ul, pybind11::gil_scoped_release>(pybind11::cpp_function::cpp_function<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul>, pybind11::gil_scoped_release&&) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#28 0x00007ffff75e8c9a in std::enable_if<!std::is_void<std::shared_ptr<VPF::Surface> >::value, std::shared_ptr<VPF::Surface> >::type pybind11::detail::argument_loader<PyNvDecoder*, pybind11::array_t<unsigned char, 16>&>::call<std::shared_ptr<VPF::Surface>, pybind11::gil_scoped_release, pybind11::cpp_function::cpp_function<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&>(pybind11::cpp_function::cpp_function<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&) && () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#29 0x00007ffff75d119a in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}, std::shared_ptr<VPF::Surface>, PyNvDecoder*, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(pybind11::cpp_function::initialize<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&&, std::shared_ptr<VPF::Surface> (*)(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call) const ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#30 0x00007ffff75d125b in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}, std::shared_ptr<VPF::Surface>, PyNvDecoder*, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(pybind11::cpp_function::initialize<std::shared_ptr<VPF::Surface>, PyNvDecoder, pybind11::array_t<unsigned char, 16>&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::call_guard<pybind11::gil_scoped_release> >(std::shared_ptr<VPF::Surface> (PyNvDecoder::*)(pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&)#1}&&, std::shared_ptr<VPF::Surface> (*)(PyNvDecoder*, pybind11::array_t<unsigned char, 16>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::call_guard<pybind11::gil_scoped_release> const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#31 0x00007ffff757f379 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#32 0x00000000005f5b29 in PyCFunction_Call ()
#33 0x00000000005f66f6 in _PyObject_MakeTpCall ()
#34 0x000000000050ad17 in ?? ()
#35 0x0000000000570296 in _PyEval_EvalFrameDefault ()
#36 0x00000000005f5ed6 in _PyFunction_Vectorcall ()
#37 0x000000000056b3fe in _PyEval_EvalFrameDefault ()
#38 0x00000000005f5ed6 in _PyFunction_Vectorcall ()
#39 0x000000000056b3fe in _PyEval_EvalFrameDefault ()
#40 0x00000000005f5ed6 in _PyFunction_Vectorcall ()
#41 0x000000000050a5cc in ?? ()
#42 0x00000000005f54e7 in PyObject_Call ()
#43 0x0000000000654f8c in ?? ()
#44 0x0000000000674ac8 in ?? ()
#45 0x00007ffff7daa609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#46 0x00007ffff7ee6293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
rarzumanyan commented 3 years ago

Thanks for the update, @Deformer 

#10 0x00007ffff4cc7126 in cuvidMapVideoFrame64 () from /lib/x86_64-linux-gnu/libnvcuvid.so.1
#11 0x00007ffff750993e in NvDecoder::HandlePictureDisplay(_CUVIDPARSERDISPINFO*) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so

The segfault happens at low level when reconstructed video frame obtained from decoder is mapped to CUDA memory object. Could you please address these two questions:

  1. Is the issue specific to multi-threaded environment? I mean - does it reproduce when transcoding in single thread?
  2. Any chance you share your input video? I assume the error may be content-specific.

I've been torturing this sample on my machine with Quadro RTX5000 which is very similar to Tesla T4 for a while with different amount of threads but it never failed. I assume the issue may be related to vertical video orientation (at least what I see in the code: self.encoder = nvc.PyNvEncoder( {'preset': 'll_hp', 'codec': 'h264', 'vsync': '0', 'profile': 'high', 's': '720x1280', 'bitrate': '2700K'}, gpuID)).

Deformer commented 3 years ago

@rarzumanyan Thank you for the fast response. 1) I cannot reproduce this on single thread transcoding. Even if start multiple python processes simultaneously, it finished ok 2) Video

rarzumanyan commented 3 years ago

Hi @Deformer

Please check out issue_219 branch ToT. I think I may have missed some of matching CUDA context push/pop calls in decoder class. If that doesn't help I assume the next step would be to temporarily turn off GIL lock release in C++ land.

Deformer commented 3 years ago

@rarzumanyan I have following stacktrace after running script in issue_219

Reading symbols from python...
(No debugging symbols found in python)
Starting program: /usr/bin/python m.py 0 20
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe5b7f700 (LWP 139981)]
[New Thread 0x7fffe537e700 (LWP 139982)]
[New Thread 0x7fffe0b7d700 (LWP 139983)]
[New Thread 0x7fffd5ddd700 (LWP 139987)]
Decoding on GPU 0
[New Thread 0x7fffd55dc700 (LWP 139988)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7509220 in NvDecoder::NvDecoder(CUstream_st*, CUctx_st*, cudaVideoCodec_enum, bool, int, int) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
(gdb) bt
#0  0x00007ffff7509220 in NvDecoder::NvDecoder(CUstream_st*, CUctx_st*, cudaVideoCodec_enum, bool, int, int) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#1  0x00007ffff74fad43 in VPF::NvdecDecodeFrame_Impl::NvdecDecodeFrame_Impl(CUstream_st*, CUctx_st*, cudaVideoCodec_enum, VPF::Pixel_Format) () from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#2  0x00007ffff74f806a in VPF::NvdecDecodeFrame::NvdecDecodeFrame(CUstream_st*, CUctx_st*, cudaVideoCodec_enum, unsigned int, unsigned int, unsigned int, VPF::Pixel_Format) ()
   from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#3  0x00007ffff74f7fa7 in VPF::NvdecDecodeFrame::Make(CUstream_st*, CUctx_st*, cudaVideoCodec_enum, unsigned int, unsigned int, unsigned int, VPF::Pixel_Format) ()
   from /home/sbelan/VideoProcessingFramework/install/bin/libTC.so
#4  0x00007ffff755e571 in PyNvDecoder::PyNvDecoder(unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int) ()
   from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#5  0x00007ffff75cdbe9 in PyNvDecoder* pybind11::detail::initimpl::construct_or_initialize<PyNvDecoder, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int, 0>(unsigned int&&, unsigned int&&, VPF::Pixel_Format&&, cudaVideoCodec_enum&&, unsigned int&&) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#6  0x00007ffff75b5400 in pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}::operator()(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int) const () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#7  0x00007ffff760eb43 in void pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::call_impl<void, pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, pybind11::detail::void_type>(pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul>, pybind11::detail::void_type&&) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#8  0x00007ffff7608132 in std::enable_if<std::is_void<void>::value, pybind11::detail::void_type>::type pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::call<void, pybind11::detail::void_type, pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}&>(pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}&) && () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#9  0x00007ffff75fb00b in pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}, void, pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<PyNvDecoder>&&, void (*)(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call) const () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#10 0x00007ffff75fb1b4 in pybind11::cpp_function::initialize<pybind11::detail::initimpl::constructor<unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int>::execute<pybind11::class_<PyNvDecoder>, , 0>(pybind11::class_<PyNvDecoder>&)::{lambda(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int)#1}, void, pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<PyNvDecoder>&&, void (*)(pybind11::detail::value_and_holder&, unsigned int, unsigned int, VPF::Pixel_Format, cudaVideoCodec_enum, unsigned int), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#11 0x00007ffff757f3b5 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/sbelan/VideoProcessingFramework/install/bin/PyNvCodec.cpython-38-x86_64-linux-gnu.so
#12 0x00000000005f5b29 in PyCFunction_Call ()
#13 0x00000000005f66f6 in _PyObject_MakeTpCall ()
#14 0x000000000050acd5 in ?? ()
#15 0x00000000005f54e7 in PyObject_Call ()
#16 0x000000000059c6b2 in ?? ()
#17 0x00000000005f665f in _PyObject_MakeTpCall ()
#18 0x000000000057094b in _PyEval_EvalFrameDefault ()
#19 0x00000000005f5ed6 in _PyFunction_Vectorcall ()
#20 0x000000000059c50e in ?? ()
#21 0x00000000005f665f in _PyObject_MakeTpCall ()
#22 0x00000000005704c3 in _PyEval_EvalFrameDefault ()
#23 0x000000000056951a in _PyEval_EvalCodeWithName ()
#24 0x000000000068be87 in PyEval_EvalCode ()
#25 0x000000000067b671 in ?? ()
#26 0x000000000067b6ef in ?? ()
#27 0x000000000067b7ab in PyRun_FileExFlags ()
#28 0x000000000067d82e in PyRun_SimpleFileExFlags ()
#29 0x00000000006b5682 in Py_RunMain ()
--Type <RET> for more, q to quit, c to continue without paging--
#30 0x00000000006b5a0d in Py_BytesMain ()
#31 0x00007ffff7deb0b3 in __libc_start_main (main=0x4eee60 <main>, argc=4, argv=0x7fffffffe418, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe408) at ../csu/libc-start.c:308
#32 0x00000000005f9ece in _start ()
rarzumanyan commented 3 years ago

Thanks for the update @Deformer

This means the issue isn't specific to decoding, it reproduces in multi-threaded environment in general, hence no sense in removing the GIL lock.

I'll try to find the machine with same OS & GPU and reproduce it, meanwhile I can only recommend to switch to multi-process instead of multi-threaded transcode. I assume there's a heap data corruption issue. As I find something on this topic I'll update you in this thread.

If it's possible I also recommend you to update to latest driver, CUDA & Video Codec SDK versions, just in case it helps. You can check your current CUDA & driver version by running nvidia-smi.

Deformer commented 3 years ago

Thank you, @rarzumanyan , i'll wait for any updates.

My current driver is latest, i believe. From nvidia-smi: Driver Version: 465.19.01 CUDA Version: 11.3

theHamsta commented 1 year ago

Could not reproduce on the same input video on A6000 (driver 520.61.05), Video SDK header version 12.0). @Deformer are you still seeing this problem on your T4 with newest drivers?