GPU decode error - Githubissues

Mdaughter commented 3 years ago

I run the code in docker container and its image is pytorch/pytorch:1.7.1-cuda11.0-cudnn8-devel. I follow the example and build successfully, but something wrong happen

from decord import VideoReader
from decord import cpu, gpu
video='/data/Celeb-DF-v2/YouTube-real/00000.mp4'
vr = VideoReader(video, ctx=gpu(0))
frames = vr.get_batch([1, 3, 5, 7, 9])

The error pops out like:

CUDA error 3000 at line 341 in file /workspace/decord/src/video/nvcodec/cuda_threaded_decoder.cc: Traceback (most recent call last):
  File "/workspace/decord/python/test.py", line 5, in <module>
    x = vr[1]
  File "/workspace/decord/python/decord/video_reader.py", line 104, in __getitem__
    self.seek_accurate(idx)
  File "/workspace/decord/python/decord/video_reader.py", line 232, in seek_accurate
    success = _CAPI_VideoReaderSeekAccurate(self._handle, pos)
  File "/workspace/decord/python/decord/_ffi/_ctypes/function.py", line 174, in __call__
    check_call(_LIB.DECORDFuncCall(
  File "/workspace/decord/python/decord/_ffi/base.py", line 78, in check_call
    raise DECORDError(err_str)
decord._ffi.base.DECORDError: [11:17:09] /workspace/decord/src/video/nvcodec/cuda_threaded_decoder.cc:270: Check failed: run_.load():

Env:ffmpeg 4.3.2,cuda 11.0,RTX 2080 Ti,Ubuntu 18.04

trifle commented 3 years ago

Hi, I don't have very specific solutions but a few general observations that are always useful:

Ensure you have an up-to-date version of docker and nvidia-docker
Ensure your system drivers are new enough to support the version of CUDA you're running
!! Make sure your system (not docker image) has the required drivers installed, especially nvcuvid - these are not present in the container, they are linked in!
make sure to expose all gpu capabilities and the correct gpus to the container when running
make sure there is no version mismatch between the CUDA libraries installed in the container and those required by your python libraries

If in doubt, there's a new Dockerfile in this repo that has been provided by levan92. You can try to use that as a baseline and install torch from within that.

sandcobainer commented 3 years ago

I see the same issue with ctx=cpu(0), so I doubt if it's anything to do with the GPU

zxf864823150 commented 3 years ago

I also meet this problem. when I just from decord import VideoReader from decord import cpu, gpu vr = VideoReader(video, ctx=gpu(0)) all is ok, the len(vr)can be read,in my case,len(vr) =811. however,when I do frames = vr.get_batch([1, 3, 5, 7, 9]) this error coming DECORDError: [01:55:24] /mnt/zxf/decord_gpu/decord/src/video/nvcodec/cuda_threaded_decoder.cc:270: Check failed: run_.load()

jbohnslav commented 3 years ago

I have the same issue on one of my videos-- CPU only.

I then tried vr.next() and got a more specific error message:

/github/workspace/src/video/ffmpeg/threaded_decoder.cc:292: [12:57:23] /github/workspace/src/video/ffmpeg
/threaded_decoder.cc:218: Check failed: avcodec_send_packet(dec_ctx_.get(), pkt.get()) >= 0 (-22 vs. 0) 
Thread worker: Error sending packet.

mbanani commented 2 years ago

I am facing a similar bug where the first N frames will be okay, but anything after frame N will result in those errors (sending packets and run_.load(). Did any one figure this out?

I am not sure if it's related to this issue regarding corrupt frames or pertaining to videos coming from specific cameras.

schmidtdominik commented 2 years ago

I am facing a similar bug where the first N frames will be okay, but anything after frame N will result in those errors

I'm having this exact issue too when decoding on the CPU. It's not related to corrupt files because the problem doesn't occur if I install decord from source and use it on the GPU.