NVIDIA / DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
Apache License 2.0
5.1k stars 615 forks source link

The decoder returned a frame that is past the expected one #5665

Open runitao opened 1 day ago

runitao commented 1 day ago

Version

1.41.0

Describe the bug.

I use fn.readers.video to decode videos, the minimum reproducible example will report exception(see relevant log output).

I have test 4 cases, just the case 1 reports exception, others are well.

python dali_issue.py filelist_1.txt   # case 1
python dali_issue.py filelist_2.txt   # case 2, just reorder the video files
python dali_issue.py filelist_3.txt   # case 3, only 349047839_5.mp4
python dali_issue.py filelist_4.txt   # case 4, only 349064170_5.mp4

The video files in all cases are 349064170_5.mp4 and/or 349047839_5.mp4

PS. I test with weekly build too, the problem is still here.

NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2

Minimum reproducible example

git clone https://github.com/runitao/dali_example.git

Relevant log output

[/opt/dali/dali/operators/reader/loader/video_loader.h:179] file_list_include_preceding_frame uses the default value False. In future releases, the default value will be changed to True. iter 0 label [0 0 0 0] 140492320241408 Exception in thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: ./349064170_5.mp4 Traceback (most recent call last): File "dali_issue.py", line 56, in video, label = pipe.run() File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1327, in run return self.outputs() File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1165, in outputs return self._outputs() File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 1250, in _outputs return self._pipe.Outputs() RuntimeError: Critical error in pipeline: Error in GPU operator nvidia.dali.fn.readers.video, which was used in the pipeline definition with the following traceback:

File "dali_issue.py", line 27, in video_pipeline video, label, start_frame_num, timestamps = fn.readers.video(

encountered:

Error in worker thread: The decoder returned a frame that is past the expected one. The most likely cause is variable frame rate video. Filename: ./349064170_5.mp4

Other/Misc.

No response

Check for duplicates

JanuszL commented 20 hours ago

Let me check this and get back to you with more info.

JanuszL commented 16 hours ago

It seems that the video decoder incorrectly returns frame/time skipping the frame we are expecting to get. Let me check this with a video decoder team.

JanuszL commented 16 hours ago

Tracked internally as nv4899180.