Problem with using cv2.cudacodec.createVideoReader

[summm]

• OpenCV => 4.7

import cv2
import time

cap = cv2.cudacodec.createVideoReader("D:/obsrecord/tiktok-live-sample2.flv")
t1 = time.time()
while True:
    ret, frame = cap.nextFrame()
    if not ret:
        t2 = time.time()
        print(t2 - t1)
        break

and get the message

D:\python\Python311\python.exe C:\Users\Administrator\Desktop\code\python\normalpy311\videodecodespeedtest.py 
[ERROR:0@0.740] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (20) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
[ERROR:0@0.740] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (20) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\code\python\normalpy311\videodecodespeedtest.py", line 49, in <module>
    ret, frame = cap.nextFrame()
                 ^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.7.0-dev) D:\opencv\opencv_contrib-4.7.0\modules\cudacodec\src\video_reader.cpp:185: error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding. in function '`anonymous-namespace'::VideoReaderImpl::internalGrab'

Process finished with exit code 1

I saw another person had a similar problem, and the solution was to change the value of Maxium number of packets to 100, but it didn't work for me

D:\python\Python311\python.exe C:\Users\Administrator\Desktop\code\python\normalpy311\videodecodespeedtest.py 
[ERROR:0@0.740] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (100) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
[ERROR:0@0.740] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (100) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\code\python\normalpy311\videodecodespeedtest.py", line 49, in <module>
    ret, frame = cap.nextFrame()
                 ^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.7.0-dev) D:\opencv\opencv_contrib-4.7.0\modules\cudacodec\src\video_reader.cpp:185: error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding. in function '`anonymous-namespace'::VideoReaderImpl::internalGrab'

Process finished with exit code 1

here is the media message

General
Complete name                  : D:\obsrecord\tiktok-live-sample2.flv
Format                         : Flash Video
File size                      : 3.18 MiB
Duration                       : 5 h 31 min
Overall bit rate               : 1 342 b/s
Frame rate                     : 25.000 FPS
Writing application            : Lavf58.45.100
_default_bitrate               : 1300
_min_bitrate                   : 800
_hit_node_optimize             : 1
_os_version                    : 15.6.1
_stream_id                     : push-rtmp-f5-tt01.fcdn.us.tiktokv.com:stage:stream-3571633242648871180_or4
_h_id                          : 5
_max_bitrate                   : 2200
_fp_amode                      : 0
_push_protocol                 : rtmpq
_is_hardware_encode            : 1
_p_id                          : 5
_platform                      : iOS
_fp_coderate                   : 2252800
_qId                           : ByteAudioAUUnsplit+new_session_api
_fp_vmode                      : 1
_fp_nb                         : 0
_sdk_version                   : 13.2.0.20-mt
_publish_time_stamp            : 1681875167
_model                         : iPhone 11 Pro Max
_s_id                          : 8
_interval                      : 2
_ab                            : 0
_start_time                    : 1681875167488
_link_info                     : 

Video
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L3.1
Format settings                : CABAC / 3 Ref Frames
Format settings, CABAC         : Yes
Format settings, Reference fra : 3 frames
Format settings, GOP           : M=1, N=50
Codec ID                       : 7
Duration                       : 5 h 31 min
Bit rate                       : 2 199 kb/s
Width                          : 720 pixels
Height                         : 1 280 pixels
Display aspect ratio           : 0.562
Frame rate mode                : Constant
Frame rate                     : 25.000 FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 0.095
Stream size                    : 5.09 GiB

Audio
Format                         : AAC LC SBR
Format/Info                    : Advanced Audio Codec Low Complexity with Spectral Band Replication
Commercial name                : HE-AAC
Format settings                : NBC
Codec ID                       : 10-5
Duration                       : 5 h 30 min
Bit rate                       : 62.5 kb/s
Channel(s)                     : 2 channels
Channel layout                 : L R
Sampling rate                  : 44.1 kHz
Frame rate                     : 21.533 FPS (2048 SPF)
Compression mode               : Lossy
Delay relative to video        : 5 h 30 min
Stream size                    : 148 MiB

need you help, thank you.

[cudawarped]

Can you attach or post a link to a much much smaller test file than D:\obsrecord\tiktok-live-sample2.flv which causes the same error please?

[summm]

test other one is 926KB, with build Maxium number of packets (20) have the same problem and with build Maxium number of packets (100), without problem but can not get any frame. @cudawarped Thanks for helping again

General
Complete name                  : D:\obsrecord\tiktok-live-sample1.flv
Format                         : Flash Video
File size                      : 925 KiB
Duration                       : 5 h 18 min
Overall bit rate               : 397 b/s
Frame rate                     : 25.000 FPS
Writing application            : Lavf58.45.100
_model                         : iPhone 11 Pro Max
_h_id                          : 5
_fp_vmode                      : 1
_s_id                          : 8
_max_bitrate                   : 2200
_interval                      : 2
_qId                           : ByteAudioAUUnsplit+new_session_api
_publish_time_stamp            : 1681875167
_push_protocol                 : rtmpq
_p_id                          : 5
_fp_coderate                   : 2252800
_ab                            : 0
_fp_nb                         : 0
_platform                      : iOS
_start_time                    : 1681875167488
_sdk_version                   : 13.2.0.20-mt
_default_bitrate               : 1300
_os_version                    : 15.6.1
_stream_id                     : push-rtmp-f5-tt01.fcdn.us.tiktokv.com:stage:stream-3571633242648871180_or4
_link_info                     : n.us-w6qq0o.a8jk_2806979,n.us-w6qq0o.r57k_2595605,cd9268cb-b8e5-4e2e-9fe5-0fbbdc8b8741_6,9bf46c6f-f60b-48b9-95bc-13845c161108_6
_fp_amode                      : 0
_hit_node_optimize             : 1
_is_hardware_encode            : 1
_min_bitrate                   : 800

Video
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L3.1
Format settings                : CABAC / 3 Ref Frames
Format settings, CABAC         : Yes
Format settings, Reference fra : 3 frames
Codec ID                       : 7
Duration                       : 5 h 18 min
Bit rate                       : 2 199 kb/s
Width                          : 720 pixels
Height                         : 1 280 pixels
Display aspect ratio           : 0.562
Frame rate mode                : Constant
Frame rate                     : 25.000 FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 0.095
Stream size                    : 4.89 GiB

Audio
Format                         : AAC LC SBR
Format/Info                    : Advanced Audio Codec Low Complexity with Spectral Band Replication
Commercial name                : HE-AAC
Format settings                : NBC
Codec ID                       : 10-5
Duration                       : 5 h 14 min
Bit rate                       : 62.5 kb/s
Channel(s)                     : 2 channels
Channel layout                 : L R
Sampling rate                  : 44.1 kHz
Frame rate                     : 21.533 FPS (2048 SPF)
Compression mode               : Lossy
Delay relative to video        : 5 h 14 min
Stream size                    : 141 MiB

ok i`m looking for where can i upload the file

[cudawarped]

test other one is 926KB,

Which GPU are you using and can you attach the file so I can see why it fails?

[summm]

windows10, 3070, and here is the file

[cudawarped]

This is a result of the way the bitstream is filtered.

@alalek, @opencv-alalek You probably don't remember but when you added the capacity for encoded frames to be retrieved through FFmpeg with the raw read flag cap.set(CAP_PROP_FORMAT, -1) in https://github.com/opencv/opencv/pull/15290/commits/f98c726cbdaee1a3337f2ceafbbd408491b988c0 you placed a check to determine whether to apply bitstream filtering or not here

// check start code prefixed mode (as defined in the Annex B H.264 / H.265 specification)
if (packet.size >= 5
     && !(packet.data[0] == 0 && packet.data[1] == 0 && packet.data[2] == 0 && packet.data[3] == 1)
     && !(packet.data[0] == 0 && packet.data[1] == 0 && packet.data[2] == 1)
)
{
    filterName = eVideoCodec == CV_CODEC(CODEC_ID_H264) ? "h264_mp4toannexb" : "hevc_mp4toannexb";
}

It appears that the video files supplied by @summm are prefixed however they still need filtering. Now I am not sure if this is an error with the video source or that the above check is unecessary, I have looked through Annex B again and I couldn't find the details specifying that length prefixed bitstreams could not also have a start code. That said I may be misreading it which is why I am seeking your input.

Additionally

1. From my testing the filter does not appear to perform any action when the bitstream is already start code prefixed so I presume the only impact of removing the above check would be the allocation of the filter and a few cycles to determine that no filtering is required but again I might be wrong.
2. Without the check all tests called by opencv_test_cudacodec plus additional rtsp tests I have run pass.
3. The sample code from the Nvidia Video Codec SDK which ingests raw encoded bitstreams parsed with FFmpeg does not perform this check and applies the filter if the below is true.

   eVideoCodec == AV_CODEC_ID_H264 && (
               !strcmp(fmtc->iformat->long_name, "QuickTime / MOV") 
               || !strcmp(fmtc->iformat->long_name, "FLV (Flash Video)") 
               || !strcmp(fmtc->iformat->long_name, "Matroska / WebM")
           );

Given (1-3) should we remove the check or is there a better alternative?

[summm]

Hi @cudawarped, thanks for your hard work, download the code of opencv and oencv_contrb today and rebuit. but still no working. am i loss anything?

import cv2
cap = cv2.cudacodec.createVideoReader("D:/obsrecord/tiktok-live-sample1.flv")

while True:
    succeeded, gpu_frame = cap.nextFrame()

[ERROR:0@0.748] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (20) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
[ERROR:0@0.748] global video_parser.cpp:92 cv::cudacodec::detail::VideoParser::parseVideoData Maxium number of packets (20) parsed without decoding a frame or reconfiguring the decoder, if reading from             a live source consider initializing with VideoReaderInitParams::udpSource == true.
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\code\python\normalpy311\test3.py", line 5, in <module>
    succeeded, gpu_frame = cap.nextFrame()
                           ^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.7.0-dev) D:\opencv230426\opencv_contrib-4.x\modules\cudacodec\src\video_reader.cpp:212: error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding. in function '`anonymous-namespace'::VideoReaderImpl::internalGrab'

[cudawarped]

Hi @cudawarped, thanks for your hard work, download the code of opencv and oencv_contrb today and rebuit. but still no working. am i loss anything?

Hi, I'd guess its because you are on Windows which uses a pre-compiled FFmpeg shared library opencv_videoio_ffmpeg470_64.dll. Unfortunately updates to that don't happen all the time with the last update being 25/12/2022.

If you can't wait for that to be updated (which may be in several months) you have two options, 1) build OpenCV against FFmpeg, or 2) build your own opencv_videoio_ffmpeg470_64.dll.

[summm]

edited

[summm]

@cudawarped I feel my process was successful. I compiled opencv 4.70 with the new ffmpeg and replaced your modified cap_ffmpeg_impl.hpp file.

cap = cv2.VideoCapture("D:/obsrecord/speedtest.mp4")
working so it`s mean with new version ffmpeg is fine?

cap = cv2.VideoCapture("D:/obsrecord/tiktok-live-sample3.flv")
no working, can`t get frame.
cv2.imshow("vid", frame)
cv2.error: OpenCV(4.7.0-dev) D:\opencv470ffmpeg\sources\modules\highgui\src\window.cpp:971: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'

cap = cv2.cudacodec.createVideoReader("D:/obsrecord/tiktok-live-sample3.flv")
no working,
The error messages have indeed changed, I don't know where the problem now.
cap = cv2.cudacodec.createVideoReader("D:/obsrecord/tiktok-live-sample3.flv")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.7.0-dev) D:\opencv470ffmpeg\opencv_contrib-4.7.0\modules\cudacodec\src\cuvid_video_source.cpp:65: error: (-210:Unsupported format or combination of formats) Unsupported video source in function 'cv::cudacodec::detail::CuvidVideoSource::CuvidVideoSource'

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      NO (find_package)
      avcodec:                   YES (#define LIBAVCODEC_VERSION_MINOR  37 #define LIBAVCODEC_VERSION_MICRO 100.37.100)
      avformat:                  YES (#define LIBAVFORMAT_VERSION_MINOR  27 #define LIBAVFORMAT_VERSION_MICRO 100.27.100)
      avutil:                    YES (57.28.100)
      swscale:                   YES (#define LIBSWSCALE_VERSION_MINOR   7 #define LIBSWSCALE_VERSION_MICRO 100.7.100)
      avresample:                NO
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES

[cudawarped]

The output from CMake indicates FFmpeg has not been located. It should look similar to the output below

-- FFMPEG: YES (find_package) -- avcodec: YES (59.18.100) -- avformat: YES (59.16.100) -- avutil: YES (57.17.100) -- swscale: YES (6.4.100)

An alternative if you just want it to work on this file would be to filter it manually first with FFmpeg, e.g.

ffmpeg -i tiktok-live-sample1.flv -vcodec copy -bsf h264_mp4toannexb tiktok-live-sample1.h264

[summm]

@cudawarped thank you very much for your help , not just this file, finaly built with the ffmpeg

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (find_package)
      avcodec:                   YES (59.18.100)
      avformat:                  YES (59.16.100)
      avutil:                    YES (57.17.100)
      swscale:                   YES (6.4.100)
      avresample:                NO
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES

this built all the same with the past just fix path of ffmpeg, now when i import cv2 show

D:\python\Python311\python.exe C:\Users\Administrator\Desktop\code\python\normalpy311\test3.py 
Traceback (most recent call last):
  File "C:\Users\Administrator\Desktop\code\python\normalpy311\test3.py", line 1, in <module>
    import cv2
  File "D:\python\Python311\Lib\site-packages\cv2\__init__.py", line 181, in <module>
    bootstrap()
  File "D:\python\Python311\Lib\site-packages\cv2\__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: DLL load failed while importing cv2

Process finished with exit code 1

[cudawarped]

The easiest thing to do would be to remove the D:\python\Python311\Lib\site-packages\cv2 directory, then manually copy OPENCV_BUILD\lib\python\cv2.cp3x-win_amd64.pyd to D:\python\Python311\Lib\site-packages\ and then after you start the python interpreter add the directories containing the OpenCV, Nvidia and FFmpeg shared libraries e.g.

import os
os.add_dll_directory('C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.0/bin')
os.add_dll_directory('FFMPEG_LIBS/bin')
os.add_dll_directory('OPENCV_BUILD/bin')

before calling import cv2.

[summm]

Thank you @cudawarped ,You are right, it worked now. I forgot the very important first lesson you taught me. At this time, ffmpeg needs to add the path to let opencv know where it is.