NVIDIA / VideoProcessingFramework

Set of Python bindings to C++ libraries which provides full HW acceleration for video decoding, encoding and GPU-accelerated color space and pixel format conversions
Apache License 2.0
1.32k stars 233 forks source link

Decode timestamp and surface w/ builtin demuxer #461

Closed SiftingSands closed 1 year ago

SiftingSands commented 1 year ago

I'm working off of https://github.com/NVIDIA/VideoProcessingFramework/blob/master/samples/SamplePyTorch.py to get a PyTorch GPU tensor for a frame and the corresponding packet data such as dts with the built-in demuxer.

DecodeSingleSurface() only returns a surface, so I tried the following:

...
    packet_data = nvc.PacketData()
    frame_nv12 = np.ndarray(shape=(0), dtype=np.uint8)

    while True:
        # Decode NV12 surface
        frame_ready = nvDec.DecodeSingleFrame(frame_nv12, packet_data)
        src_surface = nvDec.DecodeSurfaceFromPacket(frame_nv12, packet_data)
        if src_surface.Empty():
            break
...

Which returns an error after a few frames

frame pts (display order)      :1019
frame dts (display order)      :1019
frame pos (display order)      :1309973
frame duration (display order) :512
frame pts (display order)      :2043
frame dts (display order)      :1531
frame pos (display order)      :1313702
frame duration (display order) :512
Decode Error occurred for picture 9
Decode Error occurred for picture 10
frame pts (display order)      :3067
frame dts (display order)      :3067
frame pos (display order)      :1599294
frame duration (display order) :512
HW decoder faced error. Re-create instance.
HW decoder reset time: 6 milliseconds
Traceback (most recent call last):
  File "/opt/test/SamplePyTorch.py", line 220, in <module>
    main(gpu_id, encFilePath, decFilePath)
  File "/opt/test/SamplePyTorch.py", line 161, in main
    frame_ready = nvDec.DecodeSingleFrame(frame_nv12, packet_data)
PyNvCodec._PyNvCodec.HwResetException: HW reset

I looked at the other issues here related to "HW reset", but those didn't appear immediately applicable. This video decodes fine if I just call DecodeSingleSurface() (aka removing my modifications to the script).

Do I need to switch to FFmpeg's demuxer to get this to work, or is there a way with the builtin demuxer that I'm missing?

SiftingSands commented 1 year ago

Nevermind, forgot that you just have to pass a PacketData object to DecodeSingleSurface

...
    packet_data = nvc.PacketData()

    while True:
        # Decode NV12 surface
        src_surface = nvDec.DecodeSingleSurface(packet_data)
...