search

kkroening / ffmpeg-python

Python bindings for FFmpeg - with complex filtering support
Apache License 2.0
9.82k stars 880 forks source link

HWACCEL + tensorflow without leaving GpuRam #837

Open BlueMagma2 opened 4 months ago

BlueMagma2 commented 4 months ago

I would like to decode a video using complex_filter and nvdec hardware acceleration. To have it be as fast as possible I would like the decoded frame to be passed to tensorflow directly without leaving the gpu ram so that we don't spend time memcpy-ing the frame from gpu memory to cpu memory and back.

Is it possible using ffmpeg-python ?

page200 commented 1 month ago

In the "Experimental" section of TensorFlow, there's the barely documented/commented tfio.experimental.ffmpeg.decode_video() -> ffmpeg_ops.io_ffmpeg_decode_video() -> libtensorflow_io_ffmpeg_4.2.so -> ...

PyTorch seems more promising. TorchAudio (a library for audio and signal processing with PyTorch) talks here about video:

Using NVIDIA’s GPU decoder and encoder, it is also possible to pass around CUDA Tensor directly, that is decode video into CUDA tensor or encode video from CUDA tensor, without moving data from/to CPU.

The bottom of that page leads to tutorials on video decoding and encoding.

Keep us posted how that goes.