abhiTronix / deffcode

A cross-platform High-performance FFmpeg based Real-time Video Frames Decoder in Pure Python 🎞️⚡
https://abhitronix.github.io/deffcode
Apache License 2.0
178 stars 3 forks source link

Can't run the code in "Hardware-Accelerated Video Transcoding" #30

Closed seascheng closed 1 year ago

seascheng commented 2 years ago

Issue guidelines

Issue Checklist

Describe your Question

Hi, Many thanks to you. This project helps me a lot.

I want use my gpu, so I tried the code in "Hardware-Accelerated Video Transcoding"

ENV:

12:13:25 ::    Utilies    ::   INFO   :: Running DeFFcode Version: 0.2.3
12:13:25 ::    Helper     ::   INFO   :: Running VidGear Version: 0.2.6

At first. my ffmpeg already support GPU.

$ ffmpeg  -hide_banner -decoders | grep h264
 VFS..D h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 decoder wrapper (codec h264)
 V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)

$ ffmpeg  -hide_banner -encoders | grep nvenc
 V....D h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 V....D hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)

But after I copied those codes and run, some errors happened.

  1. Seems '-' is a wrong character in "–hwaccel", it should be "-", like "-vsync". They look very similar
  2. Con't use ffparams ("–hwaccel", "cuvid"). The ffmpeg command used by decoder at last is :

    12:21:40 :: FFdecoder :: DEBUG :: Executing FFmpeg command: `ffmpeg -vsync 0 -hwaccel cuvid -vcodec h264_cuvid -i /data/sport/demo/deffcode_test/Big_Buck_Bunny_1080_10s_1MB.mp4 -pix_fmt bgr24 -s 1920x1080 -framerate 60.0 -f rawvideo -

    But this will have an error:

    Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'
    Error reinitializing filters!
    Failed to inject frame into filter network: Function not implemented

    I have checked commands:

    
    ffmpeg -vsync 0 -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo -

ffmpeg -vsync 0 -hwaccel_output_format cuda -hwaccel cuvid -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo -


Both commands can be run. But `-hwaccel cuvid` and `-pix_fmt bgr24` con't by used together.

So I removed `-hwaccel cuvid` to test this case.

3.  `WriteGear   ::  ERROR   :: BrokenPipeError caught, Wrong values passed to FFmpeg Pipe, Kindly Refer Docs!` , If "-resize" in output_params.

I have checked [this doc of WriteGear](https://abhitronix.github.io/vidgear/latest/gears/writegear/compression/params/).
There is no param named "-resize" both in FFmpeg Parameters and Special Internal Parameters.

So I removed `-resize` to test this case.

4. At last, the test case can run to finish. But the output video file can't be opened by video player and ffplay.

Looking forward to receiving your reply.  Thanks again.

### Terminal log output(Optional)

_No response_

### Python Code(Optional)

_No response_

### DeFFcode Version

0.2.3

### Python version

3.8.13

### Operating System version

Ubuntu 10.04

### Any other Relevant Information?

_No response_
abhiTronix commented 2 years ago

@seascheng Thanks for reporting this issue, I'll try address your issue by going over each problem you encountered.

Seems '-' is a wrong character in "–hwaccel", it should be "-", like "-vsync". They look very similar

@seascheng Yes, indeed it was a unicode error. I'll fix it.

Con't use ffparams ("–hwaccel", "cuvid"). The ffmpeg command used by decoder at last is : 12:21:40 :: FFdecoder :: DEBUG :: Executing FFmpeg command: `ffmpeg -vsync 0 -hwaccel cuvid -vcodec h264_cuvid -i /data/sport/demo/deffcode_test/Big_Buck_Bunny_1080_10s_1MB.mp4 -pix_fmt bgr24 -s 1920x1080 -framerate 60.0 -f rawvideo -

But this will have an error:

Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0' Error reinitializing filters! Failed to inject frame into filter network: Function not implemented I have checked commands:

ffmpeg -vsync 0 -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo -

ffmpeg -vsync 0 -hwaccel_output_format cuda -hwaccel cuvid -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo -

@seascheng You should not copy that example as it is, But rather you need to make changes according to your system's NVIDIA GPU. Therefore, to avoid using old CUVID decoder and directly use CUDA with NVDEC, and thereby let FFmpeg use best available encoder by default use following code instead. You can also leave frame format empty to use RGB24 instead of BGR24 with WriteGear API write() method's rgb_mode:

# import the necessary packages
from deffcode import FFdecoder
from vidgear.gears import WriteGear
import json

# define suitable FFmpeg parameter
ffparams = {
    "-vcodec": None,  # skip any decoder and let FFmpeg chose
    "-ffprefixes": [
        "-hwaccel",
        "nvdec",
        "-hwaccel_device",
        "0",
        "-hwaccel_output_format",
        "cuda",
    ],  # accelerator
}

# initialize and formulate the decoder with suitable source and params
decoder = FFdecoder("foo.mp4", verbose=True, **ffparams).formulate()

# retrieve framerate from source JSON Metadata and pass it as `-input_framerate`
# parameter for controlled framerate and define other parameters
output_params = {
    "-input_framerate": json.loads(decoder.metadata)["source_video_framerate"],
    "-vcodec": "h264_nvenc",  # H.264 NVENC encoder
}

# Define writer with defined parameters and suitable
# output filename for e.g. `output_foo.mp4`
writer = WriteGear(output_filename="output_foo.mp4", **output_params)

# grab the BGR24 frame from the decoder
for frame in decoder.generateFrame():

    # check if frame is None
    if frame is None:
        break

    # {do something with the frame here}

    # writing RGB24 frame to writer
    writer.write(frame, rgb_mode=True)

# terminate the decoder
decoder.terminate()

# safely close writer
writer.close()

So I removed -hwaccel cuvid to test this case.

WriteGear :: ERROR :: BrokenPipeError caught, Wrong values passed to FFmpeg Pipe, Kindly Refer Docs! , If "-resize" in output_params. I have checked this doc of WriteGear. There is no param named "-resize" both in FFmpeg Parameters and Special Internal Parameters.

So I removed -resize to test this case.

Yes, that parameter shouldn't be there, it was added just to resize frames directly inside GPU memory. Maybe it is not working, so it is best to remove that parameter.

abhiTronix commented 2 years ago

@seascheng Kindly try above mentioned python code and paste full terminal log here directly. Goodluck!

seascheng commented 2 years ago

@abhiTronix Thank you very much for your quick reply.

full terminal log:

(yolo) cas-air@gpu-server:/data/sport/demo/deffcode_test$ python transcode_gpu_h264.py
15:41:05 ::    Utilies    ::   INFO   :: Running DeFFcode Version: 0.2.3
15:41:05 ::    Helper     ::   INFO   :: Running VidGear Version: 0.2.6
15:41:05 ::   FFhelper    ::  DEBUG   :: Final FFmpeg Path: ffmpeg
15:41:05 ::   FFhelper    ::  DEBUG   :: FFmpeg validity Test Passed!
15:41:05 ::   FFhelper    ::  DEBUG   :: Found valid FFmpeg Version: `b'N-107796-g1af7797d21'` installed on this system
15:41:05 ::    Sourcer    ::  DEBUG   :: Found valid FFmpeg executable: `ffmpeg`.
15:41:05 ::    Sourcer    ::  DEBUG   :: Retrieving Metadata...
15:41:05 ::   FFdecoder   :: CRITICAL :: Activating Video-Only Mode of Operation.
15:41:05 ::   FFdecoder   ::  DEBUG   :: Executing FFmpeg command: `ffmpeg -hwaccel nvdec -hwaccel_device 0 -hwaccel_output_format cuda -i /data/sport/demo/deffcode_test/Big_Buck_Bunny_1080_10s_1MB.mp4 -pix_fmt rgb24 -s 1920x1080 -framerate 60.0 -f rawvideo -`
ffmpeg version N-107796-g1af7797d21 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 8 (Ubuntu 8.4.0-3ubuntu2)
  configuration: --enable-nonfree --enable-gpl --enable-libx264 --enable-cuda --enable-cuda-nvcc --enable-libx265 --enable-nvenc --enable-nvdec --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
  libavutil      57. 33.101 / 57. 33.101
  libavcodec     59. 42.101 / 59. 42.101
  libavformat    59. 30.100 / 59. 30.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 46.103 /  8. 46.103
  libswscale      6.  8.102 /  6.  8.102
  libswresample   4.  8.100 /  4.  8.100
  libpostproc    56.  7.100 / 56.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/data/sport/demo/deffcode_test/Big_Buck_Bunny_1080_10s_1MB.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:00:10.00, start: 0.000000, bitrate: 837 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 832 kb/s, 60 fps, 60 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!
15:41:06 ::   FFdecoder   ::  DEBUG   :: Terminating FFdecoder Pipeline...
15:41:06 ::   FFdecoder   ::  DEBUG   :: FFdecoder Pipeline terminated successfully

It looks the same as before

abhiTronix commented 2 years ago

It looks the same as before

@seascheng Thank you. for the logs. Yes but I think all these -pix_fmt rgb24 -s 1920x1080 -framerate 60.0 parameters can be the problem as you can't change parameters within GPU memory(but only can apply filter with -vf) when -hwaccel nvdec is enabled. But can you test other pixel formats like yuv444p instead of both rgb24/bgr24?

I have checked commands:

ffmpeg -vsync 0 -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo -

ffmpeg -vsync 0 -hwaccel_output_format cuda -hwaccel cuvid -vcodec hevc_cuvid -i /data/sport/auto_directed/ptz_camera/videos/guoxin_h265.mp4 -pix_fmt bgr24 -f rawvideo - Both commands can be run. But -hwaccel cuvid and -pix_fmt bgr24 con't by used together.

Yeah you were right about -hwaccel and -pix_fmt can't by used together: https://stackoverflow.com/a/5986748/10158117

Internal hwaccel decoders are enabled via the -hwaccel option (not supported in ffplay). https://trac.ffmpeg.org/wiki/HWAccelIntro

I don't think we can explicity enable -hwaccel while decoding. Do you see any performance gain without -hwaccel cuvid in previous code?

Also, The problem is right now I don't have access to system with NVIDIA GPU, so I can't really test the best parameters to make this work, and I'm 100% sure it will work. So you'll have to wait.

seascheng commented 2 years ago

@abhiTronix

I'm new at this.please point me out if I'm wrong I have checked all params of this ffmpeg decode command

It seams -hwaccel_output_format cuda and -pix_fmt bgr24 should not be used together , on my env or server at least.

# failed: Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0'
$ ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -c:v h264_cuvid -i input.mp4 -pix_fmt bgr24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null

# Success
$ ffmpeg -y -vsync 0  -hwaccel cuda -c:v h264_cuvid -i input.mp4 -pix_fmt bgr24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null

for hwaccel_output_format, in NVIDIA FFmpeg Transcoding Guide

-hwaccel_output_format cuda keeps the decoded frames in GPU memory

I don't know when the frame is processed in GPU or transferred to CPU and the relationship with ffmpeg params, If you have any references about this, I would be very grateful for them.

I tested decode performance on my server with Nvidia Tesla P4

  1. No -hwaccel_output_format cuda -hwaccel cuda, Just use CPU. ffmpeg -y -vsync 0 -vcodec h264 -i input.mp4 -pix_fmt rgb24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null
  1. with-hwaccel cuda , no -hwaccel_output_format cuda ffmpeg -y -vsync 0 -hwaccel cuda -c:v h264_cuvid -i input.mp4 -pix_fmt bgr24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null
  1. With -hwaccel_output_format cuda -hwaccel cuda, and -vf ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -c:v h264_cuvid -i input.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=25.0" -f rawvideo /dev/null

Yes, using pure CPU and using pure GPU almost have same performance, but using pure CPU, the CPU useage is too high, especially in a server with other services running.

My ideal is to do all steps (decode to frame, frame process, and encoded to video )in one GPU(my server has 8 GPU cards), so i search solution and find this project. Thanks for your work on this.

At last I use this ffparams:

# define suitable FFmpeg parameter
ffparams = {
    "-vcodec": "h264_cuvid",
    "-ffprefixes": [
        "-hwaccel",
        "nvdec",
        "-hwaccel_device",
        "0",
    ],  # accelerator
}

In for frame in decoder.generateFrame(), I use cv2.imwrite("images/image-{:0>5d}.jpg".format(i),frame) to save all frames(bgr24) succesfully, but after using WriteGear encode all frames to a video, as mentioned before, the video con't play.

This is my first touch WriteGear, I will take a look what happened in WriteGear and what is the difference with ffmpeg pip

abhiTronix commented 2 years ago

I tested decode performance on my server with Nvidia Tesla P4

No -hwaccel_output_format cuda -hwaccel cuda, Just use CPU. ffmpeg -y -vsync 0 -vcodec h264 -i input.mp4 -pix_fmt rgb24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null FFMPEG: frame= 5510 fps=992 q=-0.0 size=14874112kB time=00:03:40.76 bitrate=551951.1kbits/s speed=39.7x CPU: %CPU=2128, %MEM=0.7 GPU: 0% with -hwaccel cuda , no -hwaccel_output_format cuda ffmpeg -y -vsync 0 -hwaccel cuda -c:v h264_cuvid -i input.mp4 -pix_fmt bgr24 -s 1280x720 -framerate 25.0 -f rawvideo /dev/null FFMPEG: frame=11138 fps=717 q=-0.0 size=30069760kB time=00:07:25.60 bitrate=552808.5kbits/s speed=28.7x CPU: %CPU=1486, %MEM=0.7 GPU: Memory-Usage=270M, GPU-Util=24% With -hwaccel_output_format cuda -hwaccel cuda, and -vf ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -c:v h264_cuvid -i input.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=25.0" -f rawvideo /dev/null FFMPEG: frame=42794 fps=949 q=-0.0 size=57770496kB time=00:28:31.84 bitrate=276460.4kbits/s speed= 38x CPU: %CPU=170.5, %MEM=0.7 GPU: Memory-Usage=270M, GPU-Util=29% Yes, using pure CPU and using pure GPU almost have same performance, but using pure CPU, the CPU useage is too high, especially in a server with other services running.

My ideal is to do all steps (decode to frame, frame process, and encoded to video )in one GPU(my server has 8 GPU cards), so i search solution and find this project. Thanks for your work on this.

At last I use this ffparams:

define suitable FFmpeg parameter

ffparams = { "-vcodec": "h264_cuvid", "-ffprefixes": [ "-hwaccel", "nvdec", "-hwaccel_device", "0", ], # accelerator }

@seascheng Thank you for this elaborated insight, This just what I needed. I just modified DeFFcode code to drop all three -pix_fmt, -s and -framerate parameter directly from FFdecoder API, so you can test commands like ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -c:v h264_cuvid -i input.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=25.0" -f rawvideo - directly in FFdecoder. Please wait for my next PR and following comments for python code and instructions.

I don't know when the frame is processed in GPU or transferred to CPU and the relationship with ffmpeg params, If you have any references about this, I would be very grateful for them.

See this: https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/

In for frame in decoder.generateFrame(), I use cv2.imwrite("images/image-{:0>5d}.jpg".format(i),frame) to save all frames(bgr24) succesfully, but after using WriteGear encode all frames to a video, as mentioned before, the video con't play.

What Video Player are you using to play video file? for e.g vlc. mpv, wmp etc.

abhiTronix commented 2 years ago

@seascheng I've successfully done the coding part to use filters(like fps, format, scale) instead of parameters(such as -framerate, -pix_fmt, -size). Please wait for my next commits/PR and following comments for python code and instructions.

abhiTronix commented 2 years ago

@seascheng Apologies for the waiting so long, I got stuck in some work. The support for Discarded parameters and utilizing Filter values has been just added to DeFFcode APIs in patch-31 branch and you can install it as follows:

# clone the repository and get inside
git clone https://github.com/abhiTronix/deffcode.git && cd deffcode

# checkout the latest `patch-31` brach
git checkout patch-31

# install normally
pip install .

After installing, run following test code which implements ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -i foo.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=bgr24,fps=30.0" -f rawvideo - command:

# import the necessary packages
from deffcode import FFdecoder
import cv2

# define suitable FFmpeg parameter
ffparams = {
    "-vcodec": None,  # skip any decoder and let FFmpeg chose
    "-ffprefixes": [
        "-vsync",
        "0",
        "-hwaccel",
        "cuda",
        "-hwaccel_output_format",
        "cuda",
    ],  # accelerator
    "-custom_resolution": "null",  # discard `-custom_resolution`
    "-framerate": "null",  # discard `-framerate`
    "-vf": "scale_npp=format=yuv420p,hwdownload,format=bgr24,fps=30.0",  # define your filters
}

# initialize and formulate the decoder with params and custom filters
decoder = FFdecoder(
    "foo.mp4", frame_format="null", verbose=True, **ffparams  # discard frame_format
).formulate()

# grab the BGR24 frames from decoder
for frame in decoder.generateFrame():

    # check if frame is None
    if frame is None:
        break

    # {do something with the frame here}

    # Show output window
    cv2.imshow("Output", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

# close output window
cv2.destroyAllWindows()

# terminate the decoder
decoder.terminate()

Kindly test it as soon as possible and share your terminal logging results here. I'm waiting for you valuable response, Goodluck!

seascheng commented 2 years ago

@abhiTronix Sorry for this late response. format=bgr24 could't be the output of hwdownload.

 $ ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -i input.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=bgr32,fps=25.0" -f rawvideo -

[hwdownload @ 0x55cc95fc6340] Invalid output format rgba for hwframe download.
[Parsed_hwdownload_1 @ 0x55cc9715f800] Failed to configure output pad on Parsed_hwdownload_1
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #0:0
Conversion failed!
abhiTronix commented 2 years ago

@abhiTronix Sorry for this late response. format=bgr24 could't be the output of hwdownload.

@seascheng Yeah, my mistake. Can you test if it is working with yuv420p by testing by following these steps:

1. Install deffcode latest as above:

# clone the repository and get inside
git clone https://github.com/abhiTronix/deffcode.git && cd deffcode

# checkout the latest `patch-31` brach
git checkout patch-31

# install normally
pip install .

2. Install vidgear development branch as follows:

# clone the repository and get inside
git clone https://github.com/abhiTronix/vidgear .git && cd vidgear 

# checkout the latest `development` brach
git checkout development

# install normally
pip install .[core]

3. After installing, run following test code:

# import the necessary packages
from deffcode import FFdecoder
from vidgear.gears import WriteGear
import json

# define suitable FFmpeg parameter
ffparams = {
    "-vcodec": None,  # skip any decoder and let FFmpeg chose
    "-ffprefixes": [
        "-vsync",
        "0",
        "-hwaccel",
        "cuda",
        "-hwaccel_output_format",
        "cuda",
    ],  # accelerator
    "-custom_resolution": "null",  # discard `-custom_resolution`
    "-framerate": "null",  # discard `-framerate`
    "-vf": "scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=30.0",  # define your filters
}

# initialize and formulate the decoder with params and custom filters
decoder = FFdecoder(
    "foo.mp4", frame_format="null", verbose=True, **ffparams  # discard frame_format
).formulate()

# retrieve framerate from JSON Metadata and pass it as 
# `-input_framerate` parameter for controlled framerate
# and add input pixfmt as yuv420p also
output_params = {
    "-input_framerate": json.loads(decoder.metadata)["output_framerate"],
    "-vcodec": "h264_nvenc",
    "-input_pixfmt": "yuv420p"
}

# Define writer with default parameters and suitable
# output filename for e.g. `output_foo_yuv.mp4`
writer = WriteGear(output_filename="output_foo_yuv.mp4", **output_params)

# grab the YUV420 frame from the decoder
for frame in decoder.generateFrame():

    # check if frame is None
    if frame is None:
        break

    # {do something with the frame here}

    # writing YUV420 frame to writer
    writer.write(frame)

# terminate the decoder
decoder.terminate()

# safely close writer
writer.close()

I know, it is little frustrating, but I don't have the hardware to test this, and you're only one for testing this stuff. Kindly test it as soon as possible and share your terminal logging results here. I'm waiting for you valuable response, Goodluck!

seascheng commented 2 years ago

@abhiTronix The new codes can run successfully ,and this output video could play.

Run your code:

(yolo) cas-air@gpu-server:/data/sport/demo/deffcode_test$ python patch-31.py
16:18:19 ::   Utilities   ::   INFO   :: Running DeFFcode Version: 0.2.4
16:18:19 ::    Helper     ::   INFO   :: Running VidGear Version: 0.3.0
16:18:19 ::   FFhelper    ::  DEBUG   :: Final FFmpeg Path: ffmpeg
16:18:19 ::   FFhelper    ::  DEBUG   :: FFmpeg validity Test Passed!
16:18:19 ::   FFhelper    ::  DEBUG   :: Found valid FFmpeg Version: `b'N-107796-g1af7797d21'` installed on this system
16:18:19 ::    Sourcer    ::  DEBUG   :: Found valid FFmpeg executable: `ffmpeg`.
16:18:19 ::    Sourcer    ::  DEBUG   :: Extracting Metadata...
16:18:19 ::    Sourcer    ::  DEBUG   :: Metadata Extraction completed successfully!
16:18:19 ::   FFdecoder   :: CRITICAL :: Manually discarding `frame_format`(or `-pix_fmt`) parameter from this pipeline.
16:18:19 ::   FFdecoder   ::   INFO   :: FFmpeg filter values will be used for this pipeline for defining output pixel-format.
16:18:19 ::   FFdecoder   :: CRITICAL :: Manually discarding `-size/-s` FFmpeg parameter from this pipeline.
16:18:19 ::   FFdecoder   ::   INFO   :: FFmpeg filter values will be used for this pipeline for defining output resolution.
16:18:19 ::   FFdecoder   :: CRITICAL :: Manually discarding `-framerate/-r` FFmpeg parameter from this pipeline.
16:18:19 ::   FFdecoder   ::   INFO   :: FFmpeg filter values will be used for this pipeline for defining output framerate.
16:18:19 ::   FFdecoder   :: CRITICAL :: Activating Video-Only Mode of Operation.
16:18:19 ::   FFdecoder   ::  DEBUG   :: Executing FFmpeg command: `ffmpeg -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i /data/sport/demo/deffcode_test/input_1.mp4 -vf scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=30.0 -f rawvideo -`

...

FFMPEG: frame= 1075 fps=234 q=-0.0 size= 1449900kB time=00:00:36.13 bitrate=328715.3kbits/s speed=7.85x CPU: %CPU=1512, and %CPU=48. (two ffpmeg processes) GPU: Memory-Usage=750M, GPU-Util=4%

Add "-vcodec": "h264_nvenc", in output_params

FFMPEG: frame= 9002 fps=189 q=-0.0 Lsize=12152700kB time=00:05:00.10 bitrate=331739.1kbits/s speed=6.31x CPU: %CPU=66, and %CPU=43, GPU: Memory-Usage=1000M, GPU-Util=8%

It is my pleasure to improve this repo, hope this test could give you some help.

Use this new codes, we get the frame with yuv420p format, so if i want use opencv to process this frame, i need convert it to bgr (in CPU), right?

I dont know my ideal "decode-->process --> encode" all in GPU, refer to Use FFmpeg to Decode H.264 Stream with NVIDIA GPU Acceleration, can be achieved or can't.

abhiTronix commented 2 years ago

@abhiTronix The new codes can run successfully ,and this output video could play.

Run your code:

(yolo) cas-air@gpu-server:/data/sport/demo/deffcode_test$ python patch-31.py 16:18:19 :: Utilities :: INFO :: Running DeFFcode Version: 0.2.4 16:18:19 :: Helper :: INFO :: Running VidGear Version: 0.3.0 16:18:19 :: FFhelper :: DEBUG :: Final FFmpeg Path: ffmpeg 16:18:19 :: FFhelper :: DEBUG :: FFmpeg validity Test Passed! 16:18:19 :: FFhelper :: DEBUG :: Found valid FFmpeg Version: b'N-107796-g1af7797d21' installed on this system 16:18:19 :: Sourcer :: DEBUG :: Found valid FFmpeg executable: ffmpeg. 16:18:19 :: Sourcer :: DEBUG :: Extracting Metadata... 16:18:19 :: Sourcer :: DEBUG :: Metadata Extraction completed successfully! 16:18:19 :: FFdecoder :: CRITICAL :: Manually discarding frame_format(or -pix_fmt) parameter from this pipeline. 16:18:19 :: FFdecoder :: INFO :: FFmpeg filter values will be used for this pipeline for defining output pixel-format. 16:18:19 :: FFdecoder :: CRITICAL :: Manually discarding -size/-s FFmpeg parameter from this pipeline. 16:18:19 :: FFdecoder :: INFO :: FFmpeg filter values will be used for this pipeline for defining output resolution. 16:18:19 :: FFdecoder :: CRITICAL :: Manually discarding -framerate/-r FFmpeg parameter from this pipeline. 16:18:19 :: FFdecoder :: INFO :: FFmpeg filter values will be used for this pipeline for defining output framerate. 16:18:19 :: FFdecoder :: CRITICAL :: Activating Video-Only Mode of Operation. 16:18:19 :: FFdecoder :: DEBUG :: Executing FFmpeg command: ffmpeg -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i /data/sport/demo/deffcode_test/input_1.mp4 -vf scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=30.0 -f rawvideo -

...

@seascheng Thank you for the logs, I really appreciate your help on this. With this working, now we know we are working the right direction. BTW, you can play the output video is not because of corrupt video in previous case, but it is because of this issue explained in this comment and solved by using yuv420p pixel-format

FFMPEG: frame= 1075 fps=234 q=-0.0 size= 1449900kB time=00:00:36.13 bitrate=328715.3kbits/s speed=7.85x CPU: %CPU=1512, and %CPU=48. (two ffpmeg processes) GPU: Memory-Usage=750M, GPU-Util=4%

Yeah high CPU utilization due to two seperate FFmpeg processes. I'm working on a fixing this issue.

Add "-vcodec": "h264_nvenc", in output_params FFMPEG: frame= 9002 fps=189 q=-0.0 Lsize=12152700kB time=00:05:00.10 bitrate=331739.1kbits/s speed=6.31x CPU: %CPU=66, and %CPU=43, GPU: Memory-Usage=1000M, GPU-Util=8%

Wow :heart:, @seascheng you tested the very next I was thinking of doing (Adding "-vcodec": "h264_nvenc", in output_params), and I'm pretty impressed with overall GPU performance. I know CPU is over utilized but I'll work on it next.

It is my pleasure to improve this repo, hope this test could give you some help.

And, I greatly appreciate your help. Genuinely thank you.

Use this new codes, we get the frame with yuv420p format, so if i want use opencv to process this frame, i need convert it to bgr (in CPU), right?

I dont know my ideal "decode-->process --> encode" all in GPU, refer to Use FFmpeg to Decode H.264 Stream with NVIDIA GPU Acceleration, can be achieved or can't.

Sadly I tried everything, but I'm unsuccessful in converting FFmpeg generated YUV420p frames to BGR and OpenCV doesn't out of the box support any format other than bgr/bgra.

abhiTronix commented 2 years ago

Use this new codes, we get the frame with yuv420p format, so if i want use opencv to process this frame, i need convert it to bgr (in CPU), right?

@seascheng I just solved this problem, wait for my upcoming instructions.

abhiTronix commented 1 year ago

@seascheng This issue has been resolved completely in commit https://github.com/abhiTronix/deffcode/commit/3cba34a028f615df9029883b7784c23c970a6cb0 with introduction of new -enforce_cv_patch attribute. Kindly try it earliest as possible by installing dev branch as follows:

# clone the repository and get inside
git clone https://github.com/abhiTronix/deffcode.git && cd deffcode

# checkout the latest dev branch
git checkout dev

# Install it
pip install .

Then test these examples:

  1. https://abhitronix.github.io/deffcode/v0.2.5-dev/recipes/advanced/decode-hw-acceleration/#cuda-accelerated-hardware-based-video-decoding-and-previewing
  2. https://abhitronix.github.io/deffcode/v0.2.5-dev/recipes/advanced/transcode-hw-acceleration/#cuda-nvenc-accelerated-video-transcoding-with-writegear-api (Will be slower due to subprocess bug)