abhiTronix / deffcode

A cross-platform High-performance FFmpeg based Real-time Video Frames Decoder in Pure Python 🎞️⚡
https://abhitronix.github.io/deffcode
Apache License 2.0
178 stars 3 forks source link

[Idea]: Discarding default FFmpeg parameters and using FFmpeg Filters instead. #31

Closed abhiTronix closed 2 years ago

abhiTronix commented 2 years ago

Issue guidelines

Issue Checklist

Describe your Idea

This idea targets two problem that currently DeFFcode APIs fails to address:

  1. Discarding default FFmpeg parameters: Certain default FFmpeg parameters in DeFFcode API such as -framerate, -pix_fmt, -size cannot be removed/discarded from Decoding Pipeline in any way possible, and thereby present in every Pipeline that DeFFcode API has ever created. These parameters were thought to be extremely necessary for Pipeline to work but my recent tests shown that even if we remove these parameters the Decoding pipeline (with some changes) works totally fine. And instead, FFmpeg itself handles these parameter in best possible manner based on source and decoder provided by the user.
  2. Using FFmpeg Filters: Currently DeFFcode APIs fails to recognize values from filters like fps, format, scale which tends to break the pipeline with artifacts appearing on the output frames. This is due to fact that -framerate, -pix_fmt, -size like parameter are given advantage over filters values by the DeFFCode APIs and was primarily used by them to decode frames. For example if we scale=320:240 filter as follows, the output will be improper(with artifacts) since API tries to use FFmpeg parameter -s i.e. equal to source video frame size(for e.g. 1920x1080) which is way higher than require 320x240 as defined by the filter:

    # import the necessary packages
    from deffcode import FFdecoder
    import cv2
    
    # define params and custom filters
    ffparams = {
        "-vf": "scale=320:240",  # framerate=60fps
    }
    
    # initialize and formulate the decoder with params and custom filters
    decoder = FFdecoder(
        "t.mp4", frame_format="bgr24", verbose=True, **ffparams
    ).formulate()
    
    # grab the BGR24 frames from decoder
    for frame in decoder.generateFrame():
    
        # check if frame is None
        if frame is None:
            break
    
        # {do something with the frame here}
    
        # Show output window
        cv2.imshow("Invalid Output", frame)
    
        # check for 'q' key if pressed
        key = cv2.waitKey(1) & 0xFF
        if key == ord("q"):
            break
    
    # close output window
    cv2.destroyAllWindows()
    
    # terminate the decoder
    decoder.terminate()

Use Cases

Both Discarding default FFmpeg parameters and using FFmpeg Filters, are absolute necessity for decoding/transcoding video frames in GPU, as with FFmpeg commands like ffmpeg -y -vsync 0 -hwaccel_output_format cuda -hwaccel cuda -c:v h264_cuvid -i input.mp4 -vf "scale_npp=format=yuv420p,hwdownload,format=yuv420p,fps=25.0" -f rawvideo - fails to work with FFmpeg parameters -framerate, -pix_fmt, -size and throws Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scale_0' Error reinitializing filters! Failed to inject frame into filter network: Function not implemented error. Therefore, implementing both these features will resolves issue with decoding/transcoding with DeFFcode APIs automatically (thus resolving #30).

Any other Relevant Information?

No response

abhiTronix commented 2 years ago

Successfully fixed and merged fix in commit https://github.com/abhiTronix/deffcode/commit/1918c8e88513ab53ee5b368c0d7e7a12c615579a