search

PyAV-Org / PyAV

Pythonic bindings for FFmpeg's libraries.
https://pyav.basswood-io.com/
BSD 3-Clause "New" or "Revised" License
2.43k stars 359 forks source link

AV 12.0.0 decoded frames do not match source as in AV 11.0.0 (potential colorspace issue) #1378

Closed mark-oshea closed 3 months ago

mark-oshea commented 4 months ago

Overview

Decoding image frames from media which does not have a colorspace tag does not produce images that visually matche the source movie file.

Historically using both ffmpeg and pyav I have needed to provide the following color conversion when extracting YUV video frames to an RGB format. It's been this way for at least 5 years.

ffmpeg video filter: -vf "colormatrix=bt709:bt601" pyav video reformatter args: {"src_colorspace": "ITU709", "dst_colorspace": "ITU601"}

This has been working fine for me in pyav 10.0.0 and 11.0.0 however as of 12.0.0 this no longer works and I cannot get my source video frames to match their RGB counterpart visually.

I believe the colormatrix/colorspace conversion is required due to libav defaulting to bt601 for video files which do not have a colorspace tag.

Expected behavior

Decoded video frames should match the source video frames.

Actual behavior

Decoded video frames do not match the source video frames.

Investigation

Tested the same code with pyav 11.0.0 and pyav 12.0.0. Extracted frames visually match with av 11.0.0 but do not with av 12.0.0 when executing the same python script.

Reproduction

Minimum example script below - note generally I want to extract video frames to numpy arrays for further processing external to pyav which is why I'm converting to an array here and saving using PIL. The roundtrip from av.VideoFrame -> ndarray -> av.VideoFrame should produce colors that match the source and historically they have.

I did try without the video_reformatter_args and without them, I do not get a visual match from either av 11 or av 12.

Media produced by script (along with source video file) can be accessed here: https://drive.google.com/drive/folders/1OHHkJVP-6sHm9eQQ9iardT8VtwjVz0YT?usp=sharing

Loading up the files above in any media player which supports video and images will show that the image produced by av 11 matches the source video whilst the image produced by av 12 does not.

Example script:

import av
from PIL import Image

SOURCE = "trainspotting.mov"
TARGET = f"trainspotting_frame_{av.__version__}.png"

container = av.open(SOURCE)
stream = container.streams.video[0]

# Needed for saved frame color to match source movie frame
video_reformatter_args = {"src_colorspace": "ITU709", "dst_colorspace": "ITU601"}

# Decode first frame and get 32bit array
frame = next(container.decode(stream))
pixels = frame.to_ndarray(format="gbrpf32le", **video_reformatter_args)

# Scale to 8-bit to save with PIL
pixels *= (2**8) - 1
pixels = pixels.astype("uint8")

image = Image.fromarray(pixels)
image.save(TARGET)

Versions

av==11.0.0

PyAV v11.0.0
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-libtheora --disable-mediafoundation --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-lzma --enable-version3 --enable-zlib
library license: GPL version 3 or later
libavcodec     60.  3.100
libavdevice    60.  1.100
libavfilter     9.  3.100
libavformat    60.  3.100
libavutil      58.  2.100
libswresample   4. 10.100
libswscale      7.  1.100

av==12.0.0

(venv_video_reader) marko@DESKTOP-2JQUU86:~/work/repos/media_io$ python -m av --version
PyAV v12.0.0
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-libtheora --disable-mediafoundation --disable-videotoolbox --enable-fontconfig --enable-gmp --enable-gnutls --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libxcb --enable-libxml2 --enable-lzma --enable-zlib --enable-version3 --enable-libx264 --disable-libopenh264 --enable-libx265 --enable-libxvid --enable-gpl
library license: GPL version 3 or later
libavcodec     60. 31.102
libavdevice    60.  3.100
libavfilter     9. 12.100
libavformat    60. 16.100
libavutil      58. 29.100
libswresample   4. 12.100
libswscale      7.  5.100

Research

I have done the following:

mark-oshea commented 4 months ago

Apologies this was intended to be a runtime bug report.

WyattBlue commented 3 months ago

This may be caused by commit https://github.com/PyAV-Org/PyAV/commit/8aa5fe74d5a581980dd6fde8167adfc75222495e Edit: confirmed with git bisect

WyattBlue commented 3 months ago

Your issue is actually a color range issue. By using {"src_colorspace": "ITU709", "dst_colorspace": "ITU601", "src_color_range": 0, "dst_color_range": 0} I was able to get the right colors using the current version of PyAV.