5-second jump changes pixel format on HDR videos

Amsitram commented 4 days ago

IINA Version

1.3.5

Release Channel

Release

macOS Version

15.1.1

Steps to Reproduce

Launch IINA and play an HDR video
Quickly tap "Right" on keyboard (or "Left" if the video is playing for more than 6 seconds)

Expected Behavior

The pixel format should remain the same.

Actual Behavior

The pixel format changes and seems no longer hardware accelerated, which causes lags.

https://github.com/user-attachments/assets/7c7bb7ff-96d5-4c26-9fc5-e5966ed1280a

Crash Report or Process Sample

No response

Logs

No response

Sample Files

https://drive.google.com/uc?id=1bQvSl89OX0rStQLgBc60_6mNCJichMk6

How often can you reproduce the issue

Always.

Required Actions

[X] I tried my best to search from the existing issues (including closed issues) and didn't find duplicated issues.

Optional Actions

[X] I tested with the Nightly build and confirmed the issue is not fixed.
[ ] I tested with the latest version of mpv and confirmed mpv doesn't have this issue.

low-batt commented 3 days ago

Was this encountered with a Mac that has an Intel chip, or one with an Apple Silicon chip?

Amsitram commented 3 days ago

Apple Silicon chip.

low-batt commented 3 days ago

Excellent clear report.

As hardware decoding is chip dependent often it works on one and not the other. I've not tested on my M1 yet. I'm currently using my Intel Mac and it reproduced for me, so both architectures exhibit problems. After taking a closer look I can explain why.

For audio/video playback support IINA uses a library from the mpv project. Certain types of video problems involve IINA code, but this problem is outside of IINA's code base and in the libraries IINA uses. Worse, as you indicated and I have confirmed, this problem still occurs with the development version of IINA which uses upgraded dependencies.

The failure is exactly as you described. Something goes wrong after seeking, hardware acceleration fails and decoding falls back to software as seen in the following clip from the mpv log.

mpv log:

```text [ 19.032][v][file] stream level seek from 127896896 to 127749440 [ 19.035][e][ffmpeg/video] hevc: Could not find ref with POC 272 [ 19.035][e][ffmpeg/video] hevc: Could not find ref with POC 270 [ 19.036][w][ffmpeg/video] hevc: vt decoder cb: output image buffer is null: -17694 [ 19.036][e][ffmpeg/video] hevc: hardware accelerator failed to decode picture [ 19.036][w][vd] Error while decoding frame (hardware decoding)! [ 19.036][e][ffmpeg/video] hevc: Could not find ref with POC 273 [ 19.036][e][ffmpeg/video] hevc: Could not find ref with POC 272 [ 19.037][d][ffmpeg/video] hevc: VideoToolbox decoder needs reconfig, restarting.. [ 19.070][w][ffmpeg/video] hevc: vt decoder cb: output image buffer is null: -12909 [ 19.070][e][ffmpeg/video] hevc: hardware accelerator failed to decode picture [ 19.070][w][vd] Error while decoding frame (hardware decoding)! [ 19.070][e][ffmpeg/video] hevc: Could not find ref with POC 273 [ 19.070][e][ffmpeg/video] hevc: Could not find ref with POC 272 [ 19.070][e][ffmpeg/video] hevc: Could not find ref with POC 275 [ 19.070][d][ffmpeg/video] hevc: VideoToolbox decoder needs reconfig, restarting.. [ 19.078][w][ffmpeg/video] hevc: vt decoder cb: output image buffer is null: -12909 [ 19.079][e][ffmpeg/video] hevc: hardware accelerator failed to decode picture [ 19.079][w][vd] Error while decoding frame (hardware decoding)! [ 19.082][w][vd] Falling back to software decoding. ```

I'm also seeing malformed thumbnails when hovering over the OSC. That is related to the problem at hand.

The problem extends beyond hardware decoding. If you disable hardware decoding there are still problems.

The video file uses a BDAV MPEG transport stream container file format. There are known problems with mpv and FFmpeg when seeking in a .m2ts file. See IINA issue #4785. That issue explains why the OSC thumbnails are malformed as IINA is seeking using FFmpeg to capture those thumbnails.

There is a lot of discussion about this in mpv issues. Many were closed citing the limitations of the MPEG-TS format and recommending remuxing the file into MP4 or Matroska if the ability to seek is needed.

For #4785, instead of reporting this problem to mpv as a bug, I entered a feature request asking for improved MPEG-TS seeking, pointing out that VLC is doing something to overcome the limitations of the MPEG-TS encoding. See the mpv issue https://github.com/mpv-player/mpv/issues/13345. As per the latest comment on #4785 the FFmpeg ticket #9412 is part of the problem.

It is likely these mpv and FFmpeg issues are not getting attention because people use the workaround of remuxing.

Unfortunately IINA has to wait for the upstream projects to provide better support for MPEG-TS encoded files.

Amsitram commented 3 days ago

Thank you so much for taking the time to analyze this and for responding in such detail. Thanks to your report, I understand that the source of the problem lies in the limitations of m2ts format support by the ffmpeg libraries and therefore of mpv and iina. As a result, the problem is probably not directly linked to the HDR nature of the video.

Since you've already made the feature request, all that's left is to wait and hope.

Thanks again for your help :)

low-batt commented 2 days ago

When I have time I try and explain my analysis in detail. This allows people to confirm my understanding of what they are reporting is correct and to point out any mistakes in my analysis. Your understanding is correct, the problem is not related to HDR.

I had a chance today to try remuxing as mpv suggests. I used FFmpeg to copy the streams into a Matroska file:

ffmpeg  -i 00002.m2ts -c copy -map 0 00002.mkv

Running FFmpeg:

```text low-batt@urp 5282$ ffmpeg -i 00002.m2ts -c copy -map 0 00002.mkv ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers built with Apple clang version 16.0.0 (clang-1600.0.26.4) configuration: --prefix=/usr/local/Cellar/ffmpeg/7.1_3 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox libavutil 59. 39.100 / 59. 39.100 libavcodec 61. 19.100 / 61. 19.100 libavformat 61. 7.100 / 61. 7.100 libavdevice 61. 3.100 / 61. 3.100 libavfilter 10. 4.100 / 10. 4.100 libswscale 8. 3.100 / 8. 3.100 libswresample 5. 3.100 / 5. 3.100 libpostproc 58. 3.100 / 58. 3.100 Input #0, mpegts, from '00002.m2ts': Duration: 00:01:04.06, start: 9060.333333, bitrate: 65326 kb/s Program 1 Stream #0:0[0x1011]: Video: hevc (Main 10) (HDMV / 0x564D4448), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn Stream #0:1[0x1100]: Audio: dts (dca) (DTS-HD MA) ([134][0][0][0] / 0x0086), 48000 Hz, stereo, s32p (24 bit) Stream mapping: Stream #0:0 -> #0:0 (copy) Stream #0:1 -> #0:1 (copy) Output #0, matroska, to '00002.mkv': Metadata: encoder : Lavf61.7.100 Stream #0:0: Video: hevc (Main 10) (HDMV / 0x564D4448), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbr, 1k tbn Stream #0:1: Audio: dts (DTS-HD MA) ([1] [0][0] / 0x2001), 48000 Hz, stereo, s32p (24 bit) Press [q] to stop, [?] for help [out#0/matroska @ 0x7fa07a705a40] video:474871KiB audio:13145KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.014271% frame= 1536 fps=823 q=-1.0 Lsize= 488085KiB time=00:01:04.02 bitrate=62453.2kbits/s speed=34.3x low-batt@urp 5282$ ```

When playing the resulting .mkv file with IINA 1.3.5:

Seeking does not result in a hardware decoding failure
Corrupted frames are not displayed
The OSC thumbnails are not corrupted

However, with the development version of IINA I'm seeing these errors from FFmpeg when seeking:

[  78.148][e][ffmpeg/video] hevc: First slice in a frame missing.

From a quick investigation this is looking like a FFmpeg regression. I need to investigate more and confirm that. This did not seem to disrupt playback, but regressions are concerning.

I noticed in the posted screen capture video that IINA's inspector window failed to update to show that hardware decoding was no longer being used. I consider that a defect. I have posted PR #5283 with a fix for that. The fix will need to be reviewed, but it should make it into the next release of IINA. Very helpful that you posted that screen capture.

iina / iina