fraunhoferhhi / vvenc

VVenC, the Fraunhofer Versatile Video Encoder
https://www.hhi.fraunhofer.de/en/departments/vca/technologies-and-solutions/h266-vvc.html
BSD 3-Clause Clear License
899 stars 158 forks source link

Completely broken video stream #379

Closed l-l-l-l-l-l closed 1 month ago

l-l-l-l-l-l commented 1 month ago

I was trying to convert old DVD backups (VOB files) with the library integrated into FFMpeg (after following the steps in your guide).

My first attempt showed a DTS error (well, five) during concatenation (all at the joint) and resulted in a video stream so broken that any player gave up and just didn't bother to read it, showing a handful of errors. I tried again, prejoining the files using MKVToolnix and using a 2-pass approach, and now I didn't get a single error during compression. However, the video file returned the exact same errors and showed the exact same behaviour.

When opening the file with FFPlay, after the FFMPEG data (enabled and disabled features, plus library versions), the following errors are reported:

[libvvdec @ 0000023fbe4a1bc0] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 0000023fbe4a1bc0] Failed to read unit 0 (type 14).
[matroska,webm @ 0000023fbe49bc80] Failed to open codec in avformat_find_stream_info
Invalid value at vps_layer_id[i]: bitstream ended.
Failed to read unit 0 (type 14).
[NULL @ 0000023fbe4a1bc0] Failed to parse extradata.
[libvvdec @ 0000023fbe4a1bc0] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 0000023fbe4a1bc0] Failed to read unit 0 (type 14).

then comes the stream overview, with this description for the video stream: Stream #0:0: Video: vvc (Main 10) (vvc1 / 0x31637676), none(tv, top coded first (swapped)), 720x480, SAR 8:9 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn (default) then two more errors:

[libvvdec @ 0000023fbe51b400] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 0000023fbe51b400] Failed to read unit 0 (type 14).

and then the player opens with an appropiate size, but showing the sound spectrogram instead.

When using the latest VLC Player, which accepts this codec, the error given is that the codec " " is not recognized (no codec name).

When using MPC-BE (VVC compatible), it gives the following error when using the inbuilt splitter: image image

and the following one when using LAV Splitter: image image

If I use FFProbe on the file, it reports this right after the library versions:

[libvvdec @ 000002059d354680] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 000002059d354680] Failed to read unit 0 (type 14).
[matroska,webm @ 000002059d9e7c00] Failed to open codec in avformat_find_stream_info
Invalid value at vps_layer_id[i]: bitstream ended.
Failed to read unit 0 (type 14).
[NULL @ 000002059d354680] Failed to parse extradata.
[libvvdec @ 000002059d354680] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 000002059d354680] Failed to read unit 0 (type 14).

then this after the stream overview (which is the same as in FFPlay):

[libvvdec @ 000002059d3551c0] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 000002059d3551c0] Failed to read unit 0 (type 14).
Could not open codec for input stream 0

Trying to fix the video with -c copy (which seems to work in other instances) now gives this right after the library versions:

[libvvdec @ 00000254d2b38800] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 00000254d2b38800] Failed to read unit 0 (type 14).
[matroska,webm @ 00000254d2aa8b00] Failed to open codec in avformat_find_stream_info
Invalid value at vps_layer_id[i]: bitstream ended.
Failed to read unit 0 (type 14).
[NULL @ 00000254d2b38800] Failed to parse extradata.
[libvvdec @ 00000254d2b38800] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 00000254d2b38800] Failed to read unit 0 (type 14).

(let's note that the hashes after the @ in each line are consistent when using the same program, but change from program to program), and during transcription it returns the following string of errors

[vost#0:0/copy @ 00000254d2b3c480] Invalid DTS: 874 PTS: 475, replacing by guess
[vost#0:0/copy @ 00000254d2b3c480] Invalid DTS: 907 PTS: 208, replacing by guess
[vost#0:0/copy @ 00000254d2b3c480] Invalid DTS: 940 PTS: 74, replacing by guess
[vost#0:0/copy @ 00000254d2b3c480] Invalid DTS: 973 PTS: 7, replacing by guess
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: -26; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 7; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 41; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 74; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 107; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 141; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 174; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 208; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 241; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 274; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 308; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 341; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 374; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 408; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 441; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 475; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 508; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 541; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 575; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 608; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 641; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 675; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 708; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 741; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 775; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 808; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000254d2b3c480] Non-monotonic DTS; previous: 845, current: 842; changing to 845. This may result in incorrect timestamps in the output file.

that don't completely go away even when using -fflags igndt (repeatedly), ending up like this:

[vost#0:0/copy @ 00000159e8b27cc0] Invalid DTS: 874 PTS: 842, replacing by guess
[vost#0:0/copy @ 00000159e8b27cc0] Invalid DTS: 907 PTS: 843, replacing by guess
[vost#0:0/copy @ 00000159e8b27cc0] Invalid DTS: 940 PTS: 844, replacing by guess
[vost#0:0/copy @ 00000159e8b27cc0] Invalid DTS: 973 PTS: 845, replacing by guess
[vost#0:0/copy @ 00000159e8b27cc0] Non-monotonic DTS; previous: 845, current: 842; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000159e8b27cc0] Non-monotonic DTS; previous: 845, current: 843; changing to 845. This may result in incorrect timestamps in the output file.
[vost#0:0/copy @ 00000159e8b27cc0] Non-monotonic DTS; previous: 845, current: 844; changing to 845. This may result in incorrect timestamps in the output file.

I'm going to try again redoing the DTS of the original file, but I don't think every issue stems from that (mainly because the original does not return those errors).

EDIT: I've checked with a different video from a different source, and the errors are the same. It's interesting to note that FFMPlay seems to identify the correct resolution, but outputs no video stream (only the audio spectrogram).

EDIT2: I've tried the following with the shorter video (the speed at which my computer encodes with this codec, mainly due to lack of proper GPU encoding, is quite low): ffmpeg it to yuv, then encode only the video stream both with ffmpeg and vvencapp. Vvencapp was way faster and both ways resulted in a playable video (except that the framerate was wrong on playback, as it often happen with .h2x files), but that video stream cannot be muxed back into a file, as ffmpeg once again gives the error

[libvvdec @ 0000020d2d60bf80] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 0000020d2d60bf80] Failed to read unit 0 (type 14).
[matroska,webm @ 0000020d35ef9a00] Failed to open codec in avformat_find_stream_info
Invalid value at vps_layer_id[i]: bitstream ended.
Failed to read unit 0 (type 14).
[NULL @ 0000020d2d60bf80] Failed to parse extradata.
[libvvdec @ 0000020d2d60bf80] Invalid value at vps_layer_id[i]: bitstream ended.
[libvvdec @ 0000020d2d60bf80] Failed to read unit 0 (type 14).

and MKVmerge cannot identify the bitstream file type, so it errors out.

l-l-l-l-l-l commented 1 month ago

Weird enough, it seems to be a LAV Filters issue. After installing the ones pached for VVCEasy, MPC-BE managed to read the files. FFMpeg still doesn't and MPC-BE's internal filter has nothing to do with LAV (and it should theoretically be compatible).

lehmann-c commented 1 month ago

Thanks for your detailed explanation but we need to know how you created the content.
A detailed log of the command line of the ffmpeg call would be helpful.

l-l-l-l-l-l commented 1 month ago

It seems the main issue is the fact that, since that's what I often do, I encoded the videos as Matroska files, and while Matroska specifications accept VVC (Codec ID: V_MPEGI/ISO/VVC, Codec Name: VVC/H.266), the only widely available muxer, MKVMerge (made by one of the authors of the specifications), still doesn't support it (last mention of VVC was about 3 years ago in the to-do list), so it muxes it under the ID V_QUICKTIME.

For reasons I don't understand, MKV files with that header result in the aforementioned errors while MKV files manually modified with MKVPropEdit to the correct headers don't recognize the codec (you have to change the id back to V_QUICKTIME to fix that).

Using a different container (like a .266 bistream or a TS file), the file seems to play without issue (although you have to manually set the framerate for .266 bitstreams). To replicate the issue, just use the command ffmpeg -i <input> -c:v libvvcenc <output>.mkv (with appropriate values for and ).

Like I say, the issue can be "fixed" for these in fact malformed files by just using VVCEasy's patched LAV Filters, but that's suboptimal, not to mention that it seems to break the DTS of the video due to the mismatch between ID and real codec.

I really should have read the "Playback" section at the wiki for the currently supported file formats, but I would suggest adding a note saying that Matroska files not only are currently unsupported, they are wont to cause these issues.

lehmann-c commented 1 month ago

You didn´t mention that you encoded an mkv container. As vvc support for matroska is currently not implemented in ffmpeg it won´t work properly. Currently only the vvc codec ids are available and are handled in the same way as AVC and HEVC, but i think there is a specific handling missing in the multiplexer. So this is not a particular vvenc issue. As long as there is no vvc encoder integrated into the ffmpeg master, there is only support for ts, mp4 and raw bitstream (.vvc/.266).

adamjw24 commented 1 month ago

Closing as resolved.