Null/drop frames in AVI files are silently dropped when using default settings

[JohnstonJ]

Problem summary

While capturing a video using VirtualDub, it may insert what is known as a "null frame" or "drop frame" into the resulting AVI file. For example, if the capture device does not provide frames fast enough to keep up with the wall clock, then VirtualDub may insert these to maintain the framerate and keep audio/video synchronized.

Unfortunately, LWLibavVideoSource silently drops these frames entirely, which could lead to issues like audio/video becoming desynchronized, or other unexpected behavior depending on what the user is trying to do.

This is entirely unexpected behavior to me (I didn't even know that null/drop frames were a special thing that require special handling in AVI files until today). Probably many other users might inadvertently handle the data incorrectly, without even realizing it.

Test case

Here is an example AVI file containing 6 frames. Frame 3 is a null/drop frame.

Null-Frame-Example.zip

And here is an example AviSynth script that demonstrates the problem. Note that you will need the Huffyuv codec to be installed for the AVIFileSource to work correctly.

LoadPlugin("LSMASHSource.dll")

# This will yield the expected 6 frames
avi_source = AVIFileSource("Null-Frame-Example.avi").ShowFrameNumber().ConvertToYUY2()

# This yields only 5 frames, which is incorrect.
# The null/drop frame is removed completely.
libav_source = LWLibavVideoSource("Null-Frame-Example.avi").ShowFrameNumber().ConvertToYUY2()

# Clearly highlight how the rendered frames are different
ovl = Overlay(libav_source, avi_source, mode="difference")

StackHorizontal(avi_source, libav_source, ovl)

The output of this script clearly shows that the null/drop frame was removed completely by LWLibavVideoSource, whereas it was handled fine by AVIFileSource.

Additional investigation

Since LWLibavVideoSource is based on FFmpeg, I tried converting the test case to an MKV file using FFV1 codec:

ffmpeg -i Null-Frame-Example.avi -c:a flac -c:v ffv1 Null-Frame-Example.mkv

According to VirtualDub2, this output MKV has 6 frames, but attempting to view frame 3 results in a "frame missing" error, and it does not work. It's clear the issue stems from FFmpeg somehow.

I wondered if other users have encountered similar issues. Yes, they have:

• A user submitted an issue noting that Lagarith decoder does not handle null frames, but it was quickly closed by someone with an unhelpful message that doesn't provide alternative guidance: https://trac.ffmpeg.org/ticket/10173. I tried searching the mailing list, but didn't have much luck locating any related discussion.
• A StackOverflow user mentions using the -vsync cfr option when using FFmpeg as a way to resolve this issue: https://stackoverflow.com/questions/45563023/ffmpeg-decoder-seems-to-drop-frames#comment78343457_45577941
• I tried adding that option to the above FFmpeg command, but I then ran into this problem: https://trac.ffmpeg.org/ticket/8063. It appears that this argument will again yield a valid 6 frame output file, but unfortunately, it duplicates the wrong frame data when handling the null frame! I've added my test case to that Trac issue. See the comment I added there to see how FFmpeg doesn't handle the frame correctly even with -vsync cfr.

Workaround or solution

I found that explicitly specifying the frame rate seems to fix the issue. For example:

libav_source = LWLibavVideoSource("C:\VideoProject\Johnston1\dupframe\Null-Frame-Example.avi", fpsnum=30000, fpsden=1001).ShowFrameNumber().ConvertToYUY2()

I feel this is really, really non-obvious to most average users who have captured an AVI file using VirtualDub and want to work with it an AviSynth.

To help with this, I've edited / proposed a change to the wiki documentation at http://avisynth.nl/index.php?title=LSMASHSource%2FLWLibavVideoSource&diff=13403&oldid=12057

A corresponding pull request to update README files is at https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/pull/68

However, a good argument could be made that this source filter should handle this automatically, so that users don't need to specify the frame rate manually just to make sure their AVI file reads OK, for what is probably a fairly common scenario for users working with captured video.

Output of the test case

Here is the output of this AviSynth script, frame by frame:

[Asd-g]

In nutshell - it's how ffmpeg deals with null frames.

Related https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/28