Encoding drops the first two frames of all videos

[galenlynch]

After fixing #240 in a PR that I'm preparing, many of the VideoIO tests would fail because flushing the decoder returned the frames cached by the decoder, and the tests were written assuming these frames would be dropped. Of particular note, this line failed, returning 98 frames instead of 96 (ffprobe also counted 98):

https://github.com/JuliaIO/VideoIO.jl/blob/600f60cb33fbb5836138524ceb5c71d3b2e116cf/test/avio.jl#L217

I no longer really believed the comment that the (claimed four, now two) missing frames are just a feature of h.264 and wondered if this should really by 100 frames, not 98. Indeed, writing each of the frames to individual PNG files and then encoding them with ffmpeg using the same settings produced a video with all 100 frames. By reading the frames back and inspecting each I saw that the first two frames were always dropped.

It seems like the whole encoding.jl file in VideoIO is more or less a direct translation of the FFMPEG encoding example into Julia, and I couldn't find any discrepancy in that translation (except not setting the GOP number, which I think is also causing problems). However, to be honest I didn't really understand what sort of file the FFMPEG example, and by extension VideoIO, was generating. The encodevideo function in VideoIO then passes the resulting file to the ffmpeg binary to be muxed, but otherwise the first file (temp.stream by default) should be identical to what the FFMPEG example produces. If I used ffprobe to look at either the temp.stream file or the muxed version of it, it would always throw the warnings "Invalid NAL unit 0, skipping" and "decoding for stream 0 failed" in the raw stream, or "Invalid NAL unit 8, skipping" in the muxed file. In both cases, ffprobe seemed to be able to make sense of 98 of the frames, despite these errors. After some googling, I think this intermediate file is a MPEG elementray stream.

In the FFMPEG decoding example, and VideoIO, a sequence of bytes is written to the end of the file with a comment "add sequence end code to have a real MPEG file":

https://github.com/JuliaIO/VideoIO.jl/blob/600f60cb33fbb5836138524ceb5c71d3b2e116cf/src/encoding.jl#L206

If you look at the documentation for FFMPEG, this constant is called SEQ_END_CODE, and is right next to SEQ_START_CODE, which is also described in the wikipedia elementaray stream page as the start of the header for MPEG-2 video elementary streams.

Adding write(io, 0x000001b3) after the following line:

https://github.com/JuliaIO/VideoIO.jl/blob/600f60cb33fbb5836138524ceb5c71d3b2e116cf/src/encoding.jl#L252

eliminates the ffprobe errors, and causes all 100 frames make it into the resulting video file!

So it seems like there is a bug in the FFMPEG example, and that bug moved into encoding.jl, though the fix is simple.

I will include this fix in the PR that I'm preparing.

[IanButterworth]

This is awesome. Thank you.

I'm keen to see if this also fixes #208 where videos play back out of order in QuickTime (but not vlc)

[galenlynch]

I declared victory too soon. While the MPEG-2 stream has 100 frames, after muxing into MPEG-4, there are still only 98. If I run the mux command on the command line, I see that there is a warning message that is normally hidden when using the binary from VideoIO:

[mp4 @ 0x55f4fdd94540] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0x55f4fdd94540] pts has no value

And furthermore the resulting mp4 has 97 frames, not 98. If I use ffmpeg to re-encode the mp4 with the loglevel set to debug, the output contains the following three lines, which seem related to the three missing frames:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e642e74c00] drop a frame at curr_cts: 0 @ 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e642e74c00] drop a frame at curr_cts: 48000 @ 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e642e74c00] drop a frame at curr_cts: 98001 @ 2

This seems to be related to a edit list for the mp4 file, and suggests that the timestamps might be causing the muxer to drop frames. If I really crank up the verbosity I see uneven sample durations, which possibly supports this hypothesis:

sample_count=1, sample_duration=48000
sample_count=1, sample_duration=50001
sample_count=1, sample_duration=50000
sample_count=1, sample_duration=50001
sample_count=2, sample_duration=50000
...

If I re-encode the mp4 with ffmpeg using the -ignore-editlist true option, then all 100 frames are there:

ffmpeg -ignore_editlist true -i <original_output>.mp4 -c copy <fixed_output>.mp4

So the muxer seems to be generating an edit list that throws out frames, for reasons I don't understand.

The warning messages make me think the pts values for the frames are somehow being omitted.

[galenlynch]

Still struggling with this, I think I'm going to give up on it for the moment. I put a SO question up about it since I think I'm out of my league with trying to understand when FFMPEG decides to make multiple edit lists in mp4 containers, and what controls the size of these edits lists, if that really is the core problem here.

Would be overjoyed if anyone has thoughts on the problem.