vidcap file: B-frames or frame paralellism cause video delay relative to audio

[MartinPulec]

Originally posted by @TheSashmo in https://github.com/CESNET/UltraGrid/issues/316#issuecomment-1587527412

Quoted original issue:

Great news the audio does work in this method and I can get the audio from the file as well. But I think a bit of understanding is still needed. Here are my commands:

./UltraGrid-continuous-x86_64.AppImage -s embedded -t switcher -t decklink:0 -t file:output.mp4:loop -c libavcodec:encoder=libx264:bitrate=20000k:preset=ultrafast --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -m 1316 -P 22816 192.168.99.228 --control-port 10000`
./UltraGrid-continuous-x86_64.AppImage -d decklink:device=0 -r embedded 192.168.99.228 -P 22816

When switching from decklink to file, a ton of audio buffer overflow messages a thrown in the decoder logs.

[Decklink display] audio buffer overflow! (1956 written, 348 dropped, 4044 buffer size)
[Decklink display] audio buffer overflow! (2048 written, 256 dropped, 3952 buffer size)
[Decklink display] audio buffer overflow! (900 written, 252 dropped, 5099 buffer size)
[Decklink display] audio buffer overflow! (2052 written, 252 dropped, 3948 buffer size)
[Decklink display] audio buffer overflow! (2238 written, 66 dropped, 3762 buffer size)
[Decklink display] audio buffer overflow! (906 written, 246 dropped, 5093 buffer size)
[Decklink display] audio buffer overflow! (2053 written, 251 dropped, 3947 buffer size)
[Decklink display] audio buffer overflow! (896 written, 256 dropped, 5104 buffer size)
[Decklink display] audio buffer overflow! (2243 written, 61 dropped, 3758 buffer size)
[Decklink display] audio buffer overflow! (2057 written, 247 dropped, 3943 buffer size)
[Decklink display] audio buffer overflow! (2063 written, 241 dropped, 3938 buffer size)
[Decklink display] audio buffer overflow! (2049 written, 255 dropped, 3951 buffer size)
[Decklink display] audio buffer overflow! (2042 written, 262 dropped, 3958 buffer size)
[Decklink display] audio buffer overflow! (2254 written, 50 dropped, 3746 buffer size)
[Decklink display] audio buffer overflow! (2059 written, 245 dropped, 3941 buffer size)
[Decklink display] audio buffer overflow! (2058 written, 246 dropped, 3942 buffer size)
[Decklink display] 150 frames in 5.00624 seconds = 29.9626 FPS
[Decklink display] audio buffer overflow! (2042 written, 262 dropped, 3958 buffer size)
[Decklink display] audio buffer overflow! (891 written, 261 dropped, 5108 buffer size)
[Decklink display] audio buffer overflow! (2261 written, 43 dropped, 3739 buffer size)
[Decklink display] audio buffer overflow! (2050 written, 254 dropped, 3950 buffer size)
[Decklink display] audio buffer overflow! (2056 written, 248 dropped, 3943 buffer size)
[Decklink display] audio buffer overflow! (2045 written, 259 dropped, 3956 buffer size)
[Decklink display] audio buffer overflow! (2253 written, 51 dropped, 3747 buffer size)
[Decklink display] audio buffer overflow! (2248 written, 56 dropped, 3751 buffer size)
[Decklink display] audio buffer overflow! (2250 written, 54 dropped, 3750 buffer size)
[Decklink display] audio buffer overflow! (2257 written, 47 dropped, 3743 buffer size)`

This happens at every time the file loops itself as well.

[MartinPulec]

well, seems a bit weird, it is reproducible with (source here) file capture:

uv -s embedded -t file:Sintel-1080i59.mp4 -d decklink -r embedded      # but not
uv -s embedded -t testcard:mode=Hi59 -d decklink -r embedded

so it seem to be related to file vidcap after all

[TheSashmo]

Testing with your file, and it is no longer crashing. But throwing tons of audio decode errors.

[lavc mp3float @ 0x7f62a8250100] overread, skip -5 enddists: -4 -4 [lavc mp3float @ 0x7f62a810c940] overread, skip -5 enddists: -1 -1 [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62a81eaf00] overread, skip -5 enddists: -4 -4 [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62a81eaf00] overread, skip -6 enddists: -2 -2 [Audio decompress] Empty channel returned ! [Decklink display] audio buffer underflow! [Audio decompress] Empty channel returned ! [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62a8250100] overread, skip -6 enddists: -4 -4 [Audio decompress] Empty channel returned ! [Audio decompress] Empty channel returned ! [Decklink display] audio buffer underflow! [lavc mp3float @ 0x7f62a8250100] overread, skip -6 enddists: -2 -2 [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62a810c940] overread, skip -6 enddists: -1 -1 SSRC 0x64a18796: 3328/3328 packets received (100.0000%), 0 lost, max loss 0 SSRC 0x53d55501: 9728/9728 packets received (100.0000%), 0 lost, max loss 0 [Audio decompress] Empty channel returned ! [Audio decoder] Received 2288640/2288640 B, decoded 172908 samples in 5.01 sec. [Audio decompress] Empty channel returned !

[MartinPulec]

Testing with your file, and it is no longer crashing.

is this related really to this issue? Because I didn't resolve the issue yet and also it didn't exhibit with the crash

[MartinPulec]

The root of the problem is that first GOP of H.264 file is not decoded, eg.:

uv -s embedded -t file:~/Videos/Sintel-1080i59.mp4 | grep 'File cap.'
[File cap.] Video format: 1920x1080 @59.94i, codec UYVY
Error decoding video frame (Resource temporarily unavailable)
[File cap.]  Error while decoding frame (rc == -11): Resource temporarily unavailable.
[ the above 2 lines repeat several times ]

But audio frames are captured normally. When the displaying/playback device on the other side is a DeckLink, it causes audio overflow from the reason that no video frames are to be matched with the audio. Slight improvement can be achieved when disabling multi-threaded decode: -t file:<opts>:threads=0.

[TheSashmo]

Just to be clear what you are saying, because of the GOP thats why the audio and video are causing a problem on the switch? So assuming I make all I frames it should be fine... ?

[TheSashmo]

Testing with your file, and it is no longer crashing.

is this related really to this issue? Because I didn't resolve the issue yet and also it didn't exhibit with the crash

Yes when I test with your file instead of my sample file, everything seems to work fine and no longer crash. I suspect the issue is coming down to how the audio channels are being handled. I will continue to play with it to see if I can make it crash. I belive my sample file was not at the same frame rate and audio channels as my decklink source.

[MartinPulec]

Just to be clear what you are saying, because of the GOP thats why the audio and video are causing a problem on the switch? So assuming I make all I frames it should be fine... ?

Well, P-frames won't matter as well. In theory yes, but A/V delay problem may not just be related to B-frames but also thread-level parallelized decoding, which also adds N frame delay (where N -1is the count of threads; toggled on by default in file vidcap). So using a file without B-frames and running file:<name>:threads=0 wouldn't induce any A/V delay.

[MartinPulec]

Yes when I test with your file instead of my sample file, everything seems to work fine and no longer crash.

But this issue is not dedicated to any crash but only the vidcap file A/V delay problem. Feel free to create an additional issue - I'd do it by myself, but I don't know what was the problem. Also solving [Audio decompress] Empty channel returned ! above might be useful - I cannot solve it because I don't have any steps to reproduce.

[TheSashmo]

I think giving you ssh access to my test setup might be more useful. I believe the setup is the same as it was before, but I would have to check the forwarding rules in the firewall. Let me know if you would be interested.

[MartinPulec]

The issue is closed, but I am still thinking that you may be mixing here some other problem.

The thing is that running uv -s embedded -t file:Sintel-1080i59.mp4 -d decklink -r embedded should newer give [lavc mp3float @ 0x7f62a8250100] overread error. MP3 isn't used anywhere (transmission not compressed, input file has AAC).

I think giving you ssh access to my test setup might be more useful.

From the above, I don't think that I'll be able to reproduce it - using the above-mentioned command should never give me that error even in your setup.

[MartinPulec]

Never mind, I've managed to reproduce it in the end (#327) so there potentially remains only the crash, but in this case I am not able to deduce the problem without additional info.

UPDATE: As far as for the crashing, if you were testing with mp3 compressed transmission, I believe that commit 83f8d559 (2023-06-13) should have turned it really into #327. So in that case, it would be no longer an issue, indeed.