search

CESNET / UltraGrid

UltraGrid low-latency audio and video network transmission system
http://www.ultragrid.cz
Other
499 stars 53 forks source link

Feature request or solution that I could not find natively #316

Closed TheSashmo closed 1 year ago

TheSashmo commented 1 year ago

I'd like to be able to switch sources to some file playback when there is no SDI signal detected. I was looking at how to playback recordings to the decklink, and it seems cumbersome. I noticed there is a testcard option with a file name that can be used instead of the default bars. What file is supported there? I was thinking that some sort of audio and bars to be used as a playback source, or beset I can just make a recording of my favourite and have that loop in the background as a switchable source. Any suggestions on how to do this easily? Additionally, if the testcard resolution and setup could match the input format automatically, then it would make the whole process easier.

alatteri commented 1 year ago

Maybe not the exact solution you are looking for, but I have an old NUC and an UltraStudio Monitor 3G that I use with UG, to loop a QT file and output via sdi -d decklink. I then put that SDI signal on the router, and switch as neccessary.

alatteri commented 1 year ago

-t file:path to file:loop -s embedded -d decklink:drift_fix -r analog

TheSashmo commented 1 year ago

Yeah thats what I do now, but that requires me to have another source and a router.

I want to be able to have one device and while I wait for a source then I can switch.

But if -t file works, then I can use the built in switcher.

Just a regular QT file works?

alatteri commented 1 year ago

One thing I used to do, was have a QT file play at UG startup while waiting for incoming stream. But that only works the first time it is launched, not later on if the stream drops.

./UltraGrid.AppImage --tool uv -t file:$INSTALLDIR/splash/$SPLASHSCREEN -d $VIDEODEVICE -r $AUDIODEVICE Here basically, the QT will play until UG gets a stream, then stream takes over.

Just a regular QT file works?

Yeah.

TheSashmo commented 1 year ago

Thanks a bunch @alatteri I will close this. I think I have enough to get the ball rolling in what I need to do. Although I can't get audio to work for some reason. Do you have a sample QT file with multiple audio channels kicking around?

TheSashmo commented 1 year ago

So reopening this. Looks like the file dosnt play well with the switcher. I am sure I am missing something here.

TheSashmo commented 1 year ago

./UltraGrid-continuous-x86_64.AppImage -t file:new.mp4 --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=2 -m 1316 -P 22816 192.168.99.228 (This works without audio, havn't got there yet.)

./UltraGrid-continuous-x86_64.AppImage -t switcher -t file:new.mp4 --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=2 -m 1316 -P 22816 192.168.99.228 (This crashes)

vidcap_switcher_init uv: src/module.c:193: get_root_module: Assertion `node->cls == MODULE_CLASS_ROOT' failed. Backtrace: /tmp/.mount_UltraGKnEpdj/usr/bin/uv(crash_signal_handler+0xa6)[0x56431ecb3aa6] /lib/x86_64-linux-gnu/libc.so.6(+0x430c0)[0x7fd9ddd1c0c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fd9ddd1c03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fd9ddcfb859] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7fd9ddcfb729] /lib/x86_64-linux-gnu/libc.so.6(+0x34006)[0x7fd9ddd0d006] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(+0xb0125)[0x56431ed14125] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(keycontrol_register_key+0xcb)[0x56431ecbb30b] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(playback_register_keyboard_ctl+0x120)[0x56431ecc5010] /tmp/.mount_UltraGKnEpdj/usr/bin/../lib/ultragrid/ultragrid_vidcap_file.so(+0x5276)[0x7fd9d4c54276] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(initialize_video_capture+0xb8)[0x56431ed39588] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(+0xd95be)[0x56431ed3d5be] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(initialize_video_capture+0xb8)[0x56431ed39588] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(main+0x2644)[0x56431eca7ee4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fd9ddcfd0b3] /tmp/.mount_UltraGKnEpdj/usr/bin/uv(_start+0x2e)[0x56431eca937e]

UltraGrid has crashed (Aborted).

alatteri commented 1 year ago

Do you have a sample QT file with multiple audio channels kicking around?

Email has been sent privately with DL link to prevent data scrapers.

TheSashmo commented 1 year ago

Thanks a million @alatteri

TheSashmo commented 1 year ago

I even tried matching the formats of the content of both inputs and it still dosnt work.

./UltraGrid-continuous-x86_64.AppImage -t switcher -t decklink:0:Hi59:v210 -t file:audio_channel_test_clip_single_track_multi_channel.mov -m 1316 -P 22816 192.168.99.228 (crashes)

[Decklink capture] Deprecated syntax used, please use options in format "key=value" [Decklink capture] Using codec: v210 [DeckLink capture] Using device DeckLink SDI Micro [DeckLink capture] bmdDeckLinkConfigCapturePassThroughMode set to: 'pdis' The desired display mode is supported: 1080i59.94 [DeckLink capture] Enable video input: 1080i59.94 uv: src/module.c:193: get_root_module: Assertion `node->cls == MODULE_CLASS_ROOT' failed. Backtrace: /tmp/.mount_UltraGcccBdn/usr/bin/uv(crash_signal_handler+0xa6)[0x55a2191dbaa6] /lib/x86_64-linux-gnu/libc.so.6(+0x430c0)[0x7f331ba0e0c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f331ba0e03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f331b9ed859] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f331b9ed729] /lib/x86_64-linux-gnu/libc.so.6(+0x34006)[0x7f331b9ff006] /tmp/.mount_UltraGcccBdn/usr/bin/uv(+0xb0125)[0x55a21923c125] /tmp/.mount_UltraGcccBdn/usr/bin/uv(keycontrol_register_key+0xcb)[0x55a2191e330b] /tmp/.mount_UltraGcccBdn/usr/bin/uv(playback_register_keyboard_ctl+0x120)[0x55a2191ed010] /tmp/.mount_UltraGcccBdn/usr/bin/../lib/ultragrid/ultragrid_vidcap_file.so(+0x5276)[0x7f3312946276] /tmp/.mount_UltraGcccBdn/usr/bin/uv(initialize_video_capture+0xb8)[0x55a219261588] /tmp/.mount_UltraGcccBdn/usr/bin/uv(+0xd95be)[0x55a2192655be] /tmp/.mount_UltraGcccBdn/usr/bin/uv(initialize_video_capture+0xb8)[0x55a219261588] /tmp/.mount_UltraGcccBdn/usr/bin/uv(main+0x2644)[0x55a2191cfee4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f331b9ef0b3] /tmp/.mount_UltraGcccBdn/usr/bin/uv(_start+0x2e)[0x55a2191d137e]

TheSashmo commented 1 year ago

Just experimenting with working sources, and I am confused for sure, but wouldnt the excl_init and fallback both be an option that could run at the same time?

vidcap_switcher_init [switcher] Unknown initialization option! switcher capture Usage --control-port -t switcher[:excl_init][:fallback] -t -t ....]

is a configuration of device to be switched specifies port which should be used to control switching excl_init - devices will be initialized after switching to and deinitialized after switching to another fallback - in case that capture doesn't return a frame (in time), capture from next available device(s) When I test it, it throws an errors saying [switcher] Options "excl_init" and "fallback" are mutualy incompatible! But reading it: excl_init - devices will be initialized after switching to and deinitialized after switching to another Which to my thoughts would be the source is there, but only initialize it after its switched to, maybe like playing a file and it starts it from the beginning instead of looping it. And then the following: fallback - in case that capture doesn't return a frame (in time), capture from next available device(s) Would be used when the first soruce i.e. decklink has nothing there, then switch the file instead. I am sure its me, but thats just the way I think it should work. So when I run the command above: ./UltraGrid-continuous-x86_64.AppImage -t switcher:excl_init -t decklink:0:24ps:v210 -t file:audio_channel_test_clip_single_track_multi_channel.mov -m 1316 -P 22816 192.168.99.228 I can get UG to start working, but the moment I switch to that source it crashes. Custom keybindings: '1' - switch to video input 1 '2' - switch to video input 2 [g] indicates that that option is guarded (Ctrl-x needs to be pressed first) [switcher] 121 frames in 5.04153 seconds = 24.0007 FPS 202 Accepted [switcher] Switched to device 1. uv: src/module.c:193: get_root_module: Assertion `node->cls == MODULE_CLASS_ROOT' failed. Backtrace: /tmp/.mount_UltraGBefhlc/usr/bin/uv(crash_signal_handler+0xa6)[0x559ba5cfeaa6] /lib/x86_64-linux-gnu/libc.so.6(+0x430c0)[0x7f9db994d0c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f9db994d03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f9db992c859] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f9db992c729] /lib/x86_64-linux-gnu/libc.so.6(+0x34006)[0x7f9db993e006] /tmp/.mount_UltraGBefhlc/usr/bin/uv(+0xb0125)[0x559ba5d5f125] /tmp/.mount_UltraGBefhlc/usr/bin/uv(keycontrol_register_key+0xcb)[0x559ba5d0630b] /tmp/.mount_UltraGBefhlc/usr/bin/uv(playback_register_keyboard_ctl+0x120)[0x559ba5d10010] /tmp/.mount_UltraGBefhlc/usr/bin/../lib/ultragrid/ultragrid_vidcap_file.so(+0x5276)[0x7f9db0885276] /tmp/.mount_UltraGBefhlc/usr/bin/uv(initialize_video_capture+0xb8)[0x559ba5d84588] /tmp/.mount_UltraGBefhlc/usr/bin/uv(+0xd929a)[0x559ba5d8829a] /tmp/.mount_UltraGBefhlc/usr/bin/uv(vidcap_grab+0x25)[0x559ba5d84745] /tmp/.mount_UltraGBefhlc/usr/bin/uv(+0x102b59)[0x559ba5db1b59] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f9dba150609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f9db9a29163]
TheSashmo commented 1 year ago

Playing with the switcher more and I am able to get switching between test card and decklink quite well, but when not in decklink I get tons of errors on the console about dropping audio packets. I am not sure if thats the correct behavior or not?

-9.04/-1.89, [10] -9.18/-1.89, [11] -9.11/-1.89, [12] -9.04/-1.89, [13] -9.38/-1.89, [14] -9.16/-1.89, [15] -9.32/-1.89 dBFS RMS/peak [DeckLink capture] Dropping audio packet, queue full. Last message repeated 18 times [switcher] 150 frames in 5.00503 seconds = 29.9699 FPS [DeckLink capture] Dropping audio packet, queue full. Last message repeated 130 times [Audio sender] Sent 240225 samples in last 5.005317 seconds. [Audio sender] Volume: [0] -9.36/-1.89, [1] -9.15/-0.01, [2] -9.25/-1.89, [3] -9.09/-1.89, [4] -9.04/-1.47, [5] -9.39/-1.24, [6] -9.23/-1.89, [7] -9.42/-1.72, [8] -9.36/-1.89, [9] -9.15/-1.89, [10] -9.26/-0.11, [11] -9.10/-1.89, [12] -9.04/-1.89, [13] -9.39/-1.89, [14] -9.23/-0.71, [15] -9.41/-1.89 dBFS RMS/peak [DeckLink capture] Dropping audio packet, queue full. Last message repeated 18 times [switcher] 150 frames in 5.00501 seconds = 29.97 FPS [DeckLink capture] Dropping audio packet, queue full. Last message repeated 130 times

MartinPulec commented 1 year ago

We've tried to replicate the issue with @mpiatka but unsuccessfully. Isn't it possible that this also was a temporarily broken version? Current version (066d1f4) seems to work without problems: 

UltraGrid-continuous-x86_64.AppImage -t switcher -t testcard:mode=VGA:pattern=ebu_bars -t testcard:mode=VGA:pattern=smpte_bars -d gl

(and alternating by pressing '1' and '2').

TheSashmo commented 1 year ago

Switching seems to work fine when using testcards, but when using the decklink as a source I get the following messages Last message repeated 130 times [DeckLink capture] Dropping audio packet, queue full.

And when trying to use a file as a source to switch to and from, UG crashes as I stated above.

MartinPulec commented 1 year ago

Playing with the switcher more and I am able to get switching between test card and decklink quite well, but when not in decklink I get tons of errors on the console about dropping audio packets. I am not sure if thats the correct behavior or not?

Basically yes, the switcher captures from one card or another. When DeckLink is not selected, audio frames are also not captured resulting in the error (because it overflows after some time). It depends how problematic this is for you - from my point of view it is, if not intended, at least expected behavior. But I guess that it may be annoying.

TheSashmo commented 1 year ago

Thats fine. Just making sure thats the expected behaviour.

TheSashmo commented 1 year ago

When testing with audio, messages about the source show:

[testcard] capture set to 1920x1080 @59.94i, codec v210, bpc 10, pattern: ebu_bars, audio off [testcard] capture set to 1920x1080 @59.94i, codec v210, bpc 10, pattern: smpte_bars, audio off Audio sending started.

But clearly there is audio on the bars.

TheSashmo commented 1 year ago

Additionally, the audio on the bars is bad and when doing switching all audio is broken... I can start clean on decklink, but after the first switch it gets bad. I can DM you a link if you like so you can hear/see what I being output?

./UltraGrid-continuous-x86_64.AppImage -t switcher -t testcard:1920:1080:59.94i:v210 -t decklink:device=0 -s embedded -t testcard:1920:1080:59.94i:v210 -t testcard:1920:1080:59.94i:v210:pattern=ebu_bars -t testcard:1920:1080:59.94i:v210:pattern=smpte_bars --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -m 1316 -P 22816 192.168.99.228

This example crashes the moment I switch to decklink.

But if I initiate decklink first I can do the switching just fine.

BUT, the audio is broken on the bars. I can start clean with no audio issues on decklink if I use that as the first item, but the moment I switch to bars, all the audio get bad and even the decklink is bad.

TheSashmo commented 1 year ago

I found my issue, first I wasn't passing the command correctly.

Lastly is it safe to assume that the switcher "as per the wiki" only allows video source switching and not audio with that source, correct?

./UltraGrid-continuous-x86_64.AppImage -t switcher -t decklink:device=0 -s embedded -t testcard:1920:1080:59.94i:v210 -t testcard:1920:1080:59.94i:v210:pattern=ebu_bars -t testcard:1920:1080:59.94i:v210:pattern=smpte_bars -c libavcodec:encoder=libx264:bitrate=20000k:preset=ultrafast --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -m 1316 -P 22816 192.168.99.228 --record

Edit: If I place the -s embedded for the decklink audio in the chain, then it will take that audio and use it, but once I switch to the testcard as a source, the audio goes crazy. This is why I was getting confused, "assuming" that it was audio follow video, but in fact it isn't. Would be nice feature to have.

Update: while testing, its seems that some audio is trying to be processed, I did a sample video to show what I see on a scope. Its hard to hear the good audio, but you can definitely hear the bad audio and the audio gets completely broken on the decoder side after the I switch to some other source. Here is the decoded video recording: https://www.dropbox.com/s/mv2omhsn9ilf025/IMG_0921.MOV?dl=0 and here is the source recording from ug: https://www.dropbox.com/s/ycvckcwo5yqmoll/content.mp4?dl=0

Update2: trying to playback the content file as a source for fun and encoder crashes as above.

MartinPulec commented 1 year ago

Lastly is it safe to assume that the switcher "as per the wiki" only allows video source switching and not audio with that source, correct?

Not really, it should actually work. I've slightly looked into the issue and there seem to be 2 another possible problems:

I'll try to look at the issue # 2 first in next day(s).

UPDATE: I've converted first problem to issue #318

TheSashmo commented 1 year ago

I can confirm that in fact when passing additional audio after each -t command it will only take the first audio and not pass the audio. At least that was in my test setup. Remember that this all started when testing -t file as a source when using switcher and it crashes.

MartinPulec commented 1 year ago

I can confirm that in fact when passing additional audio after each -t command it will only take the first audio and not pass the audio. At least that was in my test setup.

I am not sure if I understand entirely. To clarify, how (I suppose) it works right now - selecting analog/embedded/aesebu selects that input for all video inputs, selecting different audio device (eg. -s testcard) sets testcard. But, there is no support for multiple audio devices in general, so only the last -s parameter is used. It however works for the audio bundled with video..

Remember that this all started when testing -t file as a source when using switcher and it crashes.

Do you have steps to reproduce? Ideally a minimal (non-)working example. Hard to say if it is supposed to work or not without more info.

MartinPulec commented 1 year ago

embedded audio for vidcap testcard with framerates using 1001 fractions (tested 23.98, 29.97 and 59.94) should now be working

MartinPulec commented 1 year ago

please see also #318 for further comments

MartinPulec commented 1 year ago

./UltraGrid-continuous-x86_64.AppImage -t switcher -t file:new.mp4 --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=2 -m 1316 -P 22816 192.168.99.228 (This crashes)

vidcap_switcher_init uv: src/module.c:193: get_root_module: Assertion `node->cls == MODULE_CLASS_ROOT' failed.

This should be already fixed with a24e194d. File capture module was just not connected to root when initialized with switcher.

TheSashmo commented 1 year ago

Awesome! Will the audio follow the video when switching, sorry for duplicate question.

MartinPulec commented 1 year ago

yes, with some limitations (see #318)

TheSashmo commented 1 year ago

Well.... You are correct. It's not working, its just limited. Any thoughts on if this is possible to work, I think the feature is called "AFV" audio follows video. Its used on production switchers all the time.

MartinPulec commented 1 year ago

I am sorry but can you be more specific what exactly is not working?

TheSashmo commented 1 year ago

I was reading #318 and it says that portaudio works but not decklink audio. The point of having a switcher is being able to not only switch sources with a static audio source, but also have the ability to switch sources with different audio sources. #318 says that it dosnt work. Or at least thats how I read it.

MartinPulec commented 1 year ago

No, it says:

uv -s embedded -t switcher -t testcard -t decklink -t decklink:device=1 [..] works

It also switches embedded audio with video devices.

uv -t switcher -s embedded -t decklink -s analog -t decklink:device=1 doesn't (only one -s can be used at this time)

TheSashmo commented 1 year ago

I think we are confusing things. Example: I have two devices and one file. I want to switch between the file and the decklink devices, but as I switch the devices the audio should go along with the switching. i.e. device 1 file, then device 2 embedded, device 3 embedded, and follow along the audio as I switch from device to device. This is called AFV. (Audio Follow Video)

Your example shows switching devices, which works 100%. What dosen't work is audio switching to the respective device. So when its switched form device 1 to device 2, device 2 audio should be playing, and then when switching back to device 1, should now be using device 1 audio.

MartinPulec commented 1 year ago

Your example shows switching devices, which works 100%. What dosen't work is audio switching to the respective device.

But it does, eg.: uv -s embedded -t switcher -t decklink:device=1 -t decklink:device=2 switches between decklink 1 and 2 including audio. (If I understand correctly what you want.) Also, in terms of UltraGrid, audio for file capture is "embedded" so uv -s embedded -t switcher -t decklink:device=1 -t file:in.mp4:loop should work as well.

Please follow this specific problem in #318 with further comments not to mess this discussions with multiple issues. We can discuss more examples/use cases there. I've also slightly updated examples/description there.

And sorry for the issues, the switcher capture is seldom used so not everything is (currently) handled as it should be.

TheSashmo commented 1 year ago

Great news the audio does work in this method and I can get the audio from the file as well. But I think a bit of understanding is still needed. Here are my commands:

./UltraGrid-continuous-x86_64.AppImage -s embedded -t switcher -t decklink:0 -t file:output.mp4:loop -c libavcodec:encoder=libx264:bitrate=20000k:preset=ultrafast --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=16 -m 1316 -P 22816 192.168.99.228 --control-port 10000

./UltraGrid-continuous-x86_64.AppImage -d decklink:device=0 -r embedded 192.168.99.228 -P 22816

When switching from decklink to file, a ton of audio buffer overflow messages a thrown in the decoder logs. [Decklink display] audio buffer overflow! (1956 written, 348 dropped, 4044 buffer size) [Decklink display] audio buffer overflow! (2048 written, 256 dropped, 3952 buffer size) [Decklink display] audio buffer overflow! (900 written, 252 dropped, 5099 buffer size) [Decklink display] audio buffer overflow! (2052 written, 252 dropped, 3948 buffer size) [Decklink display] audio buffer overflow! (2238 written, 66 dropped, 3762 buffer size) [Decklink display] audio buffer overflow! (906 written, 246 dropped, 5093 buffer size) [Decklink display] audio buffer overflow! (2053 written, 251 dropped, 3947 buffer size) [Decklink display] audio buffer overflow! (896 written, 256 dropped, 5104 buffer size) [Decklink display] audio buffer overflow! (2243 written, 61 dropped, 3758 buffer size) [Decklink display] audio buffer overflow! (2057 written, 247 dropped, 3943 buffer size) [Decklink display] audio buffer overflow! (2063 written, 241 dropped, 3938 buffer size) [Decklink display] audio buffer overflow! (2049 written, 255 dropped, 3951 buffer size) [Decklink display] audio buffer overflow! (2042 written, 262 dropped, 3958 buffer size) [Decklink display] audio buffer overflow! (2254 written, 50 dropped, 3746 buffer size) [Decklink display] audio buffer overflow! (2059 written, 245 dropped, 3941 buffer size) [Decklink display] audio buffer overflow! (2058 written, 246 dropped, 3942 buffer size) [Decklink display] 150 frames in 5.00624 seconds = 29.9626 FPS [Decklink display] audio buffer overflow! (2042 written, 262 dropped, 3958 buffer size) [Decklink display] audio buffer overflow! (891 written, 261 dropped, 5108 buffer size) [Decklink display] audio buffer overflow! (2261 written, 43 dropped, 3739 buffer size) [Decklink display] audio buffer overflow! (2050 written, 254 dropped, 3950 buffer size) [Decklink display] audio buffer overflow! (2056 written, 248 dropped, 3943 buffer size) [Decklink display] audio buffer overflow! (2045 written, 259 dropped, 3956 buffer size) [Decklink display] audio buffer overflow! (2253 written, 51 dropped, 3747 buffer size) [Decklink display] audio buffer overflow! (2248 written, 56 dropped, 3751 buffer size) [Decklink display] audio buffer overflow! (2250 written, 54 dropped, 3750 buffer size) [Decklink display] audio buffer overflow! (2257 written, 47 dropped, 3743 buffer size)

This happens at every time the file loops itself as well.

The next issue is it seems that the file player controls are not accessible/working:

Custom keybindings: ' ' - playback pause '1' - switch to video input 1 '2' - switch to video input 2 'q' - playback quit Up - playback seek +60s Down - playback seek -60s Right - playback seek +10s Left - playback seek -10s Page-Up - playback seek +600s Page-Dn - playback seek -600s

Its not too important, but I felt worth bringing up.

Next is, how the file is being handled as a source. Based on the documentation the sources should all be initialized at the start of the command line. I can validate that if I am using multiple vidcap sources that is correct, but once I add the file, no matter if I use the excl_init - devices will be initialized after switching to and deinitialized after switching to another or not, every time I switch to the file it is initialized, which causes a drop to black in the video. For the case of testing, the same source and the same format is being used, and it still drops to black, from what I can determine from the encoder logs, its reinitialized each time it switches sources.

The last issue, but this is probably only a me issue, but I used this example to make a recording: ./UltraGrid-continuous-x86_64.AppImage -t decklink:device=0 -s embedded -c libavcodec:encoder=libx264:bitrate=20000k:preset=ultrafast --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=2 -m 1316 -P 22816 192.168.99.228 --record

Then I follow the instructions from here: https://github.com/CESNET/UltraGrid/wiki/Recording-and-Playback

And if I am using two channels as my capture, everything works just fine. But If I have more than 2, eg. 16 channels, following the same instructions causes UG to crash each time when trying to initialize switcher or not. I am sure this is something to do with how I am making the recording and using ffmpeg to convert it. I tested with another file that was shared by @alatteri which works very well with the file and multiple audio channels, BUT after letting the file play for a bit, then trying to switch between sources, the decoder crashes.

alatteri commented 1 year ago

When switching from decklink to file, a ton of audio buffer overflow messages a thrown in the decoder logs.

Maybe you still have to specify :drift-fix for the decklink?

TheSashmo commented 1 year ago

Actually seems to make it worse during the switch. @alatteri

Update: still causes the decoder to crash when I switch sources.

[Decklink display] audio buffer overflow! (992 written, 160 dropped, 5008 buffer size) [Decklink display] audio buffer overflow! (2051 written, 251 dropped, 3949 buffer size) SSRC 0x11a4c9be: 1792/1792 packets received (100.0000%), 0 lost, max loss 0 [Decklink display] audio buffer overflow! (1089 written, 62 dropped, 4910 buffer size) [Decklink display] 120 frames in 5.00539 seconds = 23.9742 FPS [Audio decoder] Received 1277952/1277952 B, decoded 239467 samples in 5.01 sec. [Audio decoder] Volume: [0] -99.92/-90.31, [1] -99.88/-90.31, [2] -21.18/-2.37, [3] -33.02/-18.00, [4] -99.91/-90.31, [5] -99.95/-90.31, [6] -99.95/-90.31, [7] -99.92/-90.31 dBFS RMS/peak SSRC 0x41e36220: 7040/7040 packets received (100.0000%), 0 lost, max loss 0 SSRC 0x11a4c9be: 1664/1664 packets received (100.0000%), 0 lost, max loss 0 New incoming audio format detected: 48000 Hz, 16 channels, 32 bits per sample, codec MP3 uv: src/audio/codec/libavcodec.c:538: libavcodec_decompress: Assertionchannel->data_len > 0' failed. Backtrace: /tmp/.mount_UltraGHNFGFM/usr/bin/uv(crash_signal_handler+0xa6)[0x55fe7ca91c96] /lib/x86_64-linux-gnu/libc.so.6(+0x430c0)[0x7f570838e0c0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f570838e03b] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f570836d859] /lib/x86_64-linux-gnu/libc.so.6(+0x22729)[0x7f570836d729] /lib/x86_64-linux-gnu/libc.so.6(+0x34006)[0x7f570837f006] /tmp/.mount_UltraGHNFGFM/usr/bin/../lib/ultragrid/ultragrid_acompress_libavcodec.so(+0x4427)[0x7f5706bad427] /tmp/.mount_UltraGHNFGFM/usr/bin/uv(_Z22audio_codec_decompressP17audio_codec_stateP12audio_frame2+0x173)[0x55fe7cacd3c3] /tmp/.mount_UltraGHNFGFM/usr/bin/uv(decode_audio_frame+0xaae)[0x55fe7caadcbe] /tmp/.mount_UltraGHNFGFM/usr/bin/uv(pbuf_decode+0x92)[0x55fe7caab3a2] /tmp/.mount_UltraGHNFGFM/usr/bin/uv(+0x83246)[0x55fe7cac6246] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f5708b91609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f570846a163]

UltraGrid has crashed (Aborted).

Please send a bug report to address ultragrid-dev@cesnet.cz. You may find some tips how to report bugs in file doc/REPORTING_BUGS.md distributed with UltraGrid (or available online at https://github.com/CESNET/UltraGrid/blob/master/doc/REPORTING-BUGS.md). Aborted (core dumped) `

MartinPulec commented 1 year ago

-t file:output.mp4:loop

If I am not mistaken, in only occurs with H.264, not eg. with MJPEG, correct? If so, then see #322.

The next issue is it seems that the file player controls are not accessible/working:

You mean in the swithcher? It seems so, I'll look into it. #320

Based on the documentation the sources should all be initialized at the start of the command line.

yes, it is, as you correctly mentioned, without excl_init it is so (and don't bother with that option - it has been added there for a reason but unless you really need it for some reason, you can forgot that)

every time I switch to the file it is initialized, which causes a drop to black in the video.

Do you mean that the file jumps to beginning? I've tested and it doesn't seem to be the case, please try (source file Sintel to 1080i59 here):

uv -t switcher -t file:Sintel-1080i59.mp4 -t testcard:mode=Hi59 -d gl -c libavcodec:codec=H.264 # or libavcodec:q=5

With H.264 it stops for a while (and it doesn't for MJPEG), so blank frame may be produced by output decklink? but it should keep last frame, as far as i know. In this particular case, compression reintialized because change in mode (progressive vs. interlaced, fixed #321), but it can also be due to different codec.

The last issue [...] causes UG to crash

can you create a minimal working examle? I've tried:

uv -a channels=16 -s testcard --record=rec           # and also:
uv -a channels=16 --audio-codec=MP3:sample_rate=48000:bitrate=256k  -s embedded -t decklink --record

neither of the above crashed for me with current continuous. I've also checked the testcard-recorded WAV and it looked ok.

MartinPulec commented 1 year ago

UltraGrid has crashed (Aborted).

fixed (perhaps), but I won't expect the option to help now, anyways

TheSashmo commented 1 year ago

So I believe the main issue is a format mismatch. Using your file and testing there is no crashing and no dropping to black, so all works as it should as long as I am 100% matching my sources. I think the main confusion is the recorded file that I was assuming would have been good enough for my test since it came from actually recording the UG output, but clearly my ffmpeg process breaks it. I will use your file as the example of how a multiple channel source should look like moving forward.

I do however still have problems. At first I thought the problem was the audio channels. File has 6, decklink has 16, command line says encode 16. So maybe that was my issue, so I dropped my audio layout to 2 channels. But I still have an issue. I can let the file play....... Ok so I will stop here for a moment, I was typing this and testing at the same time, BUT I think I finally found the problem.

My command is here: Encode ./UltraGrid-continuous-x86_64.AppImage -s embedded -t switcher -t file:Sintel-1080i59.mp4:loop -t decklink:0 -c libavcodec:encoder=libx264:bitrate=20000k:preset=ultrafast --audio-codec=MP3:sample_rate=48000:bitrate=256k --audio-capture-format channels=2 -m 1316 -P 22816 192.168.99.199 --control-port 10000

Decode ./UltraGrid-continuous-x86_64.AppImage -d decklink:device=0:drift_fix -r embedded 192.168.99.199 -P 22816

So I set two channels of audio for the encode, but look at the decoder log output, says I have 6 which is what the file actually has, that should not be like that right?: z): 1960 / Average Buffer: 2742 / Average Added Frames: 1193.44 / Max time diff audio (ms): 41 / Min time diff audio (ms): 0 [Audio decoder] Volume: [0] -21.81/-5.36, [1] -21.73/-6.26, [2] -18.87/-0.39, [3] -37.70/-15.63, [4] -25.41/-8.05, [5] -25.46/-7.07 dBFS RMS/peak [Decklink display] 150 frames in 5.00748 seconds = 29.9552 FPS SSRC 0x04fcd518: 9984/9984 packets received (100.0000%), 0 lost, max loss 0 SSRC 0x65b7443d: 1280/1280 packets received (100.0000%), 0 lost, max loss 0 [Audio decoder] Received 958464/958464 B, decoded 239611 samples in 5.01 sec. [Audio decoder] Volume: [0] -35.79/-13.44, [1] -33.89/-10.73, [2] -35.12/-14.35, [3] -51.16/-33.69, [4] -46.08/-28.76, [5] -45.97/-29.13 dBFS RMS/peak [Decklink display] 150 frames in 5.00184 seconds = 29.9889 FPS SSRC 0x04fcd518: 9728/9728 packets received (100.0000%), 0 lost, max loss 0 SSRC 0x65b7443d: 1280/1280 packets received (100.0000%), 0 lost, max loss 0 Played 2944000 audio samples in 78.836815 seconds (37342.959688 samples per second). Audio dec stats (cumulative): 2554 played / 2558 total audio frames Pbuf: total 80805/80896 packets received (99.88751%). Video dec stats (cumulative): 1840 total / 1784 disp / 56 drop / 0 corr / 0 missing. Pbuf: total 4647936/4647936 packets received (100.00000%).

Ignore the rest of this but I will leave it for historical data! LOLOL.

Starting the command line with switcher and file as first option.

Video dec stats (cumulative): 5580 total / 5541 disp / 38 drop / 0 corr / 1 missing. Audio dec stats (cumulative): 7682 played / 7951 total audio frames [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62a8218bc0] overread, skip -6 enddists: -5 -5 [Audio decompress] Empty channel returned ! [Decklink display] audio buffer underflow! [Decklink display] 150 frames in 5.00536 seconds = 29.9679 FPS [Audio decompress] Empty channel returned ! [Audio decompress] Empty channel returned ! [lavc mp3float @ 0x7f62aa0cfa00] overread, skip -8 enddists:

I still get tons of error messages as per #322 and from your previous note in #318 about the messages of the [DeckLink capture] Dropping audio packet, queue full. I do think those should be truncated.

MartinPulec commented 1 year ago

So I set two channels of audio for the encode, but look at the decoder log output, says I have 6 which is what the file actually has, that should not be like that right?:

323

MartinPulec commented 1 year ago

Okay, all sub-issues are fixed from my perspective, let me know if I am something missing.

Returning back to the original question.

noticed there is a testcard option with a file name that can be used instead of the default bars. What file is supported there?

It used to be simply a RAW file with data corresponding given props (color_spec + size). Now it can also handle Y4M (for YUV; but only first frame is taken). PAM or PNM (RGB). But file capture is perhaps more eligible, indeed.

using switcher

I believe that you could use something like this:

uv -s embedded -t switcher:fallback -t decklink:device=1 -t file:<fallback_f>:loop

correct?

But there was still one catch with this approach. When DeckLink is the first device, but not having a signal, DeckLink's timeout caused that the file capture produced just 1/2-1/3 FPS (i.e. 10-20 FPS for 59.94i). I've provisionally fixed this with commit 1db584e6 which utilizes the fact, that DeckLink locks its clock to the signal and triggers nosignal exactly when the frame should be presented, even if there is none. It needs a bit testing but I think it could work.

UPDATE (7th Aug '23): fixed commit number

MartinPulec commented 1 year ago

I believe that the issue has been resolved with above changes and fixes in sub-issues. You can reopen if disagree (or open an another issue to report a more specific related problem).