search

DusteDdk / u64view

Viewer for Ultimate64 VIC network stream
Other
36 stars 6 forks source link

Audio/Video goes out of sync when using -o #3

Closed DusteDdk closed 3 years ago

DusteDdk commented 4 years ago

After some minutes, the audio and video goes out of sync, original description:

"Hey, I was just wondering if you've verified that video and audio stay in sync. I've done some test captures and after about 5 minutes, audio is behind by quite a bit. Changing the sample rate to something lower than the documented 47983Hz does improve "

MeAndMyRhythmbox commented 3 years ago

Hi have you made any progress on the sync issues? I don't use the -o option, but for me if I start the viewer with streams auto enabled (-u) the audio is always lagging behind... Pressing S to stop and then again to restart the streams gets things back in sync, but audio soon lags over time :(

FYI my sound card is set to 44.1KHz. Would love to get to the bottom of this.

I find the 8-Bit Guy's simple delay testing program that flashes the screen and beeps at the same time is useful: http://gurce.net/c64mini_files/delaytest.d64

DusteDdk commented 3 years ago

Hi have you made any progress on the sync issues?

Hi, I've not had any issues with this, I'm wondering if it depends on the platform, are you using the program on Linux, Mac or Windows ?

I am aware that the sample rate I'm getting is not exactly 44.1khz, so it does not surprise me too much if there is a problem, I will need to implement resampling in that case. Unfortunately, life gets in the way at the moment, so it might be a while, but I appreciate the feedback!

MeAndMyRhythmbox commented 3 years ago

Thanks for the swift response. I'm building on Mac (10.14.6) so maybe there are some peculiarities in this environment... Anyway, thanks for sharing such a great tool. I've been using it almost daily ever since Gideon added the streaming option. It's also the first time I've compiled from source on Mac and I've had a great time tweaking and customising it to my needs!

DusteDdk commented 3 years ago

Thanks for getting back to me :) I'm curious if the problem may be that the audio specification requested is not met by the OS. I should need to handle this, probably by detecting if this is the case, and then do some audio resampling. However, before implementing that, I'd like to see if that's actually the problem in your case.

I've added so that if you run ./uview64 -V it should report the requested and recieved audio specification.

-V will also tell you if any packets are dropped. The audio will probably desync if packets are lost, so I'm asking you to check two things here:

Run with -V and show me the output about the audio, mine looks like:

dusted@detail:~/code/u64view$ ./u64view -V

Ultimate 64 view!
-----------------
  Try -h for options.

Verbose is on.
Opening UDP socket on port 11000 for video...
Opening UDP socket on port 11001 for audio...
Requested audio configuration:  Freq: 48000
  Format: 32784
  Channels: 2
  Samples: 192
Got audio configuration:  Freq: 48000
  Format: 32784
  Channels: 2
  Samples: 192

Running...

Also, try running it with -V for a bit, until it desyncs the audio, then check the console to see if any packets were missed, I am pretty sure that would cause a desync issue, fixing that would probably cause stuttering since I'd only detect it when getting a packet with a wrong sequence number, meaning it's already too late.

Thanks for helping me improve this ^_^

MeAndMyRhythmbox commented 3 years ago

Thanks for your continued interest... So I've done some more tests using your current code and confirmed 'requested' and 'got' audio configurations are exactly as expected :) I imagine my system is resampling the 48KHz audio from SDL to mix with system audio output at 44.1 anyway, but to be sure I have also tested with my system clock set to 48KHz and everything seems the same. I inserted my UDP code for sending start/stop audio stream commands, since the default telnet/keystroke version is still a bit iffy for me with the latest firmware, but don't expect that to affect performance.

The issue I have is still as follows:

I'm just curious what's different when manually starting the stream to bring audio back in sync vs behaviour on app launch? Oh, and yes if I launch app with -I (with streams not running) and manually start streams, audio is in sync. Auto start -U is so convenient, it's a shame it misbehaves for me.

Regarding sync drift over time: You were right, there are dropped packets but only when U64viewer is not the focussed app - i.e. switching to other apps causes dropped packets, no doubt due to viewer not having high enough priority.

Interestingly only dropped video packets are reported, never audio. Fortunately stop/start streams soon restores sync.

DusteDdk commented 3 years ago

Thanks for the detailed info! Life has a way of getting in the way of playing with fun computer things, so I reply a bit late.

I'm seeing that when doing the automatic start, the start-stream commands are sent before the listening sockets are opened and before the application is ready to receive data.. Maybe due to this, some packets are missed from one of the streams, but not from the other, that would cause them to become out of sync..

I'm guessing that if I make it start the stream inside the main loop, it will behave more correctly.. I don't have my u64 setup at the time (sadface), so I'd appreciate it if you'd try out the patch I'll put up in a moment :)

MeAndMyRhythmbox commented 3 years ago

Yes, that makes sense! Will happily test tomorrow :)

BTW Can you tell me how you generated data in 64.h for help screen graphic? I would like to update it to include extra hotkeys…

On 18 Jul 2021, at 13:52, Jimmy @.***> wrote:

 Thanks for the detailed info! Life has a way of getting in the way of playing with fun computer things, so I reply a bit late.

I'm seeing that when doing the automatic start, the start-stream commands are sent before the listening sockets are opened and before the application is ready to receive data.. Maybe due to this, some packets are missed from one of the streams, but not from the other, that would cause them to become out of sync..

I'm guessing that if I make it start the stream inside the main loop, it will behave more correctly.. I don't have my u64 setup at the time (sadface), so I'd appreciate it if you'd try out the patch I'll put up in a moment :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

DusteDdk commented 3 years ago

Great! The patch is up, looking forward to finding out if this helps, it didn't hurt anyhow :)

Yeah, the .h file is generated by gimp (actually, gimp exports a .c file, and I did a bit of copy-pasting to make it into a static constant instead of whatever they did, I am afraid I lost the original .xcf (gimp file format), so a screenshot is the easiest way to get the graphics out of the program xD

MeAndMyRhythmbox commented 3 years ago

Thanks again. Moving the start stream command did not make any immediate difference to my specific use case (-U and restarting app after losing sync) but then I tried something else -u ...

To be fair this is an edge case and I realise my biggest issue is actually dropped audio packets, which only occurs when u64view is not the focussed app (causing audio to lag and the need to send a start/stop stream.) It's only the lag caused by dropped packets that ever made me quit and restart the app anyway.

With my choice of -U ("same as -u but don't stop the streaming when u64view exits.") I notice the following behaviour:

Interestingly, I tried launching u64view with-u and found it actually behaves properly now! StopStreamOnExit seems to be key, and every quit/relaunch of u64view has audio still in sync :)

Very happy to have got to the bottom of this (minor) annoyance. Now just need to understand how to deal with dropped audio packets and why they cause lag in the first place...

DusteDdk commented 3 years ago

Hmmmmmm, I'm just guessing here, but.. maybe your OS will not let SDL_RenderPresent refresh the screen as often when the application is unfocused? u64view is single-threaded, so if any call in the main loop blocks, hmm, well, it might cause packets to be dropped :s does the audio playback seem affected when it is unfocused?

DusteDdk commented 3 years ago

I don't know if this works, but I've made it only refresh the screen if the window is not hidden.. Could you test if this solves it? Also, can you check if it will still update the screen if the window is unfocused but still visible on the screen..

MeAndMyRhythmbox commented 3 years ago

Yes, there is an audible glitch when the packets drop, after which sync is affected. I often drag the viewer onto a 2nd display while playing music on U64 and doing other things (pressing fast forward in the ultimate player is a dead giveaway when the audio is lagging) so will be interesting to see how this behaves now.

I think it’s worth bearing in mind that realtime audio is always more timing sensitive: 48,000 / 192 = 250 (roughly) audio packets every second, compared to only 50 (larger) video frames. Also, you can simply hold the last video frame until the next arrives, but once the audio buffer is drained, you’re left with silence ;)

Still a bit confused why dropped audio packets (audible glitch is heard as audio buffer drains) results in late audio (in relation to video) thereafter? My guess is that they aren’t actually dropped or lost, but just interrupted … so there is a gap in audio processing and then audio packets are put into audio buffer again (sound continues, but after a short delay.)

Will test ASAP.

MeAndMyRhythmbox commented 3 years ago

Just tried it: no change. Focus does not seem to affect screen update - still updates fine with no dropped frames even when partially obscured by another app.

Testing for window visible/hidden probably irrelevant as I never minimise the window anyway - just click on other windows and move between virtual desktop spaces (which is usually what will cause a glitch.) Think the issue for me is one of audio processing being interrupted or not at a high enough (realtime) priority in OSX. I wonder if SDL has any options to raise thread priority?

MeAndMyRhythmbox commented 3 years ago

Tried adding this just before the run loop: r = SDL_SetThreadPriority(SDL_THREAD_PRIORITY_HIGH); if(r < 0) { printf("SDL_SetThreadPriority error: %s\n", SDL_GetError()); }

Executes without error and I think it 'may' be a little more resilient, but as soon as there's a "UDP video packet missed or out of order" error, then audio is hosed.

DusteDdk commented 3 years ago

Thanks for the feedback, I guess at some point I will have to dig out my U64 (should do that anyway) and see if I can maybe reproduce this issue..

I'm guessing that if somehow the application is blocked during main loop, it may not drain the queue of audio packets, and it might become full, if it is full, I don't know the behavior, if it start discarding oldest or newest packets, but I guess anyway, that when the main loop resumes, it might be "one buffer length behind" on the audio making the audio lag behind, I will try one more patch before redecorating my office with glorious commodore computers :D I think maybe if I drain/reset the audio stuff when audio packet loss is detected..

DusteDdk commented 3 years ago

I've added the -d flag, because I've got NO idea what will happen, but here's my theory:

The main loop has been blocked, so that audio packets have been queued up in the udp receive buffer instead of immediately consumed by the application.. Now, when the application starts consuming them again, they will be consumed quickly enough, but they're written to the audio api... which will buffer them! and then the audio device will consume them at the requested rate. This will result in the audio getting behind. My solution is, after consuming an audio packet and sending it off to the soundcard, I immediately consume any remaining packets and do nothing with them.. This will probably cause a glitch in the audio on resuming, but it should stay in sync..

I'm really looking forward to hearing if this fixes it! Thanks so much for your patience in trying to hunt for this bug with me! ^^ So, please try running it with -d and see what happens ^^

MeAndMyRhythmbox commented 3 years ago

Sorry for delay, but I finally had chance to try it... am very happy to report that YOU NAILED IT! Essentially this is aggressive audio resync (and it works!) so might be nice to keep as an option and name that way.

Some more detail: I can always force the issue by repeatedly scrolling across multiple desktop 'spaces' (so may not be that common a problem for others.) With the new option enabled, audio can be made to glitch as expected (a little more harshly than previous drop outs) but sync remains solid as soon sound settles again. This is a great new feature, as I never have to deal with laggy sound again :) The odd (self inflicted) glitch yes, but no more lag.

One thought is that perhaps we could implement some sort of threshold (for the number of dropped audio packets) required to start an audio buffer purge / resync, and thus make it a little less obtrusive? Look forward to having a poke around in the code later. Also i've noticed that exiting and restarting the viewer will cause a lot of audio purging on viewer relaunch. This is because the current start/stop stream messages you added don't actually stop the audio stream. Did you see the code I added as a comment on the other open issue? It's what I've used for a long time and easy to slot in.

Thanks again for taking such an interest in this and I hope you get to fire up your Ultimate64 and play again soon.

Just need to see if I can include SDL frameworks and make my .app bundle portable enough for other Mac users to run without jumping through hoops first.

DusteDdk commented 3 years ago

excellent! I'm considering, probably making this behavior default, and able to turn it off with a switch instead. I'm also thinking about detecting how many packets are in queue and doing some simple "rescaling" to a shorter length, to let the audio "eventually catch up" over a few seconds. I'm going to open this as an improvement. I did see the mentioning of stopping the audio stream, I'm going to add that eventually, life gets in the way xD

Thanks for helping me improve this!