Audio Latency and Autosyncing

[dtinth]

This commits in audio latency branch tries to fix Issue #8.

Latency Class

The autosyncing code in the Render class has been factored out into the Latency class.

This class can automatically adjust its latency based on the accuracy of note hits, so we can have one syncing code for both display lag and audio latency.

Changes to Autosound

In the old version, the "Autosound" option makes a keysounded music behave like a non-keysounded music.

In this branch its behavior is changed:

• Keysounded music behaves like a keysounded music. If you press the keys when there are no notes, you will hear its keysound.
• When this option is turned on, the game plays the keysound for you if you hit the note late (because of audio latency, you'll hit most notes late, and this mode corrects it by playing the sound early).
• When you miss a note, the game stops playing keysounds automatically for you until you hit another note.
• Audio latency takes place when this option is turned on.

  Audio Latency

Audio latency works by adjusting the display and judgment timing behind the audio time (which has latency).

How to use Autosync

1. Autosync the display lag first: Mute the sound, select an easy song that you know well, set the speed modifier to your comfortable speed. Check the "autosync" box next to the "display lag" box, then play that song. Try to play that song visually (because you muted the audio) and ignore the judgment the game gives. You may get a lot of GOOD or BAD judgments first but the game will adjust itself.
2. Autosync the audio latency: Turn the sound on, select an easy song that you know well, set the speed modifier so that's it's a bit lower than your comfortable speed. Check the "auto-sync" box next to the "audio latency" box, then play that song. Try to make sure you hit the keyboard the moment the sound actually goes out of your audio device into your ear, not when you see the note falls down and hit the judgment bar. That is, try to make the sound of your finger hitting the keyboard match exactly with the sound coming out of your audio device.

[keigen-shu]

Keyboard latency and audio-video sync both need different calibrations. To calibrate for keyboard latency you need to test the keyboard input relative to system time, i.e. measure the audio sampling offset between the hit keysound and the BGM with the monitor off, and then do something with that offset(skip those samples?). Audio-video sync on the other hand can be calibrated using StepMania's method.

[dtinth]

There is no calibration for keyboard latency. The input latency is already included in Display Lag and Audio Latency value. They are both done using StepMania's method.

In display lag, the player try to play without hearing the music, and in audio latency, the player try to play by concentrating on the music.

Note that there are different values configurable, one for display lag and another for audio latency. The code is now in master, which you can try running it by downloading the code and running it in NetBeans.

Now, since we cannot determine in advance when the player will press the key, when the game has audio latency correction enabled, the game will emit the keysound for the player automatically, so the player hears it when the user has to press the button (except in the case that the player presses the key before the time that the game will trigger the keysound). That means the player will notice that the sounds are more on beat and hears less gap between keysounds.

The auto playing of keysounds will be disabled when the player misses a note, and re-enabled when the player pressed a note again. This means that the keysound of the note that the player missed will still be played, but I think that it's better than having the audio latency in every note.

---

To show that the method works, suppose that:

• There is a keyboard latency of duration _k_. (duration from the moment the player presses the key to the moment that the game processes that input)
• There is a display lag of duration _d_. (duration from the moment the game updates the display to when the player actually sees it)
• There is an audio latency of duration _a_. (duration from the moment the game plays a sound to when the player actually heres it).

and also:

• _D_ is the display lag settings in the game, which when calibrated properly, will equals to k + d.
• _A_ is the audio latency settings in the game, which when calibrated properly, will equals to k + a.

Let _t_ be the time that the game expects the input from the player,

• Because of the display lag settings, the game will render the note at time t - D = t - k - d. The player will actually see it at time t - k - d + d = t - k. Assuming that the player presses the key at that exact time, the game will recognize it at time t - k + k = t, which is the correct time that the game expects.
• Because of the audio latency settings, the game will emit sounds early by A, which means that the player will hear it early by A - a = k. Assuming that the player presses the key at that exact time, the game will recognize it at time t - k + k = t, which is the correct time that the game expects.