[Issue]: 300ms audio latency on Fire Cube 3rd gen that disappears with Google OBOE implementation

WTassoux commented 1 year ago

Describe the bug

Hello,

I have tried to run Moonlight on a Fire Cube 3rd generation but noticed that no matter the settings used, I have a latency of around 300ms between the video and the audio (the audio being later than the video, which has no visible latency).

Looking at the statistics, the average decoding time is 12-13ms in HEVC which is consistent with the fact that the video itself has no issue. The Fire Cube is connected through Ethernet so my network latency stays at 1ms (but I have tried Wifi as well). Also, the same stream works perfectly fine on my phone, implying that the issue comes from the Fire Cube.

Checking the logcat, I have found the following lines when Moonlight creates the track in AndroidAudioRenderer::setup:

createTrack_l(): mismatch between requested flags (00000004) and output flags (00000002) AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 480 -> 6152

After meddling with the AudioTrack options (attempting to change the Mode, increasing the buffer size or reducing it, etc ...), I decided to write a simple Audio Renderer using Google OBOE, as it seems the de-facto standard to play low-latency audio on Android devices.

Using this implementation, the audio latency completely disappears ! (or at least, is not noticeable anymore, I'll have to measure it properly). Which is a good news to me because it means the Fire Cube 3rd gen is capable of playing low-latency audio.

So now I am wondering whether it is possible to reproduce those results without Google OBOE, but my knowledge about Android development (or audio) is quite limited.

Is there something I missed that could allow low latency audio to play on the Fire Cube 3rd ? Should I continue with the OBOE implementation and make a pull-request ?

Thanks & regards

Steps to reproduce

Run Moonlight on a Fire Cube 3rd generation. The settings don't seem to matter, as I've tried every option possible on both Moonlight and the Fire Cube itself.

Affected games

All.

Other Moonlight clients

PC

Moonlight adjusted settings

No

Moonlight adjusted settings (please complete the following information)

N/A (doesn't matter)

Moonlight default settings

Yes

Gamepad-related connection issue

No

Gamepad-related input issue

No

Gamepad-related streaming issue

No

Android version

Android 9.x (FireOS 7.x)

Device model

Fire Cube 3rd gen

Server PC OS version

Windows 11 22H2

Server PC GeForce Experience version

Sunshine 0.20

Server PC Nvidia GPU driver version

535.98

Server PC antivirus and firewall software

Windows default AV/firewall

Screenshots

No response

Relevant log output

createTrack_l(): mismatch between requested flags (00000004) and output flags (00000002)
AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 480 -> 6152

Additional context

No response

cgutman commented 1 year ago

That is a great find! I was looking at using Oboe with the hopes that it would fix issues like this.

How did you end up integrating it with the ndk-build build system? I looked briefly and it seemed the "easy" way to get prebuilt binaries required CMake.

WTassoux commented 1 year ago

Hello,

You are right, there is a prebuilt delivred by Google when using CMake. However, my understanding is that it is difficult to mix both build systems, so I ended up writing an Android.mk for Oboe based on the content of CMakeLists.txt.

You can find the PoC of Oboe implementation here: https://github.com/moonlight-stream/moonlight-android/commit/3ee4e391cc774eea3957afd6fe2eb14032036a11

This is pretty dirty as I took a lot of shortcuts to make it work. A few notes:

Oboe 1.70 requires a minimum SDK of 21, which is higher than the current used in Moonlight. (16). I guess it should be doable to only load the library if the API of the system is greater or equal to version 21. Another solution would be to use Oboe 1.50 which is compatible with API 16 (but that would be a downgrade in term of features and performance)
I didn't implement the setup() and cleanup() functions which means for now this PoC only works for stereo, among other things.
I reused LockFreeQueue from Oboe samples to implement the communication/callback between Moonlight and Oboe but I'm not sure that's the right approach. Maybe there is a way to communicate directly between moonlight-core/moonlight-common-c instead of going through the JNI.

Regards

Dark-talon commented 1 year ago

Any progress made on this? I'd really like to use the Cube 3 as my primary moonlight device to my tv instead of having to go buy a dock for my steam deck.

WTassoux commented 1 year ago

Any progress made on this? I'd really like to use the Cube 3 as my primary moonlight device to my tv instead of having to go buy a dock for my steam deck.

I haven't touched this since I created this issue, but I could try to implement properly if this can help. Especially since Fire Cube 3 is a powerful, inexpensive device and I still don't have any explanation why it rejects the AUDIO_OUTPUT_FLAG_FAST flag.

ddovod commented 11 months ago

Hello guys. I have the same issue with error from the first message and audio latency, but the device is Xiaomi Mi Stick 4K. Btw Oboe version of the moonlight client didn't help much, the delay is reduced from 500ms to 400ms. Also Oboe client has mush less delay during the first 20-30 seconds, but later it develops to 400-450ms. Probably there're another APIs or libraries to work with low latency audio on android?

morgan35543 commented 10 months ago

I'm also experiencing this on a fire cube gen 3

moi952 commented 10 months ago

Hello, I also have the sound lag problem with Fire tv 4k Max (2023). This is a very long delay of around half a second (by ear). No bluetooth speaker.

WTassoux commented 7 months ago

Following the activity on the other issue regarding the latency on Fire Stick 3rd generation, I have updated my Fire Cube to the latest firmware see if Amazon's update improved the situation.

After making several tests, it feels that the issue almost entierely disappeared: Even using the standard audio renderer, the latency is nowhere near as bad as it used to be. Furthermore, I do not see the errors about AUDIO_OUTPUT_FLAG_FAST being denied by server in the logcat.

Doing tests with both renderers, OBOE implementation still has less latency than its native Android counterpart, but the difference isn't as drastic as before.

moonlight-stream / moonlight-android