Raw audio stream with AudioPlayer

[zhiyan114]

Which application or package is this feature request for?

voice

Feature

I'm currently trying to setup a real-time audio system between discord and another platform. The problem is that createAudioResource introduces 3 seconds of latency. With VoiceConnection.receiver, however, I'm always able to receive the audio with subsecond latency (from the other platform).

I want to see if the implementation can be changed with how the audio is being played so that the latency can be significantly reduced.

Ideal solution or implementation

Potentially, I think the issue could be with the way createAudioResource handles the stream. Instead of relying on all the middleman classes, have the VoiceConnection.subscribe function return a raw writable stream, I can then pass through raw Opus bitstream directly.

Alternative solutions or implementations

Currently, I'm using both ffmpeg and prism-media's OggDemuxer to process the audio before being fed to createAudioResource, which doesn't solve the issue.

Other context

I'm unable to confirm if the latency is introduced by prism-media's OggDemuxer or the createAudioResource class since there's no reliable way for me to test that. I only speculated that the issue is within the class since the receiver stream is also processed by the same library's decoder and ffmpeg before being transmitted with the latency being under a second.

[alexbu92]

@zhiyan114 did you make any progress on this?

[zhiyan114]

@zhiyan114 did you make any progress on this?

Unfortunately, no. Using either createAudioResource's audio processing method or directly feeding ogg stream will both cause the 3 second latency. I'm still waiting for dev's response to see if the issue is actually coming from how createAudioResource handles the stream.

Otherwise, I'll probably be using a different lib to do my project with.

[RealAlphabet]

@zhiyan114 did you make any progress on this?

Unfortunately, no. Using either createAudioResource's audio processing method or directly feeding ogg stream will both cause the 3 second latency. I'm still waiting for dev's response to see if the issue is actually coming from how createAudioResource handles the stream.

Otherwise, I'll probably be using a different lib to do my project with.

Do you have a repro ? For me, sending an opus stream directly from a file introduces virtually no latency apart from the latency of the voice server roundtrip + the jitter buffer delay.

[zhiyan114]

@zhiyan114 did you make any progress on this?

Unfortunately, no. Using either createAudioResource's audio processing method or directly feeding ogg stream will both cause the 3 second latency. I'm still waiting for dev's response to see if the issue is actually coming from how createAudioResource handles the stream.

Otherwise, I'll probably be using a different lib to do my project with.

Do you have a repro ? For me, sending an opus stream directly from a file introduces virtually no latency apart from the latency of the voice server roundtrip + the jitter buffer delay.

I don't have it publish since I want to create a quick prototype of the setup to make sure it even works (which it did). I definitely can if you would like to take a look.

The integration is between twilio VOIP and discord bot. For some reason, the phone to Discord has 3 seconds of latency while discord to phone has sub-second latency. I'm only making that assumption that problem might be coming from AudioPlayer since the data transport all uses the same ws connection.