Receive Audiostream - Strange behaviour

[SoerenBusse]

Hi,

I'm currently trying to develop a very simple program that records the audio of the current channel. So I wrote a basic test program:

public class Main extends ListenerAdapter {

    public static void main(String[] args) throws LoginException, InterruptedException, RateLimitedException {
        JDA jda = new JDABuilder(AccountType.BOT)
                .setToken("secretToken")
                .addEventListener(new Main())
                .buildBlocking();
    }

    @Override
    public void onGuildMessageReceived(GuildMessageReceivedEvent event) {
        System.out.println("Message Received");

        VoiceChannel userVoiceChannel = event.getMember().getVoiceState().getChannel();

        try {
            userVoiceChannel.getGuild().getAudioManager().openAudioConnection(userVoiceChannel);
        } catch (Exception e) {
            System.out.println("Error while Joining " + e.getMessage());
        }

        userVoiceChannel.getGuild().getAudioManager().setReceivingHandler(new AudioReceiveHandler() {
            @Override
            public boolean canReceiveCombined() {
                return true;
            }

            @Override
            public boolean canReceiveUser() {
                return false;
            }

            @Override
            public void handleCombinedAudio(CombinedAudio combinedAudio) {
                System.out.println(Arrays.toString(combinedAudio.getAudioData(1.0)));

                if (combinedAudio.getUsers().size() > 0) {
                    System.out.println(combinedAudio.getUsers().get(0).getName());
                }
            }

            @Override
            public void handleUserAudio(UserAudio userAudio) {

            }
        });
    }
}

What is the current behaviour: As soon as I write anything in the chat, my bot joins the same audio channel, but in the Callback the Array from getAudioData contains only zeros. Some debugging shows that this is related to this: https://github.com/DV8FromTheWorld/JDA/blob/2d19f09feda267998696768a2bbf9401e57d33b8/src/main/java/net/dv8tion/jda/core/audio/AudioConnection.java#L477

When I'm continously speaking then the console prints only the few first getAudioData arrays, but after for example 5s of speaking there're no further callbacks for that user, only the default 20ms silence with an array of zeros.

The console output is:

[19:50:09] [Info] [JDA]: Login Successful!
[19:50:09] [Info] [JDASocket]: Connected to WebSocket
[19:50:09] [Info] [JDA]: Finished Loading!
Message Received
[19:50:24] [Info] [JDAAudioManager]: Audio System successfully setup!

So am I doing something wrong? Or why won't it work as expected? And why am I only getting an array of zeros?

[MinnDevelopment]

You are checking a user that is not transmitting any audio thus you only get silence frames.

[SoerenBusse]

No that is not the issue: I've debugged more: It seems that there's an decoder issue: https://github.com/DV8FromTheWorld/JDA/blob/master/src/main/java/net/dv8tion/jda/core/audio/AudioConnection.java#L355

"decodedAudio" is null. There's audio received but it can't decode it.

[SoerenBusse]

No I know: Decoder is not null, but decodedAudio is null. Have a look on the screnshot.

[MinnDevelopment]

I misread your comment, apologies.

[MinnDevelopment]

If there is an opus decoding error I need to know what OS you are using.

[SoerenBusse]

Currently I'm using Windows 8 and Java 1.8.0_72

[MinnDevelopment]

Can you debug the specific integer that is stored in result within decodeFromOpus?

[jvtrigueros]

I'm having the same issue, I didn't use to have this issue before, if I turn on Trace level logging I see:

[13:11:26] [Trace] [JDAAudioConn]: Received audio data but Opus failed to properly decode, instead it returned an error

On incoming audio, I'd gladly provide any further information to test this out. I'm running Windows 10, but I've also run my application in an Alpine Docker container.

[queer]

Audio doesn't work in an Alpine container; I assume that it's missing some libs but I'm not sure. I ended up dealing with this by just using openjdk:8-jre instead of the -alpine variant.

[SoerenBusse]

I ran it on an Debian 8 and there it also won't work.

Before I can debug this problem, now there's another very strange issue: The method: https://github.com/DV8FromTheWorld/JDA/blob/6e23180ba9d39f26326adb0241475e6df1c8e8c3/src/main/java/net/dv8tion/jda/core/audio/AudioWebSocket.java#L420 should normally build the Inetaddress however there seems to be an illegal character in front, which result into the following:

This result into a later SocketTimeoutException.

Normally this shoudln't happen, should it.

[kantenkugel]

wow. something is really weird there Could you try updating your java installation? 72 is rather old

[jvtrigueros]

@queer even after running the application using openjdk-8:-jre which runs version 1.8.0_141 of Java I still get the Opus decoding error :(

[SoerenBusse]

@kantenkugel Even with the newest Java 8 144 there's still this encoding error. Very strange. Is it reproducable on your enviroment?

[MinnDevelopment]

Can you debug the specific integer that is stored in result within decodeFromOpus?

[jvtrigueros]

I don't want to hijack @Waishon's thread, but I'm getting -4 when audio is detected and 960 after I stop speaking:

public static final int OPUS_INVALID_PACKET = (int)-4;

[MinnDevelopment]

Ok, thank you for the information. I will investigate whether there can be something done or how this can happen.

[SoerenBusse]

@MinnDevelopment As soon as I fixed the UDP error :) Otherwise I can't debug it.

I've debugged this UDP issue with Wireshark. This doesn't seem to be an Java issue. The Discord voice server is responding with the ssrc at the beginning.

Maybe someone can have a look with Wireshark if it's the same issue in your enviroment.

Edit: This is from the official client. So it seems that it right, that Discord answers with the SSRC, but this isn't implemented in Java yet.

[SoerenBusse]

@MinnDevelopment As @jvtrigueros already said, I'm also getting error -4.

Offtopic: I've added a Pull-Request which fixes this illegal character in front of the answer UDP paket.

[MinnDevelopment]

How can it be that this is an issue for you and nobody else? On the IP discovery.

[SoerenBusse]

@MinnDevelopment Very good question :D Have you had a look in Wireshark what the answer package is? Maybe we should open a seperate issue for that. This is offtopic and not realted to this problem.

[SoerenBusse]

Another issue, I think it's related to this problem, is a Nullpointer Exception:

[23:14:24] [Fatal] [JDAAudioConn]: Encountered an exception:
[23:14:24] [Fatal] [JDAAudioConn]: java.lang.NullPointerException
    at net.dv8tion.jda.core.audio.AudioPacket.decryptAudioPacket(AudioPacket.java:178)
    at net.dv8tion.jda.core.audio.AudioConnection$2.run(AudioConnection.java:317)

This is printed randomly between the System.out of the array filled with zeros due to this above issue.

[SoerenBusse]

Just tried the same code, but with an music bot in it "FredBoat". Then I receive valid audio data, but not when a user speaks.

Edit:// Mobile also works just fine, but not from any Desktop Application

[MinnDevelopment]

I have fixed the aforementioned NullPointerException in #410 which should be merged fairly soon from now. I understand and can now properly see that decoding of audio from normal discord users seem to fail with error -4 while the audio sent from bots seems to work completely fine.

Moving this to bug label until we can figure out how to fix this issue.

[MinnDevelopment]

We have found out that discord has changed the RTP frames, adding an extension which our handling reads as payload parts causing it to fail decoding. This only happens for users that send voice data over the official client. We suspect this to be related to the recent rollout of video chat and will investigate how to handle this extension change.