Closed Autrix closed 4 years ago
Hey we have this issue too. We fixed it by changing our voice region to Europe. A lot of servers our bot is in are affected as well.
I did and still same issue ( i changed every region that on the serverlist ) still nothing works for me
same problem here (eu-central)
We believe this is a Cloudflare problem
Starting to think a bunch of companies held a secret conference and all plotted to ruin the week of bot developers.
Is there something we can do to fix this atm?
Is there something we can do to fix this atm?
The only workaround seems to migrate to a server that isn't affected.
Same thing here, it's mainly cloudflare. I'm reading into discord API changes, and it seems that they made some major changes. I'll link them here below as i find the related ones.
below is the exception from my spring.log, repeated across every public discord websocket
java.lang.IllegalStateException: Failed to connect to wss://sydney49.discord.media/?v=4
at space.npstr.magma.impl.connections.hax.ClosingUndertowWebSocketClient$1.handleFailed(ClosingUndertowWebSocketClient.java:73) ~[impl-0.12.3.jar!/:na]
at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:215) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture$1.run(AbstractIoFuture.java:211) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.IoUtils$2.execute(IoUtils.java:70) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:354) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:233) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.setException(AbstractIoFuture.java:251) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.FutureResult.setException(FutureResult.java:89) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at io.undertow.websockets.client.WebSocketClient$ConnectionBuilder$2.notify(WebSocketClient.java:342) ~[undertow-core-2.0.26.Final.jar!/:2.0.26.Final]
at org.xnio.AbstractIoFuture$1.run(AbstractIoFuture.java:211) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.IoUtils$2.execute(IoUtils.java:70) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:354) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:233) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.AbstractIoFuture.setException(AbstractIoFuture.java:251) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.FutureResult.setException(FutureResult.java:89) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:391) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:372) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at io.undertow.protocols.ssl.SslConduit$SslWriteReadyHandler.writeReady(SslConduit.java:1273) ~[undertow-core-2.0.26.Final.jar!/:2.0.26.Final]
at io.undertow.protocols.ssl.SslConduit$3.run(SslConduit.java:275) ~[undertow-core-2.0.26.Final.jar!/:2.0.26.Final]
at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:582) ~[xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:466) ~[xnio-nio-3.3.8.Final.jar!/:3.3.8.Final]
Caused by: java.nio.channels.ClosedChannelException: null
at io.undertow.protocols.ssl.SslConduit.write(SslConduit.java:369) ~[undertow-core-2.0.26.Final.jar!/:2.0.26.Final]
at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
at org.xnio.http.HttpUpgrade$HttpUpgradeState$StringWriteListener.handleEvent(HttpUpgrade.java:385) ~[xnio-api-3.3.8.Final.jar!/:3.3.8.Final]
... 7 common frames omitted
It seems to be an issue with cloudflare directly, because when i do a simple curl against the v4 api base, I get a 404.
Although according to discord API docs, we should be using v3 for transmission/receive of RTP Streams, but it seems that Lavalilnk is set up for v4 entirely based on the logging i'm getting.
Here's a v4 curl from a proper linux box...
root@vcli-ytdl-test:~# curl sydney49.discord.media/?v=4/hello -v
* Expire in 0 ms for 6 (transfer 0x5623a93f2f50)
* Expire in 1 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 1 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 2 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 2 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 2 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 0 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 2 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 1 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 1 ms for 1 (transfer 0x5623a93f2f50)
* Expire in 1 ms for 1 (transfer 0x5623a93f2f50)
* Trying 162.159.130.235...
* TCP_NODELAY set
* Expire in 149998 ms for 3 (transfer 0x5623a93f2f50)
* Expire in 200 ms for 4 (transfer 0x5623a93f2f50)
* Connected to sydney49.discord.media (162.159.130.235) port 80 (#0)
> GET /?v=4/hello HTTP/1.1
> Host: sydney49.discord.media
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Thu, 17 Oct 2019 11:22:28 GMT
< Content-Length: 0
< Connection: keep-alive
< Set-Cookie: __cfduid=d53f2f49139c720dca5309785fcf37c3e1571311348; expires=Fri, 16-Oct-20 11:22:28 GMT; path=/; domain=.discord.media; HttpOnly
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 5271e5192fd546ea-EWR
<
* Connection #0 to host sydney49.discord.media left intact
and respectively a v3
root@vcli-ytdl-test:~# curl sydney49.discord.media/?v=3/hello -v
* Expire in 0 ms for 6 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 2 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 2 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 2 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 0 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 2 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 4 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Expire in 1 ms for 1 (transfer 0x564cd8d57f50)
* Trying 162.159.137.234...
* TCP_NODELAY set
* Expire in 149997 ms for 3 (transfer 0x564cd8d57f50)
* Expire in 200 ms for 4 (transfer 0x564cd8d57f50)
* Connected to sydney49.discord.media (162.159.137.234) port 80 (#0)
> GET /?v=3/hello HTTP/1.1
> Host: sydney49.discord.media
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Thu, 17 Oct 2019 11:23:41 GMT
< Content-Length: 0
< Connection: keep-alive
< Set-Cookie: __cfduid=d88b1def1545b11853fd6d9c2aec460a71571311420; expires=Fri, 16-Oct-20 11:23:40 GMT; path=/; domain=.discord.media; HttpOnly
< CF-Cache-Status: DYNAMIC
< Server: cloudflare
< CF-RAY: 5271e6dd2afce710-EWR
<
* Connection #0 to host sydney49.discord.media left intact
Judging by this, it seems their API is all kinds of screwy and nobody has noticed yet...
Now that i've slept, and have woken up and had coffee in my system, I'm going to be doing some more investigation. Here goes nothing.
Now that i've slept, and have woken up and had coffee in my system, I'm going to be doing some more investigation. Here goes nothing.
Mainly this happened after the ban wave for the bot developers who abused the API to broadcast messages from they're bots ( Sadly i was one of them who got suspended ). After that, bots all of sudden started to lose ( connection / not stable / refuse to connect to API )
i think there's a major change that happened but no one told us about it.
@AutrixSalv Cool, thanks for the heads up. I opened an issue here on Github for Discord
@wertyy102 v4 is the version that should be used, but it isn't documented.
Looks like others are having the same issue, with whatever other voice API Calls they are making. I'm going to go ahead and suggest closing this issue as a non-issue to lavalink and just hope everyone sees the fact that it's discord's side.
Latest news on the issue for anyone scrolling, and possibly incorrect implementation on lavalink's end due to breaking api changes similar to discord.py.
Our investigation of this issue continues, and we've isolated 2 specific issues.
We recently changed our voice servers to proxy TCP traffic through cloudflare to stop some variants of naive DoS attacks.
Correct client implementations should have handled this change seamlessly. We announced that we would do this change back in July in the API server: https://canary.discordapp.com/channels/81384788765712384/381871767846780928/601151474738790512 and the proper way of connecting to our voice infra has been documented for a while now: c606741
The first issue is due to an incorrect implementation of our voice API in discord.py - I had committed a fix for this in July: Rapptz/discord.py@8fdcb4d however - no version has been released since June. So, discord.py doesn't work, because it's implementation is wrong. This is probably the issue affecting small bots using discord.py. This should be resolved in the next discord.py release (1.2.4) - and upgrading the dependency to that version when released should resole the issue.
The second is some DoS protection measure triggering on larger bots, causing them their connections to fail, regardless of library. This is the issue we are currently investigating.
EDIT -- Might be fixed EDIT 2 -- Confirmed with my bot working
We think the second issue is now resolved - based on our findings, the issue was not Cloudflare's fault. Specifically, as part of this change, we started supporting and allowing clients to negotiate a TLS v1.3 connection to our voice servers. This seems to break something deep down in Lavalink, and thus causes the connection handshake to hang and timeout.
In the interim, I've gone ahead and disabled TLS 1.3 - meaning connections will fall back to the old TLS 1.2 default.
The first issue is also resolved, following a release of v1.2.4 with the change for discord.py: https://pypi.org/project/discord.py/#history
We think the second issue is now resolved - based on our findings, the issue was not Cloudflare's fault. Specifically, as part of this change, we started supporting and allowing clients to negotiate a TLS v1.3 connection to our voice servers. This seems to break something deep down in Lavalink, and thus causes the connection handshake to hang and timeout.
In the interim, I've gone ahead and disabled TLS 1.3 - meaning connections will fall back to the old TLS 1.2 default.
This was fixed a while ago. Closing.
Anyone still having this issue should revert from Java v14 to v13.
Anyone still having this issue should revert from Java v14 to v13.
I'm using v13 and still getting the problem
I am using java 14 and getting the issue. I thought it might be my dns servers but its not.
I'm using Java 14 and getting the issue too. Private bot for 2 guilds only, around 100 users...
Description
it don't allow me to play music, it shows error: Failed to connect to [server.region] ( log included down in the post )
Version info
Client used:
Output of
java -jar Lavalink.jar --version
:spring.log