New exploit

[masakarada]

There's new exploit for bungee cord, in console there is just a spam of errors: 19:15:25 [SEVERE] [/xxx.xxx.xxx.xxx:xxxxx] <-> InitialHandler - encountered exception io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(1) exceeds writerIndex(5): UnpooledSlicedByteBuf(ridx: 5, widx: 5, cap: 5/5, unwrapped: PooledUnsafeDirectByteBuf(ridx: 6, widx: 1024, cap: 1024)) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:236) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:494) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:408) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:308) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IndexOutOfBoundsException: readerIndex(5) + length(1) exceeds writerIndex(5): UnpooledSlicedByteBuf(ridx: 5, widx: 5, cap: 5/5, unwrapped: PooledUnsafeDirectByteBuf(ridx: 6, widx: 1024, cap: 1024)) at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1405) at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:707) at net.md_5.bungee.protocol.DefinedPacket.readVarInt(DefinedPacket.java:139) at net.md_5.bungee.protocol.DefinedPacket.readVarInt(DefinedPacket.java:129) at net.md_5.bungee.protocol.packet.Handshake.read(Handshake.java:29) at net.md_5.bungee.protocol.DefinedPacket.read(DefinedPacket.java:224) at net.md_5.bungee.protocol.MinecraftDecoder.decode(MinecraftDecoder.java:33) at net.md_5.bungee.protocol.MinecraftDecoder.decode(MinecraftDecoder.java:10) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) ... 30 more

[Janmm14]

That doesn't look like it can be classified as an exploit and its also not "fixable", after such an error bungee usually forcefully closes the connection.if the ip stays the same, use iptables to block it or use iptables to add some limits like max 10 new connection in 10 seconds per ip or sth like that.

[HP888]

idk man, use brain to fix it

[v404]

hmmm it looks like an exploit that can not be fixed

[HP888]

+1

[ApocalypsjeNL]

It just looks like junk that's being sent to your BungesCord

[v404]

It looks like a handshake exploit, but an unusual one

[Janmm14]

An exploit in its core is when there is some form of bug present which allows one to circumvent security mechanisms. Being able to send bad data and making a software log that bad data was sent is no exploit. You might wanna classify it as an exploit because it causes a higher cpu usage than regular execution, but then the whole minecraft server and any other application which is logging errors would have countless "exploits".

I don't see anything else in there that could be some kind of exploit how I'd define it.

[Xayanix]

There is no exploit here. Just bad packet received.

[Duartteno]

Hi! I have the same problem. Today my server was attacked throughout the day. How can I secure the server?

There is way to block an ip address if someone sends such a packet?

[ApocalypsjeNL]

Hi! I have the same problem. Today my server was attacked throughout the day. How can I secure the server?

There is way to block an ip address if someone sends such a packet?

You have to add that IP to your IP tables then

[Duartteno]

There are a lot of different ip, they use a proxy

[Duartteno]

I used this code in MinecraftDecoder

                if ( packet instanceof Handshake )
                {
                    try
                    {
                        packet.read( in, prot.getDirection(), protocolVersion );
                    } catch ( IndexOutOfBoundsException e )
                    {
                        ctx.close();
                        System.out.println( "[" + ( (InetSocketAddress) ctx.channel().remoteAddress() ).getAddress().getHostAddress() + "] sent wrong Handshake packet. Junk??)" );
                        return;
                    }
                }

It's a bit better, but the server still lags during the attack.

[Xayanix]

Why you think it will work? We told you its not exploit, just your server can't hang high amount of connections incomming (like bot attack) and this code won't optimize it.

[Duartteno]

@Xayanix so what can I do? my server is attacked several times a day.

[Xayanix]

Making a proper firewall.

[simon44556]

I think this can happen if you open a web browser and type in serverip:port in the search bar and hold F5 it just spams the errors.

[krusic22]

Can confirm. Sending trash (http for example) to any Bungee server directly will make it unstable and refuse to take new connections after some time passes (1-2 days depending on how much trash you send).

[perseus5]

Any viable solution besides blocking the IP with IP tables? and tbh that isn't a solution in the first place, it's wayyy too easy to change your ip lol. Anyway, this exploit has been kicking players off and killing any motivation to play on my server. There has to be something I can do...

[md-5]

No one thus far has provided any evidence that there's any exploit besides simply flooding bungee with so much garbage that legit users can't join (read: DoS / DDoS).

[minecraft7net]

Hi

my impression is that someone try to explot BungeeCord - i didn't report this directly to BungeeCord becouse i was not sure and i have start with Travertine what i use right now for one of my network that need 1.7 support.

but my impression is that someone try to exploit or search something.

its 10-20 connections from differen IPs.

i found this becouse last days i have try to solve some discionnection problems and day by day performance tunning i found some errors.

this is my search criteria on bungeecords and similiar on spigot/paper servers

cat /home/bc1/craftbukkit/logs/latest.log | grep -i "too many packets" cat /home/bc1/craftbukkit/logs/latest.log | grep -i "encountered"

after debug:

cat finderrors.sh cat /home/bc1/craftbukkit/logs/latest.log | grep -i "too many packets" cat /home/bc1/craftbukkit/logs/latest.log | grep -i "encountered"

cat find.resolve.sh

!/bin/bash

SEARCH=./finderrors.sh | awk -F "/" {'print $3'} | awk -F ":" {'print $1'} | sort

for a in $SEARCH do host $a

./finderrors.sh | awk -F "/" {'print $3'} | awk -F ":" {'print $1'} | sort

shows IPs

and filder with ".pl" becouse i have players mostly from Poland (regular users) and criteria to find exceptions at 7AM.

./find.resolve.sh | grep -v ".pl"

100.91.43.100.in-addr.arpa domain name pointer 100-43-91-100.spider.yandex.com. 143.143.8.141.in-addr.arpa domain name pointer 141-8-143-143.spider.yandex.com. 139.244.154.178.in-addr.arpa is an alias for 139.128/25.244.154.178.in-addr.arpa. 139.128/25.244.154.178.in-addr.arpa domain name pointer 178-154-244-139.spider.yandex.com. 145.87.9.37.in-addr.arpa is an alias for 145.128/25.87.9.37.in-addr.arpa. 145.128/25.87.9.37.in-addr.arpa domain name pointer 37-9-87-145.spider.yandex.com. 196.87.9.37.in-addr.arpa is an alias for 196.128/25.87.9.37.in-addr.arpa. 196.128/25.87.9.37.in-addr.arpa domain name pointer 37-9-87-196.spider.yandex.com. 223.87.9.37.in-addr.arpa is an alias for 223.128/25.87.9.37.in-addr.arpa. 223.128/25.87.9.37.in-addr.arpa domain name pointer 37-9-87-223.spider.yandex.com. 224.87.9.37.in-addr.arpa is an alias for 224.128/25.87.9.37.in-addr.arpa. 224.128/25.87.9.37.in-addr.arpa domain name pointer 37-9-87-224.spider.yandex.com. 11.250.255.5.in-addr.arpa domain name pointer 5-255-250-11.spider.yandex.com. 121.250.255.5.in-addr.arpa domain name pointer 5-255-250-121.spider.yandex.com. 139.250.255.5.in-addr.arpa domain name pointer 5-255-250-139.spider.yandex.com. 83.250.255.5.in-addr.arpa domain name pointer 5-255-250-83.spider.yandex.com. 31.47.88.77.in-addr.arpa domain name pointer 77-88-47-31.spider.yandex.com. 75.47.88.77.in-addr.arpa domain name pointer 77-88-47-75.spider.yandex.com. 17.5.88.77.in-addr.arpa domain name pointer 77-88-5-17.spider.yandex.com. 126.161.158.93.in-addr.arpa domain name pointer 93-158-161-126.spider.yandex.com. 150.161.158.93.in-addr.arpa domain name pointer 93-158-161-150.spider.yandex.com. 80.161.158.93.in-addr.arpa domain name pointer 93-158-161-80.spider.yandex.com.

whois yandex.com | grep -i .ru Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited Name Server: NS1.YANDEX.RU Name Server: NS2.YANDEX.RU Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited Name Server: NS1.YANDEX.RU Name Server: NS2.YANDEX.RU

i think no more comments are nedded

[minecraft7net]

and with timeframe:

https://pastebin.com/z3mXfcvK

is it possible to have modsecurity or something similar when i have HAProxy before BungeeCord ? probably the option is to have BroIDS or Suricatta-IDS inline but its resource invensive operation.

[minecraft7net]

and another the same day.

https://pastebin.com/iwvUaLfR

[minecraft7net]

my solution for this is ebtables/iptables block.

!/bin/bash

./find.resolve.sh | grep -v ".pl" | awk -F " " {'print $5'} | grep -i .com | sed 's/.com./.com/g' | sort | uniq

BLOCK=./find.resolve.sh | grep -v ".pl" | awk -F " " {'print $5'} | grep -i .com | sed 's/.com./.com/g' | sort | uniq

BLOCKIP=for blockip in $BLOCK; do host $blockip ; done | awk -F " " {'print $4'}

for block in $BLOCKIP; do echo /sbin/iptables -A INPUT -s $block -d 0/0 -j DROP; done

./generate.block.sh ... 93-158-161-80.spider.yandex.com /sbin/iptables -A INPUT -s 100.43.91.100 -d 0/0 -j DROP /sbin/iptables -A INPUT -s 141.8.143.143 -d 0/0 -j DROP

and block on bungeecord servers

[minecraft7net]

OK after additional - "test".

my case is just a webbrowser connection to bungee and it can be reproduce by connection from firefox to IP:25565

its my connection from firefox. "..." java.lang.IllegalArgumentException: Unexpected packet received during login process! 4554202f20485454502f312e310d0a486f73743a20706c616e6574637261667467616d65732e6e65743a32353536350d0a557365722d4167656e743a204d6f7a at net.md_5.bungee.connection.InitialHandler.handle(InitialHandler.java:146) ~[Travertine-b69.jar:git:Travertine-Bootstrap:1.13-SNAPSHOT:43bf43a:69] "..."

so probably solution is to create additional "something" on haproxy to redirect HTTP/HTTPS connections.

[Leymooo]

I think that will be better just close a connection without throwing an exception. (https://github.com/SpigotMC/BungeeCord/blob/master/proxy/src/main/java/net/md_5/bungee/connection/InitialHandler.java#L137)

[minecraft7net]

I think that will be better just close a connection without throwing an exception. (https://github.com/SpigotMC/BungeeCord/blob/master/proxy/src/main/java/net/md_5/bungee/connection/InitialHandler.java#L137)

Thanks ! :-)

i have think about something on iptables level but it will consume a lot of CPU

something similar to this

iptables -I OUTPUT -p tcp --dport 443 -m string --string 'GET / HTTP/1.1' --algo bm -j DROP

with combination of --cpu

more examples here.

https://blog.nintechnet.com/how-to-block-w00tw00t-at-isc-sans-dfind-and-other-web-vulnerability-scanners/ https://manpages.debian.org/unstable/iptables/iptables-extensions.8.en.html

[softwareboss]

@md-5 , We have a server with +2000 online with custom build bungeecord and latest version. We was using old version (build 1267) and we did not have a problem. Now, we are using latest version and now cause this bug.

If, we sending a lot of packets to BungeeCord packet, bungeecord giving errors like this:

Yes, we known this, its a ddos attack. But its very weak attack. Because we're attacking to server.

And, after players can't join to server:

(Sorry for weak english)

[Xayanix]

You have to make firewall and limit packets per second.

[softwareboss]

:) We are using firewall.

[softwareboss]

Problem solved with this:

[md-5]

Right, so these kind of issues are incredibly difficult for me to respond to for a number of reasons, including:

• They are often slightly different in their own manner
• People don't or can't provide much info about them "new exploit, bungee crashes"
• People are running in wildly unsupported configurations (60 Spigot servers + a Bungee on a 4 core CPU, or just flat 1 core CPUs)
• They ultimately do boil down to volumetric DDoS style attacks
• People propose fixes that work for them against one specific type of attack but (assuming a systemic Bungee weakness) can easily be negated. An example of this is the "fix" of dropping HTTP connections which can obviously be bypassed by not sending HTTP as the "invalid data". The "fix" of this.ch.close() above is also subject to similar bypasses, but I won't go into that here as it is less obvious.

Onto the issue itself:

In 1.13 BungeeCord did what it should've been doing from day 1, and started dropping connections that sent invalid data during the login process. Previously it just ignored this data which is bad for a couple reasons: 1) It is not robust, and 2) It leaves open the possibility of other DoS type attacks.

This has triggered an uptick in bug reports of "new Bungee exploit". I am skeptical of this claim for a couple more reasons: 1) BungeeCord is now closing connections where previously it wasn't (yet some of the proposed fixes are just adding even more close calls) 2) This fix causes an increase in console logging which scares users into thinking something is wrong and opening bug reports. 3) Console logging is done pretty much entirely asynchronously on 1 single separate thread, so an increase in it by itself should not affect connection or Bungee reliability. The exception to this would be a) single core servers, and b) wildly overutilised servers as mentioned above. 4) I have run several "crash" scripts provided by various users and none have had a measurable impact on my servers.

BungeeCord generally has extremely verbose logging because the information provided in exceptions / stack traces is often the only information we can get out of users (see: these tickets). What much of the "fixes" have in common is that by closing the connection early, they eliminate this logging and thus prevent a CPU increase caused by the (single, isolated) logging thread. I suggest that the fact that the connection is closed microseconds earlier has nothing to do with "fixing" the exploit.

Since in this case and a couple of other cases the source of the exception is completely obvious, we don't need to provide a large stack trace. So starting from the commit mentioned below it will be suppressed from these error messages (and the packet dump size reduced slightly). I think that this will "revert" the situation to what it was before for those running single core and extremely overloaded BungeeCord servers.

Ultimately the types of attacks reported in these tickets are DDoS attacks and there are limited things that Bungee can do to "prevent" them. What I am confident of however is that this "fix" is more reliable than other proposals mentioned thus far, and for users in heavily loaded configurations brings Bungee back to what it was pre 1.13 under these circumstances.