Further DoS fixes needed

[OpenSource03]

Hello there.

The recent DoS fixes did not really do much (at least in my case). I have a sample of the java application used for sending attacks. For security reasons, I uploaded it here with a password. A developer/staff member can request me to remove the password should you wish for everyone to be able to grab this tool. Otherwise, you can contact me on Discord (OpenSource#2021) to get the file password.

Launch attack command: java -jar jarname.jar host=127.0.0.1 port=25577 proxiesFile=socks4.txt threads=10000 attackTime=320 exploit=utf16_join (it's important to keep exploit method utf16_join, all else is optional).

I hope there is something more you can do about it. This tool is able to overload any network, no matter the Bot/DDoS protection being used.

Thanks!

[electronicboy]

Being personally honest, there is only so much we can do about DDoS attacks, they take advantage of the fundamental nature of software exposed to the internet: they accept connections.

We can deal with some aspects, e.g. reducing the impact of eronious data, but, large number of connections being opened to one machine, that's out of the bounds of what waterfall can sanely defend against. Configure your firewall and look at proper load balancing to mitigate these things, such is the modern internet, this is the technique which services like CF basically take advantage of, because it's the only way to deal with a large number of connections.

[Janmm14]

@electronicboy

I analysed a pcap dump from him and based on that, bad connections are just not closed in a reasonable timeframe, like at max 0.5 seconds. They are not even closed in the full length of 2 seconds of his pcap dump. This happens because bungeecord is still waiting for the "Login Start" packet (the packet with the name field) to finish over the wire, before handling it and noticing that its too big. When this happens with thousands of connections, it gets overloaded.

The problem is the Varint21FrameDecoder.

The Varint21FrameDecoder needs to be copied for the direction Client -> Bungee to add very restrictive limits.

The copied and renamed LimitedVarint21FrameDecoder should the limit maximum packet sizes of the first 4 packets of each connection drastically (they are handshake -> login start -> encryption response -> brand plugin message or hand shake -> server list status request).

If people want to abuse the greedyness of the frame decoder further, it requires more heavy clientside computations (encryption) and a valid minecraft login to keep connections open for longer afterwards.

Further possible restrictions to apply:

• Frame decoder: Maximum fragmentation count for the first couple packets.
• Stricter timeout for the first couple packets
• Frame decoder: Maximum size for all packets sent by the client unless modding support is enabled (biggest packets sent are usually chunk packets and we know the lcient doesn't send those).

Additionally I suggest to add another ReadTimeoutHandler after the frame decoder (in PipelineUtils). This ensures that we close connections which only send us little data portions but never end the complete big packet. This likely does not prevent ddos, but it allows recovery of current ddos strategies.

[electronicboy]

There is always further stuff we can do, I cannot however commit too much due to personal health; If somebody wants to submit a PR in the meantime I can look into it, but a lot of these issues would be dealt with better in firewall vs trying to deal with app code; Stuff like this would be much easier to pull off outside of bungee due to the way the entire stack is setup

[Janmm14]

I'll put something together.

[OpenSource03]

@Janmm14 thank you very much for pointing out the importance of this issue and the ways on how to fix it 🙂

There is always further stuff we can do, I cannot however commit too much due to personal health; If somebody wants to submit a PR in the meantime I can look into it, but a lot of these issues would be dealt with better in firewall vs trying to deal with app code; Stuff like this would be much easier to pull off outside of bungee due to the way the entire stack is setup

Unfortunately, about 99.8% of service providers do not allow their clients to modify firewall rules to the point that would be needed for this type of the attack to be blocked. Furthermore, many of them would not even be willing to consider putting larger effort just for case of protecting a Minecraft Server for exploits specifically designed for Minecraft in mind. The mentioned attack will never overload the network if any service not related to Minecraft is attacked (Apache, Nginx, MariaDB) or any port without a service running on it. The attack is a little more complex so usually Linux firewall would have no chance fighting with it, especially as I had many cases where it has proven to be unstable in case of it having a larger amount of active rules. I just don't like even considering iptables or ufw for this case. I was able to work a little on it with my Serverius based provider and we are now working on blocking packets with such large values so this attack can be limited while you're working on a fix.

As an addition, I managed to do a test with the mentioned tool using a simple Google Cloud Shell. I attacked some of my servers some of which use multiple different protection methods (Serverius, Corero, Combahton, Path.net, and even CloudFlare Magic Transit). None of them was able to stop this kind of attack as packets seem to be too legit for ddos protection solutions to block them. Even one Google Cloud Shell is able to produce ~1Gbit of network traffic bypassing wast majority of protections where even the best ones struggled with >500Mbit of bypass traffic. For this I managed to upgrade my network in Serverius to 10Gbit one, however even that can not handle such attacks if attacked from multiple machines as no protection will even try to filter it.

Again, as instance, I am able to work with my provider on a fix for this issue, but what about people who are not? OVH would not even consider helping for this kind of stuff. Many others would not. Especially shared game hostings. Those would drop instantly.

It's not to mention that the network is overloaded, but a server without at least 3GB of ram and 2-4 cores (depending on the cpu type) would instantly overload. Not only does it burn network but CPU and RAM as well.

I am a Java developer, but unfortunately I think this would go a little above my knowledge so I will not be able to provide a solution, however I hope there are people who would be able to find a way of fixing the mentioned types of the attack.

Many thanks, OpenSource

[electronicboy]

with my health in particular, my interest/ability to dig deep into these issues, especially to solve stuff in a manner that doesn't involve just rewriting parts of the entire system and breaking plugins, etc. None of the things PR'd are actual fixes to the issues but workarounds to the side-effects of it, my personal interest in playing cat and mouse to resolve these issues when I have catastrophic health conditions and am not paid for this is severely limited, especially around issues which can be solved with basic firewall rules.

I understand that some are not in the position for this, but, I'm not going to offer my health up for work which I don't get a single cent for.

[Janmm14]

@Games4k Could you try attacking my fix? Download in the PR.

[OpenSource03]

@Games4k Could you try attacking my fix? Download in the PR.

Of course. I'll check it out tomorrow for you 🙂.

[OpenSource03]

Great news @Janmm14. It seems to be fully patched :)

[Janmm14]

@Pemigrade You incorrectly assumed that the problem you got is related to the problem of @Games4k. However it is not related at all. It is your AntiVPN plugin here which for some unknown reason uses the mojang api.

[kakduman]

Oh looks like I save my updates to my message in here. I talked to Games4k and we agreed that it's not the same issue that you were having. Just deleted my messages! Sorry for the confusion.

[Janmm14]

https://github.com/PaperMC/Waterfall/commit/6702e0f69b2fa32c1046d277ade2107e22ba9134

[andreasdc]

'it's the only way' 'There is always further stuff we can do' @electronicboy

[electronicboy]

Please don't bump/mention on issues for useless commentary

[andreasdc]

Please don't bump/mention on issues for useless commentary

I think it's worth mentioning that closing the idea too fast can be expensive.

[electronicboy]

I fail to understand what point you're trying to make here, you are more than welcome to make sane reasonable suggestions, playing cat and mouse with bungees dated networking stack is not a high priority especially when there are many options which can deal with specific issues much better