help investigating high traffic / ddos alert

codejanovic commented 3 months ago

What happened?

This morning i got a message from my cloud provider, stating that my server seems to DDos a specific IP Address X on Port 7623/udp every 14 to 40 milliseconds for some time. The origin was the running BedrockConnect instance on Port 19132.

After investigating the traffic and the logs, i found a ridiculous amount of traffic being sent from the BedrockConnect instance, starting this morning, sending 30 to 80 mb/s out:

I also noticed the following exceptions in the log:

[nioEventLoopGroup-2-1] ERROR org.cloudburstmc.netty.channel.raknet.RakChannelPipeline - Exception thrown in RakNet pipeline
io.netty.handler.codec.DecoderException: java.lang.NullPointerException
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:98)
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at org.cloudburstmc.netty.handler.codec.raknet.server.RakServerRouteHandler.channelRead(RakServerRouteHandler.java:60)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at org.cloudburstmc.netty.handler.codec.raknet.AdvancedChannelInboundHandler.channelRead(AdvancedChannelInboundHandler.java:48)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at org.cloudburstmc.netty.handler.codec.raknet.ProxyInboundRouter.channelRead(ProxyInboundRouter.java:66)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: java.lang.NullPointerException

After restarting the service, the exceptions are gone, but the sent traffic spikes up to 40mb/s recently are still there. This happens like every 1 to 5 Minutes at the moment, but did not happen the whole last week (see the chart at the bottom).

Sadly i am not able to verify at the moment, that the Destination IP Adress X is one of my minecraft clients.

Would be great to hear your thoughts on this, as I am trying to find out if this is a service problem or if the BedrockConnect instance got "hacked" or abused in any kind.

Cheers, Aleks

Expected Behaviour?

I would expect the BedrockConnect instance to output less traffic, as my monitoring of the last 7 days looks fine:

Steps to reproduce.

No response

Screenshots/Videos

No response

Minecraft Bedrock Version

No response

Console

Nintendo Switch

Additional Context

No response

Pugmatt commented 3 months ago

Thanks for reporting this. Interestingly enough there was high traffic on the main instances as well this morning.

I believe this may have been related - https://github.com/CloudburstMC/Network/security/advisories/GHSA-6h3m-c6fv-8hvh - Which is included in the "Protocol" library, which BedrockConnect uses.

I have released a new version of BedrockConnect that contains the upgraded library with the fix: https://github.com/Pugmatt/BedrockConnect/releases/tag/1.42.1

Let me know if these issues continue to occur after upgrading.

The main BedrockConnect instance has been upgraded. Community BedrockConnect instance maintainers are highly recommended to upgrade to this new version ASAP as well, to patch the vulnerability on their instance. @AdamAtomus @kmpoppe @LazyBirb @hasankayra04 @zaphosting

codejanovic commented 3 months ago

Thanks for the update, will update tomorrow 👍 Already added an IP whitelisting as all my minecraft clients share the same IP.

Cheers

hasankayra04 commented 3 months ago

My service is setup so it updates on restart, restarted service. Now running 1.42.1.

Pugmatt / BedrockConnect