[Critical]DoS attacks on officials servers

[Synida]

Description

So there is this on-going DoS attack on the official servers that during vs games the servers suddenly go nuts and no one can play longer under 999 ping for more than 5 minutes. So this won't be like a proper bug report like what I would like to receive, but I did find some good info on it and I think this could give a good lead on how to fix the situation.

Reproduction steps

So I created a docker instance on my VPS server to avoid it because it was said that 3rd party servers are not affected. I set the hostname to existing valve server host name; "Valve Left4Dead 2 EU West Server (srcds401-fra2.271.729)" I created a second instance with a different non valve like name as well; "Valbe Left4Dead 3 SU South Sevrer (srgds400-afro2.271.729)" The first one got attacked multiple times, the second one wasn't. The servers I set up was done in docker with the help of this repo: https://github.com/Left4DevOps/l4d2-docker

for iptables I had - I think an automatically set up config:

sudo iptables -S
[...]
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 27016 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 27016 -j ACCEPT

Which then I extended with the following:

-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m state --state NEW -m udp --dport 27016 -j LOG --log-prefix "[L4D2] "
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m state --state NEW -m tcp --dport 27016 -j LOG --log-prefix "[L4D2] "

What this does is that it logs each new connections to the iptables's log file - for me on my centos where I tested this was /var/log/messages. So for example 1 new connection looks like this in the log:

Jan 24 21:51:56 vmi327541 kernel: [L4D2] IN=eth0 OUT=docker0 MAC=00:50:56:3f:03:39:28:99:3a:4d:23:91:08:00 SRC=85.117.114.125 DST=172.17.0.2 LEN=53 TOS=0x00 PREC=0x00 TTL=114 ID=46456 PROTO=UDP SPT=29580 DPT=27016 LEN=33
Jan 24 21:51:56 vmi327541 kernel: [L4D2] IN=eth0 OUT=docker0 MAC=00:50:56:3f:03:39:28:99:3a:4d:23:91:08:00 SRC=85.117.114.125 DST=172.17.0.2 LEN=53 TOS=0x00 PREC=0x00 TTL=114 ID=46456 PROTO=UDP SPT=29580 DPT=27016 LEN=33

After that since the connection stays alive there is no new entry to this IP until the connection is closed and opened again as a new.

So then we played some and 5 mins in we got a DoS attack like one would get on the official servers. Investigating this with live recorded(via OBS) with bandwhich(https://github.com/imsnif/bandwhich) during the attack period that was recorded(3 minutes), we didn't see anything extraordinary, the player IP addresses matched with the ones found in the docker's server log file:

Investigating the same period in the iptables log file however gave a good hint on the nature of the attack. Before and after the period of the attack new connections were logged as expected like above. During the attack however every second ca. 40 new connections are registered from the same IP address.

This makes me draw the following conclusion: The server likely can't handle opening connections that well which has an effect on the performance, which doesn't really show the same effect on the server network usage and resources.

I don't know the current implementation of the connection handling, but mb it would be possible to limit the connection opening speed per IP address, and that could solve this issue.

I can get more info on this tomorrow during the attacks after I'm done with work. I can also provide the vid on the network monitoring and the iptables log if necessary.

Additional files

No response

[Synida]

I did some double checking on the recorded network and actually found the same IP sitting there as well - the one that was spamming new connections:

I also found a possible quick solution that I'm going to test today: https://superuser.com/questions/769174/limit-incoming-connections-using-iptables-per-ip

[Tsuey]

Appreciate the detail and report, and please continue if you have more to say.

Given how rampant these attacks are and the forums getting lit up with reports, there's no longer any use hiding the fact that this issue https://github.com/ValveSoftware/Source-1-Games/issues/5141 is related.

We'll push for it to be fixed, but DoS are out-of-scope for HackerOne and it'd be unprecedented and highly unlikely for Valve to accept iptables etc. as a solution, and any ETA on an engine/game code fix should not be expected.

[Tsuey]

More information from Valve:

From Valve: L4D2 Server Issues

[Synida]

I saw it in the moment it was posted, so Ig it will be resolved now. I leave it here just in case that I implemented both protection - the one I found and the one that was suggested by the reddit user in the post you linked and I didn't have any DoS on the server since.

[Tsuey]

Reported resolved or significantly mitigated as of SRCDS update:

https://steamdb.info/app/222860/patchnotes/

If https://github.com/ValveSoftware/Source-1-Games/issues/5141 persists, then it's either a different attack, or Valve's infrastructure used an iptables/firewall solution that third-party servers will also need to implement themselves, as it appears this update prioritized Valve servers.

Please submit another issue or re-open if there's further problems here.

[MrBonesYk]

Hello!

I have some information about DDoS attacks and what they are doing them with.

But it would be better to talk about it privately -> mr.bonesyk (Discord)