[3.3.5a] Authserver - long login queue

[Magic-WoW]

Rev 3f3ff8b

Hi there! This is the only place, that it can be solved and I can gain some attention.

We had a server on the rev above for 3 months. 600 online, Anti-DDoS protection on Zetservers, stable as hell, no lags at all, firewall configured - everything.

A week ago strange things happened. Players came on TeamSpeak and told us, that they can't log-in, they're stuck at 'Authenticating' after typing username/pass. Worldserver was fine, still 600 online. I was ingame too, so I did a relog and poof - authenticating. Every player which relogged confirmed, that there were the issue.

The first thought, lets restart auth/world and the whole machine. Did it, but the hacker was faster. 500 players on teamspeak saying that everything is stuck, as I can see in the console, the amount of the players is 30-40 (because they were fast and they logged in before the next attack).

Authserver doesn't even let anyone in slowly, cause it's totally stuck.

Next, a guy came and said, that he's responsible for this and happy, cause we now look like dumbs, cause we have no idea what's happening. After some conversation, we made a deal, to check if he's really responsible for the attack, we gave him what he wanted and poof, 600 players online in 4 seconds, after his 'ok, pinging turned off, thanks guys'.

The one thing he said, that in the Authserver there is no queueing for packets. He was sending oversized (compressed) packets and authserver couldn't handle them and couldn't let anyone to the world.
Also, as he said, there was a server (the biggest server in our country) which was also set on Trinity and was vulnerable, but they fixed it by queueing every incoming packet to the auth.

I really know, that this sounds like a joke, but this is the only place, where we can find the solution (or identify the bug). If I wasn't 100% sure about this, I wouldn't post here.

If someone from the devs could look on the code of an Authserver and give an opinion, it would be perfect, cause now okay, our server got crashed by one person, but tomorrow - you know, he can spread it. It's better to fix that now.

[jackpoz]

you should ask your server administrator to properly setup the firewall to allow only a few connection attempts to authserver from the same user. it's out of scope of TC project to implement a firewall. See https://github.com/TrinityCore/TrinityCore/issues/12871 and https://github.com/TrinityCore/TrinityCore/issues/12121 for reference

[Magic-WoW]

It wasn't the number of the connections... It was a malformed packet which is read wrong by the code.

He did the same thing at 4:00 AM, when only 15 players were trying to connect and noone could log-in. He's doing it from one, single connection.

[jackpoz]

how do you know it was a malformed packet ? do you have a dump of that packet ? do you have authserver debug logs saying a malformed packet was received ? did you debug authserver ?

[Magic-WoW]

The only thing I have is my word. Auth was in a debug mode, but I don't have logs. I was monitoring the network traffic during this incident at 4:00, when he was alone (he left the bot for the night I think). There was no spam of packets, just a normal behaviour;

Okay, I get it, you know what you are talking about. Can we just leave the topic opened and wait for someone's confirmation, that this is protectable by the firewall?

[Magic-WoW]

And as you said in a provided comment, what is the point (it's a serious question) of limiting connections to for example 3 on a firewall?

[jackpoz]

Authentication attempts to authserver cause a synchronous database query to be executed. A single client could keep sending authentication attempts just to trigger this and perform a Denial of Service attack. Each authentication attempt requires a new connection so by limiting the number of connections by the same client you limit the effect of their DoS attack. https://github.com/TrinityCore/TrinityCore/commit/03657525ad0c76fb1daedc68a91817df87be2676 is related to this topic too.

Unless you have informations that could help reproduce the behavior in a different way than the one I described, I'm going to assume it's just the same issue that should be mitigated by a firewall.

[Magic-WoW]

Everything is clear now, but there is one more question.

How is he doing this? I already tried monitoring and saving wow login packets via wireshark, sent them again, in loops via own java app with a tcp socket and popular apps for resending packets captured by wireshark.

Server gets these packets (I can 'cout <<' their sizes) but it doesn't recognize them as 'login' packets, there is no query etc.

Does Trinity's auth / Wow client use some kind of tokens? How is this even possible, that he is sending the same login packet 100 times per second and it gets recognized correctly as login pckt?

[Shauren]

Any remaining possible issues are now in your (and your firewall) hands.

[Magic-WoW]

I love you.

[destroller22]

hi shauren