UDP as a WebSocket extension

[Matheus28]

Here's my proposal for a WebSocket extension that would expose UDP securely to web pages:

During the initial WebSocket handshaking, the client signals support with the header Sec-WebSocket-Extensions: udp. The server replies back with the port that the client can use for UDP: either Sec-WebSocket-Extensions: udp (port 80 or 443, depending on whether it's ws or wss) or Sec-WebSocket-Extensions: udp=1234. Note that the host for the connection would be the same as the http request host.

For web developers, the WebSocket object would have a sendUnreliable method to send a datagram to that UDP port, and once the first datagram is sent, the server then knows where to send UDP packets to (and that the client supports UDP; initial packet has to come from client because of NAT). The WebSocket object would also have "udp" added to its extensions property, so the client knows that the server supports UDP. Sessions would be handled entirely by the developer and don't really need to be dealt with in the protocol.

Now, some issues:

1. SSL: It makes sense to not wrap datagrams using DTLS, since the entire purpose of using UDP is to minimize CPU and latency costs. While a possible solution is using DTLS for secure websockets (and not using it for insecure websockets), this causes problems with the fact that https origins can only use secure websockets, and game developers would force their users to only use http (basically all browser MMOs out there already redirect https to http because of this). My proposal is always using insecure datagrams, since anything supposed to be secure can just go over TCP.

2. Raw datagrams: Raw datagrams have a few issues, possibly the biggest ones are IP fragmentation and flow control. Should raw datagrams be exposed, those would be pushed onto developers to deal with, I don't really see the problem with that, as I don't think this should be solved at the protocol layer. My proposal is to just send raw datagrams after the initial handshake is made.

---

If there aren't major objections to this approach, it is possible to write a browser extension that exposes this interface, and this would get the ball moving faster for browser developers to take a look at the proposal.

[Maksims]

After WebSockets HTTP 101 handshake, the underlying implementation of extension can require UDP (or alternative) transport to perform second handshake. If underlying protocol is not connection based then some extra header information has to be implemented by an extension to prevent messages to be sent/received from other sessions or even different endpoints.

This will also release a need from application to make initial message to deal with NAT as it will be dealt by second handshake.

Regarding congestion control, this might be too much of a security concern for a web community and will unlikely allow such API to be implemented.

Regarding SSL, if developers already use non-SSL for avoiding CPU cost associated with encryption, then DTLS is not a problem as it will leave a choice to developer while preserving security requirements for the web. Additionally HTTP + HTTPS approach can be used, where main page served over HTTP and some other traffic served over SSL. It won't work otherwise due to security practices around the web.

[Matheus28]

Regarding congestion control, this might be too much of a security concern for a web community and will unlikely allow such API to be implemented.

Could you elaborate this point? You mean raw UDP is too much of a security concern because people could DDoS using it? My proposal wouldn't allow sending UDP packets to a server until it signals it wants to receive them.

Regarding SSL, if developers already use non-SSL for avoiding CPU cost associated with encryption, then DTLS is not a problem as it will leave a choice to developer while preserving security requirements for the web.

Avoiding HTTPS because of CPU costs isn't really that bad. I think it's a separate issue from avoiding DTLS for those UDP packets.

Additionally HTTP + HTTPS approach can be used, where main page served over HTTP and some other traffic served over SSL. It won't work otherwise due to security practices around the web.

You can't create insecure websockets in a secure context (https). Since we want the web to advance towards HTTPS everywhere this would mean forcing DTLS (which I don't think that is desirable). Always using insecure datagrams and letting developers use the normal websocket connection for secure communications would be a good compromise IMO.

[Maksims]

Could you elaborate this point? You mean raw UDP is too much of a security concern because people could DDoS using it? My proposal wouldn't allow sending UDP packets to a server until it signals it wants to receive them.

Congestion control is required by underlying transport protocol to avoid one side to overwhelm with traffic another side as it can be unable to process messages fast enough. This leads to network congestion issues which lead to more packet losses and increased latency. Most popular case is when someone is playing a game and neighbours on same network start streaming some videos and downloading stuff. Quality of a players connection will start to suffer due to local network congestion issues.

If a protocol does not implement congestion algorithms then it is easy to halt the network on server or client by spamming it non-stop with either large messages or many small ones. Other connected clients will suffer from that too.

You can't create insecure websockets in a secure context (https). Since we want the web to advance towards HTTPS everywhere this would mean forcing DTLS (which I don't think that is desirable). Always using insecure datagrams and letting developers use the normal websocket connection for secure communications would be a good compromise IMO.

But you can create connections other way, from non-SSL to SSL. Means you can have some of application logic to be performed on SSL connections and some on non-SSL. Main page have to be served on non SSL connection though. And cross-domain issues has to be handled too, as https and http on same hostname are considered as cross-domain.

[Matheus28]

IMO congestion control should be in the application layer in this case, since that's how UDP already works. The worst this proposal exposes is equivalent to what you can already do (with xhr, img tags, etc.): exhaust all the user's bandwidth.

But you can create connections other way, from non-SSL to SSL.

But we're moving towards HTTPS everywhere. You won't be able to have a non-HTTPS page eventually, and proposals made today need to have that in mind.

[gafferongames]

@Matheus28 Agreed on the congestion control at the application layer. Also, packet fragmentation and reassembly at the application layer + reliability done there too. Just the minimal required at the low-level that gets the connection-based and encrypted UDP without DTLS/SSL overhead is all we need.

ps. there is now a browser implementation for netcode.io that should provide what you need: https://github.com/RedpointGames/netcode.io-browser

[Matheus28]

@gafferongames

I've seen netcode.io in the past, but with all due respect, it's a just not a good solution (in my opinion). Those "connection tokens" have no place in protocol level.

[gafferongames]

Those "connection tokens" have no place in protocol level.

I'm going to have to disagree with you, because this is exactly how it's done in the AAA space.

[gafferongames]

SSL: It makes sense to not wrap datagrams using DTLS, since the entire purpose of using UDP is to minimize CPU and latency costs. While a possible solution is using DTLS for secure websockets (and not using it for insecure websockets), this causes problems with the fact that https origins can only use secure websockets, and game developers would force their users to only use http (basically all browser MMOs out there already redirect https to http because of this). My proposal is always using insecure datagrams, since anything supposed to be secure can just go over TCP.

You are proposing sending game protocol over unencrypted UDP, because anything secure can just go over TCP? This is madness...

[vvanders]

@Matheus28 You really do need the token + basic encryption. Otherwise spoof + replay attacks are pretty trivial.

[gafferongames]

And DDoS amplification attacks...

[Matheus28]

I have to disagree, because this is exactly how we solve this problem in the AAA space.

What are you referring to? Connection tokens? That should be application level.

You are proposing sending game protocol over unencrypted UDP, because anything secure can just go over TCP? This is madness.

So you use encrypted UDP for real time stuff on AAA games? I've worked with a few third-party engines and none of them did that. Not to mention my own networking engines. There's no reason to. The only stuff that go over UDP is the data to replicate the entity system. If you want secure UDP, that can go on user code, not part of the protocol.

@Matheus28 You really do need the token + basic encryption. Otherwise spoof + replay attacks are pretty trivial.

That can go on the application layer. Server gives client a token to sign his requests and identify them over TCP, and client uses that. That's not supposed to be on the protocol layer.

And DDoS amplification attacks...

Now you gotta be kidding me. I spent the last few minutes writing these replies and you come up with this? You're better than this. There's no UDP communication before a TCP handshake. You (and @vvanders) took my opinion on your netcode.io solution as a personal attack, and you're trying to pull any reason to criticize this proposal. I don't think you even read it thoroughly at this point.

[vvanders]

@Matheus28 No need to get personal here.

Yes, all of that could go into the application layer. However one of the core principals of the web is security and I think it makes sense to have that bundled in and painless to use.

Replay attacks/mitm are very much a thing. I've also never seen network protocol overhead even show up on a profile. If you're sending that much data you've toasted a user's network connection and on the server back-end you always have beefy servers or proper load balancing.

[gafferongames]

So you use encrypted UDP for real time stuff on AAA games? I've worked with a few third-party engines and none of them did that. Not to mention my own networking engines. There's no reason to. The only stuff that go over UDP is the data to replicate the entity system. If you want secure UDP, that can go on user code, not part of the protocol.

Yes. AAA multiplayer games always do this. You'd be crazy not to. Think of the MITM attacks...

[Matheus28]

Replay attacks/mitm are very much a thing.

True, but for most use cases that want UDP, they don't matter. If a malicious router is making the player walk into a wall instead of shooting at people in a game, how much damage is that really causing...? Most games don't transmit secure information over UDP.

I've also never seen network protocol overhead even show up on a profile. If you're sending that much data you've toasted a user's network connection and on the server back-end you always have beefy servers or proper load balancing.

Most of my games handle 100~1000 players in a single CPU core, in fact, most of the time is actually spent encoding and sending those messages (over TCP, of course).

[gafferongames]

Now you gotta be kidding me. I spent the last few minutes writing these replies and you come up with this? You're better than this. There's no UDP communication before a TCP handshake. You (and @vvanders) took my opinion on your netcode.io solution as a personal attack, and you're trying to pull any reason to criticize this proposal. I don't think you even read it thoroughly at this point.

I don't feel attacked. I just think you're wrong.

[Matheus28]

Yes. AAA multiplayer games always do this. You'd be crazy not to. Think of the MITM attacks...

Like preventing the player from doing an action or sending them erroneous entity states? I'll take the risk of that over imposing that overhead over all udp packets coming/going to the browser. Anything secure can go over TCP. And if you need secure real time data, you can include DTLS in the application code.

Also, as a personal example, last time I worked with Source Engine, there was no encryption. That's one I remember for sure.

[gafferongames]

Depends entirely on the game and how complex your protocol is. If it's dead simple, sure you could make an argument that encryption is unneccessary overhead. Fine. I disagree, but fine.

But you haven't addressed spoofed packet headers and DDoS amplification attacks. This is unfortunately real and quite serious. If a UDP packet from server to source packet IP address is larger than typical request or input packet, you expose your protocol to be used as part of a DDoS amplification attack.

You also haven't considered that dedicated servers for many games are quite expensive. You want to stop people from connecting to a server without having authentication to do so. This, along with the challenge/response to stop spoofed packet headers and DDoS amplification, is why AAA games do it the connect token way.

cheers

[Matheus28]

But you haven't addressed spoofed packet headers and DDoS amplification attacks. This is unfortunately real and quite serious. If a UDP packet from server to source packet IP address is larger than typical request or input packet, you expose your protocol to be used as part of a DDoS amplification attack.

You haven't read the proposal. Before any UDP packets are able to be sent, a handshake happens over TCP. You can only send UDP packets to the same host that you shake hands with. The browser also doesn't allow you to spoof the source IP.

You also haven't considered that dedicated servers for many games are quite expensive. You want to stop people from connecting to a server without having authentication to do so. This, along with the challenge/response to stop spoofed packet headers and DDoS amplification, is why AAA games do it this way.

That's application layer, not protocol. You're just running in circles here.

[gafferongames]

You haven't read the proposal. Before any UDP packets are able to be sent, a handshake happens over TCP. You can only send UDP packets to the same host that you shake hands with. The browser also doesn't allow you to spoof the source IP.

So I open up wireshark and find the IP address that UDP packets are being sent to. I note this, and later on I start sending UDP packets to that IP address and port to attack your protocol. You need to think this through Matheus...

[gafferongames]

You haven't read the proposal. Before any UDP packets are able to be sent, a handshake happens over TCP. You can only send UDP packets to the same host that you shake hands with. The browser also doesn't allow you to spoof the source IP.

I read the proposal and understand it fully. It's incredibly naive.

[Matheus28]

So I open up wireshark and find the IP address that UDP packets are being sent to. I note this, and later on I start sending UDP packets to that IP address and port to attack your protocol. You need to think this through Matheus...

Just like you can DDoS any other server? What is the issue here?

I read the proposal and understand it fully. It doesn't work.

You might have read it, but you clearly didn't understand it based the points you raised.

[gafferongames]

Just like you can DDoS any other server? What is the issue here?

You don't seem to understand how DDoS amplification attacks work.

[gafferongames]

That's application layer, not protocol. You're just running in circles here.

This is your opinion, and you've firmly stated it several times without any justification. Fine.

I still think you're wrong. It's not application layer. It's an essential security step to make it possible for UDP-like functionality to be secure enough to get included in a browser.

[vvanders]

@Matheus28 Just because something is out there and widely used doesn't mean it's not vulnerable. Here's just one of many sources you can find about how the Source engine isn't the most secure(it's has it's upsides but I wouldn't list networking as one of them, yes I've shipped titles with it).

https://www.slideshare.net/z0mbiehunt3r/ddos-amplification-attacks-with-game-servers

[Matheus28]

You don't seem to understand how DDoS amplification attacks work.

You don't seem to understand how they work. Before we waste any more time: Are you saying this proposal enables web browsers to unwillingly participate in DDoS attacks, or are you saying that the proposal as it is allows the server to get DDoS'd more easily than any other server serving traffic over UDP?

[gafferongames]

You don't seem to understand how they work.

I'm sorry, are you five years old?

[gafferongames]

@Matheus28 Google "DDoS UDP packet amplification". I'll wait...

[Matheus28]

Where does it say in my proposal how the server replies to UDP requests? Tell me how the proposal facilitates those attacks in any way. No, really, give me an example.

[gafferongames]

Example: Somebody creates a game using your WebUDP proposal. Instead of inputs to server, and server sending snapshots of world down to a client at steady state (as you almost certainly do for agar.io), they build their protocol around request/response (this is pretty common pattern for MMOs and mobile games...). Bad guy connects to their server websockets-UDP whatever and find its IP address and port via wireshark, then takes advantage of a request packet that is smaller than the response from the server to launch a DDoS packet reflection attack, eg. spins up a bunch of processes in linux with raw sockets and spoofs UDP packet IP source header to the IP address to be attacked, and the WebUDP server responds with packets larger than request packets, eg. amplification.

It's that simple. Yes, this actually happens. By saying that this is application level, you're making it possible/likely that somebody fucks this up and this situation happens. Therefore, It should not be at application level. It should be taken care of by the protocol-level so this cannot happen.

[benaadams]

Where does it say in my proposal how the server replies to UDP requests? Tell me how the proposal facilitates those attacks in any way.

That's the problem. It needs to be baked into the protocol that you only respond to the correct IP addresses. Otherwise its enabling an new generation of services which may be vulnerable as DDoS amplifiers to be deployed on the internet and the browser vendors won't support it.

Its the protocol that is the vulnerable thing; example common UDP protocols that are used as DDoS amplifiers: DNS, NTP, SNMPv2, NetBIOS, SSDP, CharGEN, QOTD, BitTorrent, Kad, Quake Network Protocol, Steam Protocol, RIPv1, Multicast DNS (mDNS), Portmap/RPC, LDAP, Network Time Protocol (NTP)

[vvanders]

Any application that uses this and sends a reply to a udp message would be susceptible. That's why you need encryption(and to send the token over a secure channel which makes it part of the protocol).

Also you missed on the reasons why UDP is used, it's not for low overhead, it's because of heading off line blocking. If you've got perf issues with just i/o at 1k clients then you need to start putting load balancing in front on your hosts. Game design plays a role in this as well(otherwise you'll run into common scaling issues that Eve and other MMOs have hit).

[gafferongames]

I mean even if you don't want packet encryption, you at least want a challenge/response... but then if you have people who run dedicated servers that are CPU intensive (lets say, 16 players per-core vs. 1000...), you start having the problem that you don't want to just let anybody with an IP address connect to your server, but only somebody who has authenticated with the web backend.

Once you go down this path, you pretty quickly converge on connect tokens...

[Matheus28]

@gafferongames

In your example, it's not a problem with the proposal, but with a badly implemented server. While making the protocol fool-proof is an argument, I disagree that the solution for this should be on the protocol level.

@benaadams

Please see reply above

@vvanders

Any application that uses this and sends a reply to a udp message would be susceptible. That's why you need encryption(and to send the token over a secure channel which makes it part of the protocol).

TCP is the secure channel here, what I'm advocating for is that being part of application code, not protocol.

Also you missed on the reasons why UDP is used, it's not for low overhead, it's because of heading off line blocking. If you've got perf issues with just i/o at 1k clients then you need to start putting load balancing in front on your hosts.

I never said I'm going for UDP for performance issues. I have over 300 servers for all my games with load balancing in front of them. I said I don't want more CPU overhead.

@gafferongames

I mean even if you don't want packet encryption, you at least want a challenge/response... but then if you have people who run dedicated servers that are CPU intensive (lets say, 16 players per-core vs. 1000...), you start having the problem that you don't want to just let anybody with an IP address connect to your server, but only somebody who has authenticated with the web backend.

That's application layer still. If you can't say "100% of the servers need to do it this way", it should really be application layer.

[gafferongames]

I think we're just going to have to agree to disagree on this one. Good luck with your proposal!

[Matheus28]

Yep. Feel free to file a different proposal with your fancy connection tokens and DTLS.

[gafferongames]

Thanks! I look forward to hacking your insecure servers in the future ❤️

[Matheus28]

Did you just edit half your comments to remove a bunch of erroneous points you made? Hours after the discussion was over? Damn you are immature. I've blocked you now. Please refrain from brigading my proposals with your friends again. Thank you.

Also, my games all support UDP, which is used for the mobile version, have fun with that.

[hach-que]

Looking at this proposal from my perspective, there's a few issues I'd like to note:

Same IP Address

It requires both the game server and the HTTPS WebSocket to be served on the same IP address. There are a few solutions to this:

• If I want my game server to sit at the network edge, I need to embed a HTTPS WebSocket server in my game server. This requires pulling in a whole stack of dependencies in most cases (including TLS encryption). This is the only mechanism which enables the game server to see the original source IP and port of UDP packets, without the use of specialized load balancers.
• Terminate the IP address with a load balancer, and use that to route HTTPS traffic to a conventional web server and UDP traffic to the game server. This requires the use of a load balancer that can do both TCP and UDP load balancing simultaneously with the same frontend IP address. At least on Google Cloud, this is not possible - TCP and UDP load balancers are separate and have separate IP addresses. Even if it were possible, this would a load balancer for each game server, which is roughly equivalent of $20/month per LB.
• This means that the only way to do load balancer termination is to run something like Nginx, which now needs to sit at the network edge and route HTTP traffic to your web server and UDP traffic to your game server (again of which there can be only one - if you want multiple game servers you need to run multiple Nginx instances).

Having to run a separate HTTPS server with LB proxying (and losing UDP source IP and port) is a lot of work as a developer. I definitely don't want to embed a HTTPS WebSocket compatible server inside my game server itself.

Multiple DNS Records

Looking at the proposal it limits the UDP packets to be sent to the same host as the WebSocket connection. The issue is that almost all of the time you don't connect to a WebSocket by IP address, you connect to it via hostname, which is a concept that exists entirely at a HTTP protocol level.

If I connect to wss://myhost.com/, I establish a HTTPS connection based on the DNS records provided for myhost.com, and then pass myhost.com as the Host header to the connection. Browsers will iterate through multiple A records based on which is connectable.

As far as I can tell, if one can set up a DNS record with multiple A records, all with very low TTL (a few seconds), then it's possible to have the browser establish a connection to wss://myhost.com, get authorized to send UDP packets to that host, and then update the DNS so that myhost.com points to an IP address I want to blast raw UDP packets to from the user's browser.

I doubt this attack would be reliable due to DNS caching, but if you could find a setup that even works 1% of the time, you can imagine the damage this could do if you could get an advertisement to run on an ad network that does this.

This isn't an issue for TCP WebSockets because the same connection is reused for sending packets as is the one that establishes it. This isn't the case for the UDP WebSockets proposal.

[Maksims]

Whoaaa guys. I know web is like anarchy, but take it easy please, lets be more constructive and less rude to each other.

Both of you guys expressed valid considerations:

1. We do need security measures to avoid easy DDoS amplification attacks.
2. Mitigate impact of MITM accessing potentially sensitive information.
3. Easy packet forgery.
4. Simple API design for back-end and front-end.

We need (constructively) come up with decent solutions on each point. In fact, a lot of motivation behind SCTP for WebRTC DataChannel, I assume was actually to avoid many of those problems. Packet forgery for SCTP is harder comparing to "naked" UDP.

Although other protocols addressed those concerns too: DTLS is designed to prevent packet forgery and MITM eavesdropping. Additionally DTLS addresses few DDoS attacks and makes them much harder. WebRTC uses DTLS between raw UDP and SCTP for many of those reasons.

We do know that DTLS and SCTP complicates DataChannel layer of WebRTC so it makes it harder to implement for server-side. SCTP implements many components which are arguably not a requirement for our efforts. Although it has optional reliability and ordered delivery, which developers might have to re-implement on top of UDP (or alternative) if the need arises. At the same time: DCCP is much simpler than SCTP, it addresses congestion control (which has to be addressed) and is connection based protocol, which maps to API design of WebSockets well.

Worth mentioning, all those security considerations have to be addressed, and they have been by many different implementations.

I do not believe that browser vendors will even consider any spec or work to be done in that direction if it is not based on IETF time proven and well designed implementations. Nor it will accept raw UDP due to many security holes.

The problem we having, is disagreeing on what has to be addressed, and what not. So taking a step back, lets first constructively list existing security risks with existing solutions as well as the ones needs to be addressed by future solution. Then we can talk about specific implementations to address them.

[Matheus28]

@hach-que

I agree that it is a bit involved to connect to a specific server using TLS, but it's not really different from what you need to already do today. All major implementations of WebSockets already implement TLS, and it's not hard to wrap an existing one to support it (since TLS works pretty much by just being put in front of the protocol to encrypt/decrypt).

For reference, all Agar.io and Diep.io servers have a certificate in all of them (for diep.io it's *.s.diep.io) and they use that to decrypt TLS traffic in the same machine that the game server runs. I avoid putting a load balancer in front of them and just give the client a hostname that connects to the server I want them to.

Regarding the DNS records thing, that's a DNS rebinding attack. It's already solved by browsers (for example, by always using the same IP address for a host for the duration that the web page is open).

I'm on my phone at the moment, sorry for not writing a very detailed reply.

[Matheus28]

Here's some ways I can think of to expose UDP to the web as part of this proposal. Remember that you can always wrap some on top of the others. You can have encryption on raw datagrams, the burden just gets shifted to application code.

1. Raw datagram

By raw access I mean raw access to the payload of the datagram, not the UDP headers of course. This would provide maximum flexibility. One could even have a separate WebSockets server running on the same machine that simply handshakes and allows the user to communicate with an UDP-only service running in that machine, without any changes to that service.

2. Session token

In the initial WebSocket handshake, the server returns a base64 encoded value in Sec-WebSocket-UDPToken, the client must prepend all UDP datagrams with that value, and the server uses it to verify that the user has a valid session. Token expires when the TCP connection is closed. If an UDP packet is received with an invalid token, the server could either: 1. Not reply at all 2. Reply with a single byte to tell the client it has an invalid token. I don't think replying is needed, as the client can confirm whether its token is valid or not by just checking whether the TCP connection is still alive.

This is susceptible to MITM attacks, as anyone watching over the connection can steal those session tokens. But many (most?) applications wouldn't be sending sensitive over UDP (for games: user inputs, and state replication data), so I honestly don't think this is a huge worry, as anything actually sensitive could go over TCP, and if one wants to send sensitive data on top of UDP, they can wrap the protocol in the application layer.

3. Signing datagrams

Like 2, but the server actually returns something that the client signs the datagram with. The datagram would also need some extra data to prevent replay attacks and yada yada. This is still vulnerable to eavesdropping, but it's protected against an attacker modifying the payload.

4. Encryption

DTLS or whatever, you get the deal. The encryption key can even be shared between the TCP and UDP connections. You get all the warm and fuzzy guarantees of encryption, with all the overhead that comes with it.

---

In practice, if solution 1 is exposed, a lot of services would simply wrap solution 2 on top of it, and that's completely fine, but mandating 2 for everyone isn't something we should do IMO.

Solutions 3 and 4 have a non trivial overhead, and I would suggest avoiding that, as not everyone needs the guarantees it provides. Anyone wanting those can simply build them on top of solution 1 or 2.

Also, one think I haven't addressed is: should the server enforce that the source IP of the UDP packets be the same as the source IP of the TCP connection? This only really applies for solution 2 and up. That would provide some excellent guarantees against someone acquiring a session token in solution 2, then using that with a spoofed IP to amplify a DDoS. I don't think there are any networks out there in which clients get a different IP for UDP packets, so this could be a valid idea.

I think @gafferongames raised an issue about what if the server doesn't want to accept a client. The server can just refuse the WebSocket handshake, so no UDP channel is ever established. Also, any WebSocket server running out there already needs to check the handshake, as you shouldn't be accepting connections from an Origin you don't recognize.

[hach-que]

I don't think that addresses the proxying concerns I had - if cloud load balancers can't support simultaneous TCP and UDP on the same IP address, the only way to serve traffic from the same IP address is to either use Nginx in front of or on the same machine as each dedicated game server or embed HTTPS WebSocket compatible server into the dedicated game server application itself.

I'm not worried about the technical problems in connecting to the server; that's handled by the web browser. I'm concerned about the technical requirements this proposal would place on the servers themselves.

[Matheus28]

@hach-que

That's true, I forgot to address that.

I'm not able to think of a secure way to allow using a different IP than the one the WebSocket connected to. What I'm see in this proposal is serves using both UDP and TCP to communicate with clients. Connecting to the server using only UDP would be outside the scope of this proposal (since this focuses on UDP as a WebSocket extension).

Nginx in front of or on the same machine as each dedicated game server or embed HTTPS WebSocket compatible server into the dedicated game server application itself

What's the problem with this solution though? I personally already have a HTTPS WebSocket compatible server in the game server process of my games that's used to handle the ws traffic.

[hach-que]

I'm not able to think of a secure way to allow using a different IP than what the WebSocket used to connect.

netcode.io supports this, so it's entirely possible to do (it solves it with a pre-shared symmetric key).

What's the problem with this solution though?

It requires more resources to run (thus increasing costs) and makes deployments more complex.

---

The model of netcode.io; where you have an authentication server running over HTTPS with a pre-shared key between it and the running dedicated game servers makes a lot more sense to me because it doesn't require proxying at all - you can run your auth servers behind HTTPS load balancing and sit your dedicated game servers at the network edge.

They don't need to share IP addresses because of the pre-shared key, and that enables greater scalability of services. You also don't need to pull in a whole HTTPS/WebSocket stack into your dedicated game server and you don't need to proxy UDP through any load balancer.

For example, in my scenario we run game servers in a Kubernetes cluster (Docker). Multiple different game server implementations (from different customers) may reside on the same physical hardware and be exposed on the same IPv4 address. In this proposal, we'd need to:

• Run a WebSocket container on port 443 that can multiplex authorization requests based on what game servers are actually running on the machine.
• Clients that want to connect to services would have to use a secondary API to figure out what IP address to connect to for the WebSocket, since game servers for a customer may be distributed over multiple nodes. How you're expected to do TLS on a WebSocket that has to be addressed by dynamically allocated IP address isn't something that I've seen addressed here.
• This proposal gives us no protection against amplified DDoS for badly written game servers (which might not be an issue if you're a developer running a game server on their own hardware and don't believe an attack is likely, but is definitely a concern for us being a service provider).

Consider the alternative with netcode.io:

• We don't have any "same node colocation" requirements for authorization and can offer the authentication services separately with proper load balancing.
• Clients don't need to use a secondary API to figure out what authorization API to talk to; they just get a token from the API which encapsulates both authorization to connect to a server and which IP address and port to talk to.
• For game servers that support netcode.io, we just schedule their containers to run directly.
• For game servers that expect raw UDP, we place a netcode.io terminator in front of the container in the same pod, where the terminator gets the external port and the raw UDP traffic only ever travels inside the same pod network to the game server (this prevents DDoS amplification attacks). This works transparently for our customers.

[Matheus28]

netcode.io supports this, so it's entirely possible to do (it solves it with a pre-shared symmetric key).

I meant in the scope of this proposal (being an extension to WebSockets). Netcode.io initial connection happens over UDP, which does bring the benefits you mentioned.

This proposal tries to stay as close as possible to the way WebSockets already work to minimize attrition for those who already use it, and want to add UDP alongside it.

While I see how netcode.io makes your job easier given the way your infrastructure is laid out, this proposal tries to stay as close to WebSockets as possible. I would encourage (and support) a proposal from you in which a token is able to be generated on a separate server and a handshake happens over UDP. I'd just be against mandatory encryption in the protocol level :)

---

Addressing a few things on your infrastructure:

• Run a WebSocket container on port 443 that can multiplex authorization requests based on what game servers are actually running on the machine.

Use one port per game server? You don't need to use port 443 for wss.

• Clients that want to connect to services would have to use a secondary API to figure out what IP address to connect to for the WebSocket, since game servers for a customer may be distributed over multiple nodes. How you're expected to do TLS on a WebSocket that has to be addressed by dynamically allocated IP address isn't something that I've seen addressed here.

A separate API that gives the client a host to connect to is fine. You should already have an unique hostname per IP anyway. If you want to remove TLS before handing the WebSocket connection to your costumers game servers, they don't even need a certificate themselves.

• We also basically have no protection against amplified DDoS for badly written game servers (which might not be an issue if you're a developer running a game server on their own hardware, but is definitely a concern for us being a service provider).

Ask how AWS, Linode, DigitalOcean, etc. handle misbehaving servers, they just send an abuse complaint to the costumer. You'll need to handle those anyway, as nothing in your service would prevent the user from abusing the server resources.

---

Also, as a sidenote: in some networks you aren't able to use UDP at all (save for dns, everything else is filtered), so you might still want a WebSocket backend as a fallback.

Edit: I added a few (personal) suggestions related to your infrastructure

[hach-que]

I meant in the scope of this proposal (being an extension to WebSockets). Netcode.io initial connection happens over UDP, which does bring the benefits you mentioned.

This proposal tries to stay as close as possible to the way WebSockets already work to minimize attrition for those who already use it, and want to add UDP alongside it.

While I see how netcode.io makes your job easier given the way your infrastructure is laid out, this proposal tries to stay as close to WebSockets as possible.

Whatever is adopted by browsers has to support a wide-range of games: not just the games that exist on the web today, but games that want to run on the web but have held off from doing so because of the missing infrastructure.

While WebSockets might work for your particular use-case, it increases costs and complexity for games that are currently not on the web, and introduces scalability concerns for service providers that want to offer game server hosting for developers that want to deploy games on the web. I believe the web is better served by a solution that can support all kinds of multiplayer games (and you can send the UDP connection token over your TCP WebSocket and polyfill the API for sendUnreliable if you want the client JS API to match this proposal).

I would encourage (and support) a proposal from you in which a token is able to be generated on a separate server and a handshake happens over UDP.

netcode.io is a proposal that does exactly this, so that is the proposal I'd like to see pushed forward.

I'd just be against mandatory encryption in the protocol level :)

It's been covered previously, but I don't think it's practical to pursue this with browser vendors. Not only will non-HTTPS websites in future be marked as non-secure and throw up the giant red "This connection is not secure" screen (https://www.wordfence.com/blog/2017/01/chrome-56-ssl-https-wordpress/), but I very much doubt the overhead of encryption on UDP packets is significant given that you'll eventually need to be doing TLS encryption for all your WebSocket communication anyway.

Also, as a sidenote: in some networks you aren't able to use UDP at all (save for dns, everything else is filtered), so you might still want a WebSocket backend as a fallback.

There are a whole class of games (e.g. fast-paced first person shooters) where the user experience with TCP fallback would be so poor that it's not worth supporting at a protocol level - games that want a TCP fallback can do so using the WebSockets API alongside netcode.io.

[Matheus28]

it increases costs and complexity for games that are currently not on the web

Uh... netcode.io would also do the same, perhaps even worse with all the extra stuff it brings in that shouldn't be protocol level

introduces scalability concerns for service providers that want to offer game server hosting for developers that want to deploy games on the web

What scalability concerns? The client will need to find a server to connect to anyway, you can have a separate cluster that gives the client a host + port to connect to, and everything else can happen between that server and the client. Using Google's (or whoever) load balancers isn't the only way to scale a service.

There are a whole class of games (e.g. fast-paced first person shooters) where the user experience with TCP fallback would be so poor that it's not worth supporting at a protocol level - games that want a TCP fallback can do so using the WebSockets API alongside netcode.io.

That was a sidenote relating to your service.

[hach-que]

Use one port per game server? You don't need to use port 443 for wss.

So I should run a dedicated WebSocket HTTPS server per game server instance? No thanks, that's going to be very expensive (essentially you're suggesting that half the compute and memory capacity of the entire game server cluster should just be running this WebSocket HTTPS server).

A separate API that gives the client a host to connect to is fine. You should already have an unique hostname per IP anyway. If you want to remove TLS before handing the WebSocket connection to your costumers game servers, they don't even need a certificate themselves.

We don't have a unique hostname per IP, and we don't have TLS certificates on those hostnames. We can't dynamically provision internet facing hostnames (DNS) and TLS certificates for each cluster node that spins up, as these nodes are spun up by Kubernetes depending on demand for game servers.

Ask how AWS, Linode, DigitalOcean, etc. handle misbehaving servers, they just send an abuse complaint to the costumer. You'll need to handle those anyway, as nothing in your service would prevent the user from abusing the server resources.

Each container is limited in the CPU and RAM that is allocated to the game server, and these limits are applied to each game server instance. Thus even if an authorized user can manipulate the game server to crash or max out it's resources, it can't affect the health of the node it's hosted on, other game instances or the cluster itself.

With netcode.io, authorized users can only communicate with the exact server instance they're authorized to talk to. If you're authorized and you send UDP packets to another server, they get dropped by netcode.io (so no impact on CPU/RAM). If you're not authorized at all, the packets get dropped.

With this proposal, someone can just spam UDP packets to all of the servers using raw Linux sockets (outside the browser) and impact all game server instances and all nodes. It's also susceptible to amplified DDoS which allows someone to use raw Linux sockets to send a UDP packet to a game server with a modified source address (the address of the victim), and have that game server send a much bigger response to the victim in order to DDoS them. It doesn't matter if the web browser doesn't allow you to do it - it's the server endpoints that need to be secured.

As a service provider our abuse handling under the netcode.io proposal is limited to people using the services to run things other than game servers. Under this proposal it's expanded to having to deal with amplified DDoS, attacks that threaten the health of all game server instances or the cluster, and many more issues.

Uh... netcode.io would also do the same, perhaps even worse with all the extra stuff it brings in that shouldn't be protocol level

Running an instance of Nginx with a TLS WebSocket connection to each user in addition to the UDP packets is in no way comparable to decrypting UDP traffic. The costs incurred between these two things are not even on the same order of magnitude.

As a baseline, Nginx requires about 90-100Mb of memory and a percentage of CPU resources for each instance. If you're trying to claim that embedding netcode.io in an application increases RAM requires of that application by 90-100Mb, and increases CPU requirements on a level comparable to running Nginx, then I can tell you that that is not correct based on real-world measurements.

What scalability concerns? The client will need to find a server to connect to anyway, you can have a separate cluster that gives the client a host + port to connect to, and everything else can happen between that server and the client. Using Google's (or whoever) load balancers isn't the only way to scale a service.

This increases costs as mentioned previously and makes it hard to just deploy game servers into a cluster. These aren't trivial costs either, we're talking $50USD/month for even a small node in a cluster, so wasting resources on unnecessary containers is a real concern.

There are no load balancing services in existence that are comparable to Google's load balancers; they are fundamentally and architecturally different. You basically have a bunch of different load balancing architectures for HTTPS:

• Application level load balancing (in the form of Nginx) where traffic is routed to a single node which then delegates the actual handling of responses to other nodes. In this scenario you have a DNS record which points to the IP of an Nginx instance which load balances to other backend instances. Because you only have a single DNS record and you aren't updating DNS, you can't load balance across multiple Nginx edge nodes.
• DNS level load balancing (in the form of AWS ELBs) where by new load balancing nodes register their IP with DNS records, and the DNS server provides random subsets of registered IP address to clients with a low TTL (60 seconds). This is why AWS ELBs provide you with a hostname that you CNAME to, instead of an IP address for the load balancer.
• Anycast IP address load balancing (in the form of Google Cloud Load Balancing) whereby Google uses their edge node routers to distribute traffic between nodes at the point of presence. This gives you a single IP address which is routed differently on the internet using BGP routing updates.

The reason why there's no other load balancing solution like Google's should now seem obvious - it's only possible to get this scalability if you're running multiple PoPs around the world with BGP routing updates and plenty of edge nodes. You simply can't match this level of availability and scalability with your own hand rolled Nginx solution.

[Matheus28]

So I should run a dedicated WebSocket HTTPS server per game server instance? No thanks, that's going to be very expensive (essentially you're suggesting that half the compute and memory capacity of the entire game server cluster should just be running this WebSocket HTTPS server).

I'm suggesting that's ran in the game server process itself.

We don't have a unique hostname per IP, and we don't have TLS certificates on those hostnames. We can't dynamically provision internet facing hostnames (DNS) and TLS certificates for each cluster node that spins up, as these nodes are spun up by Kubernetes depending on demand for game servers.

Wildcard certificates and setup dns records when a new host goes up. I have this system for all my games and it works.

With this proposal, someone can just spam UDP packets to all of the servers using raw Linux sockets (outside the browser) and impact all game server instances and all nodes. It's also susceptible to amplified DDoS which allows someone to use raw Linux sockets to send a UDP packet to a game server with a modified source address (the address of the victim), and have that game server send a much bigger response to the victim in order to DDoS them. It doesn't matter if the web browser doesn't allow you to do it - it's the server endpoints that need to be secured.

In proposal 1, it's up to the server to drop the packet. Proposal 2 onwards the server must drop the packet, just like netcode.io.

As a service provider our abuse handling under the netcode.io proposal is limited to people using the services to run things other than game servers. Under this proposal it's expanded to having to deal with amplified DDoS, attacks that threaten the health of all game server instances or the cluster, and many more issues.

Right, and I bet you'd love to charge your clients more because of the extra overhead of netcode.io.

As a baseline, Nginx requires about 90-100Mb of memory and a percentage of CPU resources for each instance. If you're trying to claim that embedding netcode.io in an application increases RAM requires of that application by 90-100Mb, and increases CPU requirements on a level comparable to running Nginx, then I can tell you that that is not correct based on real-world measurements.

Where did I say you need to use nginx? See my very first answer. The developers that use your service should be running their server that handles websockets. They listen at port X for both TCP and UDP, and that's it. The overhead is minimal.

This increases costs as mentioned previously and makes it hard to just deploy game servers into a cluster. These aren't trivial costs either, we're talking $50USD/month for even a small node in a cluster, so wasting resources on unnecessary containers is a real concern.

$50 USD/month for a small node in a cluster? You're getting ripped off.

There are no load balancing services in existence that are comparable to Google's load balancers; they are fundamentally and architecturally different. You basically have a bunch of different load balancing architectures for HTTPS: [...]

How about the following: You use DNS level load balancing that gives a few IPs for servers in the cluster that handles load balancing clients (picking the best server or whatever). That cluster gives the client a host to connect to. That host is picked from a pool of servers that autoscale. Please explain how this doesn't scale.

For simplicity, assume that those clusters are ran on AWS or Google's cloud or whatever service handles autoscaling and availability for you.

Correct me if I'm wrong: with netcode.io, you will need that initial HTTP request to an api anyway, you can't just connect to Google's UDP load balancer and hope for the best, as you'll need a connection token.

I'm gonna be honest here, from my perspective, you to be seem heavily invested in netcode.io and refuse to see how this can be accomplished in different (and more generic) ways.

[hach-que]

I'm suggesting that's ran in the game server process itself.

That means we need a dedicated IP address per game instance (because the WebSocket runs on port 443), which means one instance per physical server. That's not practical at all and prevents clustering. It also means that each customer has to deal with the complexity of embedding an Nginx server in their deployment.

Wildcard certificates and setup dns records when a new host goes up. I have this system for all my games and it works.

I'm sure it works for you, but as I've explained, the complexity introduced means that it's not practical to do this in the system we have.

In proposal 1, it's up to the server to drop the packet. Proposal 2 onwards the server must drop the packet, just like netcode.io.

Right, it's reliant on game servers to be written securely in order for the health of the cluster to be ensured. It's preferred if the protocol simply ensures that this is the case, because then we're not dependent on independent security implementations being correct.

Right, and I bet you'd love to charge your clients more because of the extra overhead of netcode.io.

I think I've been pretty respectful in this discussion, and I don't appreciate the insinuation that somehow the motivation for a protocol like netcode.io is to charge clients more. The overhead of decrypting packets is negligible compared with say, physics simulations on most game servers, or running a WebSocket instance inside the game server.

(I mean, it's not even accurate because on our service you pay per reserved slice for game servers, and the pricing of slices is directly based on the underlying Google Cloud Platform charges for nodes)

$50 USD/month for a small node in a cluster? You're getting ripped off.

These are the prices you pay for scalable cloud computing (AWS/GCE/etc). You can't run a service like the one we do on cheap VPS hosts.

How about the following: You use DNS level load balancing that gives a few IPs for servers in the cluster that handles load balancing clients (picking the best server or whatever). That cluster gives the client a host to connect to. That host is picked from a pool of servers that autoscale. Please explain how this doesn't scale.

That involves running your own DNS server so that you can perform the specialised serving of random subsets of IP addresses for records, like AWS does.

Correct me if I'm wrong: with netcode.io, you will need that initial HTTP request to an api anyway, you can't just connect to Google's UDP load balancer and hope for the best, as you'll need a connection token.

The point is that the HTTPS service and the UDP game server doesn't have to reside on the same IP address with netcode.io. This enables the use of Google's load balancing for the HTTPS services directly.

---

I think I've made some pretty good points about why the proposal is not ideal for us, especially with regard to costs, complexity and scalability. If your response to each of these issues is "just roll your own stuff at higher costs to your company and customers", then we're not going to get anywhere in this discussion.