UDP as a WebSocket extension

Here's my proposal for a WebSocket extension that would expose UDP securely to web pages:

During the initial WebSocket handshaking, the client signals support with the header Sec-WebSocket-Extensions: udp. The server replies back with the port that the client can use for UDP: either Sec-WebSocket-Extensions: udp (port 80 or 443, depending on whether it's ws or wss) or Sec-WebSocket-Extensions: udp=1234. Note that the host for the connection would be the same as the http request host.

For web developers, the WebSocket object would have a sendUnreliable method to send a datagram to that UDP port, and once the first datagram is sent, the server then knows where to send UDP packets to (and that the client supports UDP; initial packet has to come from client because of NAT). The WebSocket object would also have "udp" added to its extensions property, so the client knows that the server supports UDP. Sessions would be handled entirely by the developer and don't really need to be dealt with in the protocol.

Now, some issues:

SSL: It makes sense to not wrap datagrams using DTLS, since the entire purpose of using UDP is to minimize CPU and latency costs. While a possible solution is using DTLS for secure websockets (and not using it for insecure websockets), this causes problems with the fact that https origins can only use secure websockets, and game developers would force their users to only use http (basically all browser MMOs out there already redirect https to http because of this). My proposal is always using insecure datagrams, since anything supposed to be secure can just go over TCP.
Raw datagrams: Raw datagrams have a few issues, possibly the biggest ones are IP fragmentation and flow control. Should raw datagrams be exposed, those would be pushed onto developers to deal with, I don't really see the problem with that, as I don't think this should be solved at the protocol layer. My proposal is to just send raw datagrams after the initial handshake is made.

If there aren't major objections to this approach, it is possible to write a browser extension that exposes this interface, and this would get the ball moving faster for browser developers to take a look at the proposal.

That means we need a dedicated IP address per game instance (because the WebSocket runs on port 443), which means one instance per physical server. That's not practical at all and prevents clustering. It also means that each customer has to deal with the complexity of embedding an Nginx server in their deployment.

WSS does not need to run on port 443. Try connecting to "wss://your.server:1234/".

I think I've been pretty respectful in this discussion, and I don't appreciate the insinuation that somehow the motivation for a protocol like netcode.io is to charge clients more. The overhead of decrypting packets is negligible compared with say, physics simulations on most game servers, or running a WebSocket instance inside the game server.

I'm sorry, that was intended to be a joke. The overhead is non-negligible if you're IO bound, which some of my games are.

These are the prices you pay for scalable cloud computing (AWS/GCE/etc). You can't run a service like the one we do on cheap VPS hosts.

Those "cheap VPS hosts" have an API. It's not hard to write something to collect CPU usage on all your servers and create/destroy servers as you need them. It's completely possible. This is what I do for my game servers, it's the difference between paying $300k/mo in servers, and $10k/mo (this is not a hypothetical example). Especially when you're moving petabytes of data every month.

Right, it's reliant on game servers to be written securely in order for the health of the cluster to be ensured. It's preferred if the protocol simply ensures that this is the case, because then we're not dependent on independent security implementations being correct.

Proposal 2 and up solves that.

That involves running your own DNS server so that you can perform the specialised serving of random subsets of IP addresses for records, like AWS does.

You can also use AWS's Route53 and just add/remove records as you add/remove servers.

The point is that the HTTPS service and the UDP game server doesn't have to reside on the same IP address with netcode.io. This enables the use of Google's load balancing for the HTTPS services directly.

Yep, this proposal doesn't meet that.

If your response to each of these issues is "just roll your own stuff at higher costs to your company and customers", then we're not going to get anywhere in this discussion.

But it's not a higher cost, it's a much much lower cost to you and your costumers. Not using Google's Cloud and instead using those "cheap cloud VPSes" you mentioned (plus, once you're big enough, dedicated servers or your own servers) is a difference in 20x to 100x in costs (remember the $300k/mo vs $10k/mo I mentioned). The price that the big providers charge for bandwidth is insane, and you can decrease your costs substantially by rolling out your own solution for server management. I still use AWS for some services like RDS and S3. But I have my own infrastructure for handling my game servers, which abstracts away the cloud providers and just picks whatever is cheapest at the moment, since game servers are pretty disposable.

Regardless, just because this proposal doesn't work on a specific provider because they don't support a load balancer that does both SSL termination and UDP in the same endpoint, it doesn't invalidate it.

Coming back to some hard numbers, are you sure that encryption is going to add that much overhead? Libsodium is pretty darn fast.

Just as a quick benchmark I spun up an simple udp loopback server on my laptop(i7 but not a beefy hosting server):

> cargo bench

running 2 tests
test bench_ls_udp ... bench:       4,823 ns/iter (+/- 374)
test bench_udp    ... bench:       2,844 ns/iter (+/- 1,267)

test result: ok. 0 passed; 0 failed; 0 ignored; 2 measured; 0 filtered out

With a 512b packet encrypted and decrypted that's ~1.9 microseconds per send + recv which is pretty small. Even with 1000 clients at 15Hz you're still only looking at 28ms out of 1000ms or less than 3% overhead.

If you're IO bound like you said then you'd actually get it for free if you use completion IO. You can do the work while you're waiting for the send buffer to clear.

Here's the full source if you want to replicate it:

#![feature(test)]
extern crate test;
extern crate libsodium_sys;
use test::Bencher;

use std::net::UdpSocket;
use std::thread;

const ITER_SIZE: usize = 1;

#[bench]
fn bench_udp(bench: &mut Bencher) {
    let mut listen = UdpSocket::bind("127.0.0.1:8000").unwrap();
    let mut send = UdpSocket::bind("127.0.0.1:8001").unwrap();

    send.connect("127.0.0.1:8000");
    listen.connect("127.0.0.1:8001");

    let recv = thread::spawn(move || {
        let mut data: [u8; 512] = unsafe { ::std::mem::uninitialized() };
        for _ in 0..ITER_SIZE {
            listen.recv(&mut data).unwrap();
        }
    });

    bench.iter(|| {
        for _ in 0..ITER_SIZE {
            let mut data: [u8; 512] = [0;512];
            for i in 0..512 {
                data[i] = i as u8;
            }

            send.send(&data).unwrap();
        }
    });
}

#[bench]
fn bench_ls_udp(bench: &mut Bencher) {
    unsafe {
        let mut listen = UdpSocket::bind("127.0.0.1:8000").unwrap();
        let mut send = UdpSocket::bind("127.0.0.1:8001").unwrap();

        send.connect("127.0.0.1:8000");
        listen.connect("127.0.0.1:8001");

        libsodium_sys::sodium_init();

        let nonce = [0; 12];
        let mut key: [u8; 32] = unsafe { ::std::mem::uninitialized() };
        libsodium_sys::randombytes_buf(key.as_mut_ptr(), key.len());

        let recv = thread::spawn(move || {
            let mut data: [u8; 512 + libsodium_sys::crypto_aead_chacha20poly1305_ABYTES] = unsafe { ::std::mem::uninitialized() };

            for _ in 0..ITER_SIZE {
                listen.recv(&mut data).unwrap();

                let mut out: [u8; 512] = ::std::mem::uninitialized();
                let mut read = out.len() as u64;

                let result = libsodium_sys::crypto_aead_chacha20poly1305_ietf_decrypt(
                    out.as_mut_ptr(),
                    &mut read,
                    ::std::ptr::null_mut(),
                    data.as_ptr(),
                    data.len() as u64,
                    ::std::ptr::null_mut(),
                    0,
                    &nonce,
                    &key);
            }
        });

        bench.iter(|| {

            for _ in 0..ITER_SIZE {
                let mut data: [u8; 512] = [0; 512];
                for i in 0..512 {
                    data[i] = i as u8;
                }

                let mut out: [u8; 512 + libsodium_sys::crypto_aead_chacha20poly1305_ABYTES] = ::std::mem::uninitialized();
                let mut written = out.len() as u64;

                let result = libsodium_sys::crypto_aead_chacha20poly1305_ietf_encrypt(
                        out.as_mut_ptr(),
                        &mut written,
                        data.as_ptr(),
                        data.len() as u64,
                        ::std::ptr::null_mut(),
                        0,
                        ::std::ptr::null(),
                        &nonce,
                        &key);

                send.send(&out).unwrap();
            }
        });
    }
}

I'm on Rust nightly and couldn't compile your code (complaints about size of nonce, and for that libsodium-sys lib, no such function crypto_aead_chacha20poly1305_ietf_encrypt, but it does exist without _ietf). After fixing those two issues, it did run but panicked with a connection refused error (on udp...?). Anyway, I don't use Rust so that's as far as I tried to get it to work.

Since I couldn't run the benchmark on my servers, I'm not really able to comment on the overhead on my end (remember servers are clocked lower than consumer CPUs, and some lack recent instructions that can speed up libsodium). But it doesn't look too bad, but it's still real, I would still prefer it being optional rather than enforced by the protocol.

Oh yeah, sorry you'll need my fork since they haven't brought over the ietf variants in libsodium_sys:

[dependencies]
libsodium-sys = { git = "https://github.com/vvanders/sodiumoxide.git" }

Hey, sorry for the delay, I've been a bit busy recently, I'll look into this soon-ish. Thank you.

Maksims / web-udp-public

UDP as a WebSocket extension #6