Task: Overall conceptual design of new version of the WebSocket server library

[ghedipunk]

Note: This is a living document. Please make suggestions to change it while this task is open.

Primary goals

Implement a fully RFC 6455 compliant WebSocket server library that

• Is easy to use
• Has full unit test coverage
• Is secure by default
• Allows extensive, modular customization

  Considerations

• Minimum requirements: PHP 5.5. It could very well be fully compatible with earlier versions of PHP, but we won't guarantee it.
• Non-blocking IO by default. This is obvious with the network IO, but also applies to reading from file, such as when pulling data from a PHP session.
• Unit testing is best when using OOP design principles that are often overlooked (and often mis-implemented): Separation of Concerns, Dependency Injection, single purpose classes, code to the interface not the implementation.
• Users should be able to change nearly every aspect of the library by implementing the appropriate interface and injecting it into the application, from the network IO, event loop, message queues, down to how different apps behave.
• We want to limit our external dependencies down to just what ships with a sane Linux distro's PHP, and have advanced options that require external dependencies be optional. That is, the PECL module event, which depends on libevent can be shipped with the core library, but the default event loop would be a while loop with socket_select(). If we decide to use dependency injection, we probably can't use a mature DI container like Dice, because DI would be integral to the library (or at least integral to allowing uesrs to add their own modules and extend functionality) and we can't have an external dependency on Dice since it doesn't ship with PHP.

  Initial Design

  Key components

• Network IO
• Event loop
• Messages (Inbound and outbound)
• Message Queues (Potentially many queues, but at least an inbound and outbound one)
• Client Connection (Formerly WebSocketUser class)
• Message Router
• Apps

  Network IO

Directly handles sending and receiving messages. Responsible for setting up the socket(s), including any TLS settings. Not responsible for the WebSocket framing and deframing.

Network IO Interface:

Note: PHP 7.0 function syntax for clarity of documentation of scalar types and return type, but actually implemented to be PHP 5.5 compatible

public function sendMessage(string $framedMessage, resource $clientSocketResource) : void;
public function getPacket(resource $clientSocketResource) : string; // a packet may not be a full WebSocket message (IP fragmentation, among other reasons), or may be multiple messages (Nagle's Algorithm, etc.)

Dependencies

• Message queues

  Event Loop

Controls process flow depending on IO and timing events

Event Loop Interface

public function run() : void;

Dependencies

• Network IO
• Message Router

  Inbound Message

Holds a single message and a reference to the sender's socket resource.

Inbound Message Interface

public function getMessage() : string;
public function getSender() : ClientConnection;

Dependencies

• ClientConnection

  Outbound Message

Holds a single message and its destinations.

Outbound Message Interface

public function setMessage(string $message) : void;
public function setDestinations(array $connections) : void;
public function setDestination(ClientConnection $connection) : void;
public function addDestinations(array $connections) : void;
public function addDestination(ClientConnection $connection) : void;

Dependencies

None

Router

Routes messages between network IO queues and client applications.

Router Interface

public function registerApplication(Application $application, string $path) : void;
public function dispatchMessage(Message $message) : void;

Dependencies

• Outgoing and incoming message queues

  Application

A client program

Application Interface

public function tick() : void;
public function onMessage(Message $message, Connection $connection) : void;
public function onOpen(Connection $connection) : void;
public function onClose(Connection $connection) : void;
public function onError(Connection $connection) : void;

Dependencies

• Router

  Client Connection

Keeps track of a WebSocket client

Client Connection Interface

public function getResource() : resource;
public function getHeaders() : array;
public function getSessionValue(string $key) : (any type); // May block if another script has that session file open.
public function setSessionValue(string $key, (any type) $value); // May block if another script has that session file open.

// Not convinced on these right now, but they match the [client side API](https://www.w3.org/TR/websockets/#the-websocket-interface).
public function getReadyState() : int; // CONNECTING = 0; OPEN = 1; CLOSING = 2; CLOSED = 3;
public function getExtensions() : array;
public function getProtocol() : string;

Dependencies

None

(More may come later, but I don't want to leave this page open over the weekend and risk having a power outage cause me to lose this work in progress.)

[Xaraknid]

You should take that time to match major browser method name. That will help dev to prevent confusion and ease the use between server and client side in my opinion. ->newConnection ->onopen ->receiveMessage ->onmessage ->closedConnection ->onclose

Also I'm still not convince why you need a message queue can you explain it?

[RobertMenke]

Thanks for all the hard work, ghedipunk, the server is really easy to setup and use. One thing that may be helpful is a way to access query strings passed by the browser when creating a web socket connection. I apologize if that's already an option or something I'm overlooking, but it would certainly be helpful to send data on a user before they've interacted with the application.

[ghedipunk]

@Xaraknid Great point about the onopen, onmessage, and onclose method names. I'll make adjustments to the first post in this thread soon.

Message queues are a solution to the problem of IP fragmentation plus RFC 896 (Small packet problem, aka Nagle's Algorithm), where we can't be sure that we've collected one message, multiple messages, or parts of a message until we've started inspecting its contents.

With a message queue, we can hold a fragmented message in a buffer until we receive it, and receive multiple messages at the same time with the user applications treating them as single distinct messages without needing to be concerned about the details of TCP/IP.

@RobertMenke That data is already available in both the headers and requestedResource properties of the WebSocketUser object.

For example:

$user->headers['get']
$user->requestedResource;

[Xaraknid]

Maybe i'm wrong but having message queues will serve for sending from server to data a mean to stack multiple message. What you talk about is receiving you intend to have it for incoming message stacking them instead or processing them as they are receive?

I'm not against buffering / stacking but it's seem odd to stack it with message Queue.
How this will work, doing one full round of read and stack every message to be send after that ? Problem : potential high Roundtrip time between first message receive and first message send.

How will you clean/dump message for client disconnected ? Problem : will take cpu time to search for every occurence of ghost message depend on the size of the stack.

Instead of stacking vertically and stack multiple mini-wire on top of each other, mixing all connection together . Stack them inline you'll only need one wire per connection. Will be easier to clean/dump the stack if a connection close.

EDIT: To be exact you'll need 2 wires in and out.

[ghedipunk]

Yep, that convinces me, there's no need for message queues between the network IO layer and the rest of the system.

The way I was envisioning it at an implementation detail layer, was once an TCP segment was available in the read buffer, we would examine it for complete messages and multiple messages. Then, once it had complete messages, send all messages for that socket to the queue, then have the router process the queue before coming back to check the next socket for packets.

It occurs to me, actually thinking about this specific implementation, that there is no difference between a formal message queue where the network IO layer enqueues a batch of messages and the router dequeues that batch of message, and just having the network IO layer run through the messages in a foreach loop, calling a message dispatch method on the router for each message... and a message queue serving that purpose adds unnecessary complexity.

(Of course, we shouldn't depend on details of how the network IO layer batches and dispatches messages; we need to program for the interface, not the implementation, because the user very well could (and if they need to, should) replace the network IO layer, or the router layer, or any other component.)

You do bring up a good question, regardless of whether we'd use a queue or we dispatch directly to the router:

How will you clean/dump message for client disconnected ?

We'll need a buffer for each client to hold partial incoming messages.

To be secure by default, we have to consider a slow DoS attack, such as Slowloris, where an attacker could open multiple sockets and send messages very slowly (or send very large messages that don't have the fin bit set) eating up resources and preventing legitimate users from connecting.

I'll need to figure out what else can be removed, besides message queues, for simplicity's sake.

[Xaraknid]

That kind of DoS attack are relevant for http because after request is processed the connection is closed, but are irrelevant in websocket because connection stay up. You can limit connection per IP but can prevent multiple person from corporation/school/etc to access your site as they share the same IP.

I would be more concern about malicious client holding handshake header indifinately. That could be fix with a timeout on the handshake you can match major browser I think it's 2 seconds.

You pinpoint something about message size it's important let say we are conservative and 1000 clients sending a message ( from 1 or more frame doesn't matter ) of 1M. That mean you could have a peak memory usage of 1G just for holding message.

[philek]

This discussion is a little old, so I hope you won't me bringing it back up.

There has been some discussion regarding messages incoming to the server, however I think outgoing messages will also be an issue. Unless I'm mistaken, if you are going to implement non-blocking sockets (which is incredibly important) you will also need to have some kind of separate outbound buffer and/or queue for each client connection.

If that is the case would it not be simpler to have a ClientConnection->queueMessage(OutboundMessage $message) method to queue them directly?

[Xaraknid]

@philek : the method send() play that role allready.

In Blocking mode send() method call the function to send data and didn't return untill it's completely sent.

In non-blocking mode send() method call the function to send data and if not completely sent. add unsent data to buffer + open write flag.

That is the "standard" network pattern and easier to implement for multiple network library ( socket, stream, libev,libevent )

After that if you need more specific network pattern it's up to you I guess. As maybe process all incoming data and push data in buffer and only call once send function, this patern will work with stream and socket library but not so well in event base library.

Buffer play different role depending on the direction : Read buffer : (required) use when receiving data. Write buffer : (optional ) use when unsent data detected.