WebSockets support

[meebey]

Right now SmartIrc4net only supports bare TCP sockets to connect to IRC servers but some IRCd implementations support WebSockets and thus SmartIrc4net should provide internally an abstraction that can cope with this. This was already troublesome when proxy support was added, it had to be glued into the connection handling itself, instead of being "stackable".

[meebey]

We will need to define an interface that covers all operations that the library need to do in order to read/write messages from the underlying connection.

[meebey]

And how the user of the library picks the underlying connection type, or pass it as argument to some method. @djkaty can you pastebin or gist your websocket code so I get some idea what code needs to be glued into SmartIrc4net for this to work?

[djkaty]

I'm not sure if you're confused or I've misunderstood something but WebSocket isn't really a "socket" in the traditional sense, it's a session protocol like IRC:

IP --> TCP/UDP (transport layer / sockets) --> IRC/HTTP/WebSockets (session layer) --> message payload, ie. ":foo!foo@bar PRIVMSG #someroom hello" (presentation layer)

The current IrcConnection object establishes a TCP connection to a host which is expecting a classic IRC client, and communicates using the IRC protocol.

The current uncommitted changes i have made allow a boolean flag to be set in IrcConnection which changes its behaviour such that it will establish a TCP connection to a host which is expecting a WebSockets IRC client, negotiates the WebSockets protocol, and communicates using the IRC protocol, the mesages of which become the payload data of WebSockets frames (on the standard TCP connection). See RFC 6455 for details.

You can think of it like http. Imagine if every irc message (the payload) went in an http body and had http headers sent first, and the same for server responses. In this hypothetical scenario, http sits "between" tcp and the irc message protocol, and the irc messages are said to be formatted as http-compliant. This is exactly how WebSockets works when it comes to IRC. The underlying socket and irc protocols are identical, but each irc message is formatted to be WebSockets-compliant. Usually this means prepending a few carefully calculated header bytes before the line of text and then masking the line (see sections 5.2 and 5.3 of the RFC).

Normal: TCP + IRC With WebSockets: TCP + WebSocket + IRC

The proxy comes in at the TCP connection level and merely diverts the connection to a different IP address and port. I think both including proxy support in the IrcConnection class direct as it is now, or subclasing an IrcProxyConnection are both valid solutions. But if you do the latter, you lose the ability to inherit proxying if IrcConnection is subclassed again. So i think it is correct as it is.

Hope that clarifies it. So what's needed is not a socket abstraction but just a decision on whether to keep the boolean flag to enable WebSockets or whether payload formatting/parsing should be dependency injected using an interface. By this i simply mean, preparing an already constructed irc protocol mesage for sending or parsing a received frame into an irc message. When using IRC directly, the steps for this are "do nothing". For WebSockets, the steps are to add and strip the header bytes. The third option was to derive a new WebSocketIrcConnection class, but i think passing an interface implementation as a parameter to IrcConnection like "IrcProtocol" or "WebSocketIrcProtocol" that can subvert Connect, ReadLine and WriteLine would be the more elegant solution.

[djkaty]

https://gist.github.com/djkaty/c16eaf6a0aba9e1af6ee

Search for '_IsWebSocket' to find all the changes.

Please note this is not what I intend to commit, it is a quick and dirty implementation, not a robust one (although it does essentially work). I will wrap up the WebSocket-specific parts of Connect, ReadLine and WriteLine into their own class/methods with proper masking, more robust header parsing and comments.

Edit: Oh, and fix the UTF-8 encoding stuff as well

[meebey]

Thank you for the explanation, my understanding of WebSockets was indeed incomplete. I thought WebSockets negotiates via HTTP and upgrade to another socket/port a bit like FTP does, but instead it upgrades the HTTP session/request from HTTP to a WebSocket session to transfer arbitrary data. I wonder why HTTP CONNECT wasn't used for this instead as that worked for decades with HTTP proxies to tunnel SSL :-D

5.2 and 5.3 made clear what SmartIrc4net would need to cope with, I would say though that Smartirc4net should not provide the needed WebSocket protocol functionality but instead an external class, probably library. https://github.com/sta/websocket-sharp looks healthy and has a compatible license.

Also SmartIrc4net should not try to speak bare HTTP, but use the HTTP client of the base-class-library of .NET if possible.

[meebey]

And yes, the special part that SmartIrc4net would need to deal with are the framed messages but SmartIrc4net should not break the protocol layers and let websocket-sharp handle the WebSockets protocol instead.

[meebey]

Which IRC network does support WebSockets? Is Twitch currently the only one? Because what I meant with stacking is that one could initiate WebSockets outside of SmartIrc4net say by using websockets-sharp and then only pass the IRC connection/socket stream to SmartIrc4net, instead of doing it in SmartIrc4net itself.

Oh each IRC message is a WebSocket frame? Then this is tricky, as one needs to inject the IRC message from the outside to SmartIrc4net and SmartIrc4net would no longer own the network socket to do the connection handling.

[meebey]

The third option was to derive a new WebSocketIrcConnection class, but i think passing an interface implementation as a parameter to IrcConnection like "IrcProtocol" or "WebSocketIrcProtocol" that can subvert Connect, ReadLine and WriteLine would be the more elegant solution.

With regards to interfaces, one easy way to allow alternative implementation is by using virtual methods, but that wouldn't be very clean as one wouldn't know which methods belong to the same interface. I know that some code in SmartIrc4net is not optimal for this, the code is mostly from 2002 :-D

[djkaty]

Agree on all points. The library I had been eyeing is WebSocket4Net ( https://github.com/kerryjiang/WebSocket4Net )

Using the .NET HTTP classes is awkward because they open and close a socket for each request, but it's a moot point because WebSocket4Net does the negotiation anyway.

There are other IRC networks which support WebSockets besides Twitch; one advantage of it is that web browser IRC clients can use HTML 5 instead of Java or Flash, another is that PING/PONG will eventually no longer be needed. Right now it doesn't confer any meaningful advantages in SmartIrc4net, but I know at least that for my work with Twitch they are moving towards preferring WebSockets connections and it's good to have the support ready before raw IRC gets deprecated.

Regarding the interfaces, my idea was something like this:

public interface IProtocol {
  public string CreateFrame(string);
  public string ParseFrame(string);
}

public class IrcProtocol : IProtocol {
  public string CreateFrame(string ircmessage) { return ircmessage; }
  public string ParseFrame(string frame) { return frame; }
}

public class WebSocketIrcProtocol : IProtocol {
  public string CreateFrame(string ircmessage) { /* do something with external WS library to add the frame data */ }
  public string ParseFrame(string frame) { /* do something with external WS library to strip the frame data */ }
}

Then over in IrcConnection:

private IProtocol _SessionProtocol; // supply in constructor or whatever
...

public void WriteLine(string data, Priority priority) {
...
_WriteLine(_SessionProtocol.CreateFrame(data));
...
}

public void ReadLine(bool blocking) {
...
data = (string)(_ReadThread.Queue.Dequeue());
...
data = _SessionProtocol.ParseFrame(data);
...
}

I just typed that in here raw without testing it so it's just a concept, the idea is to send a new IrcProtocol() or new WebSocketIrcProtocol() as a constructor argument to IrcConnection, this gets stored in _SessionProtocol and the appropriate mangling will be done in ReadLine() and WriteLine() (you would add another method to the interface to do the negotiation after the TCP socket is connected in Connect() ). The benefit of this approach is that if for some reason in the future someone wants to mangle the data in a different way, they can just provide their own custom implementation of IProtcol.

[djkaty]

And btw yes, it is tricky because all the 3rd party libraries expect to manage the sockets themselves, and all of the protocol handling is in private methods. That is why I initially wrote the code the way I did, munging the protocol myself. Options that I see are:

1. write code to handle the protocol ourselves (I can do this)
2. copy paste relevant parts of a 3rd party library if the license permits it
3. abstract ownership of the socket - which is alot of work :(

[meebey]

4th fork a lib with the best API and compatible license and modify the way it handles the socket, we basically want to inject the socket and also that we own the socket, as SmartIrc4net is doing the reconnect stuff etc. With git this is easy to do, I did this forked submodule for many libs that Smuxi needs.

[djkaty]

Actually last night i figured out (in my head) how to do option 3 and also use websocket-sharp. Im away for a couple of days now but the basic idea is to wrap the current tcp socket protocol into eg. IrcSession, wrap calls to websocket-sharp into eg. IrcWebSocketSession, and make them both implement an interface which has events that fire on message receive and disconnection. Pass a created instance of one of these to IrcClient.Connect and make IrcConnection subscribe to the events. The read queue and write threads and buffers will remain in IrcConnection, but the read thread will be moved to IrcSession because we dont need to repeatedly poll the WebSocket, hence changing to an event driven model.

This approach:

• requires no changes to 3rd party libs
• allows each session/protocol type to support its own set of connection parameters, eg. Proxy, encryption, decompression or whatever else is appropriate for the protocol, set before we call IrcClient.Connect. We would lose this functionality with the other solutions
• allows new protocols to be supported by users without changing any existing code in SmartIrc4net

The interface implementer would own the socket.

What do you think?

[meebey]

I like this new approach. We should try to keep API changes low though, as in leave it backwards compatible for users who use the "usual" way to connect to IRC servers via hostname + port. Yeah I am also a bit busy right now with vacation and Smuxi 1.0 release at the moment ^^

[meebey]

I am wondering how auto-reconnect will work this way, probably forcely disable in IrcConnection if the connection was "passed" instead of self initiated.

[djkaty]

Okay, I will whip up some kind of draft code and we can move things around from there.

[djkaty]

The connection will always be self-initiated btw, IrcConnection will call some connect method on the IrcSession-implementer. If the session gets disconnected, IrcConnection will receive an OnDisconnect-style event and just re-connect.

[djkaty]

Okay I've got it all working after 10 hours of marathon coding, including WebSockets as an example secondary protocol using websocket-sharp. I'm just gonna juggle some things around and then I'll throw up a PR tomorrow :)