Try to ensure websocket IRC connections end up relatively standard

[DanielOaks]

Insp has a websocket implementation in testing as far as I know, Chary has one as well, and other software/daemons are interested in adding them.

Landing on some roughly-standard way of implementing websocket IRC connections (in terms of connection, line protocol, etc) would be very good, and help people develop general web clients that could use them.

[realJoshByrnes]

I'd be interested in this. Currently using the Primus node.js library to do this, but if a standard is decided upon, it would be great to enable support.

[syzop]

We added Websocket support in UnrealIRCd 4.0.10 in January 2017. Along with the release announcement back then was also an example URL https://www.unrealircd.org/files/dev/ws/websocket_unrealircd.html showing a very crude client implementation which can be used for testing.

One thing we did change recently (UnrealIRCd 4.0.13, 15 Aug 2017) is that on websocket connections you no longer need to send (CR)LF in input and the server will no longer send (CR)LF after each line in output. It is now simply 1 websocket message = 1 IRC message. This was on the request of a KiwiIRC developer (who also said he talked to InspIRCd) and it made sense.

UnrealIRCd accepts both BINARY and TEXT frames incoming. Outgoing messages are sent as BINARY. Due to the way IRC works a message may contain characters that would be considered invalid UTF8. If sent as TEXT frame, this could cause the client to abort the connection whenever someone in a channel speaks "malformed utf-8". Hence, BINARY is used which does not have this limitation.

[realJoshByrnes]

@syzop I'm guessing that a binary message ending in a CRLF combination would also work as expected in your implementation?

[syzop]

I'm guessing that a binary message ending in a CRLF combination would also work as expected in your implementation?

@ozjd Yup, exactly ("be liberal in what you accept")

[SadieCat]

Are Libera still objecting to text frames?

[jesopo]

uncertain; pending a discussion on potentially quantifying how much non-utf8 data we deal with. the only feasible outcome i can see is dropping nonutf8 at ingress, and that depends on how many users it would disrupt

[slingamn]

From my point of view, the current proposal is still a draft and I'm totally open to going back to a text-only version. In that case I would tentatively propose:

1. Keep subprotocol negotiation (it might be useful in future)
2. Clients SHOULD request text.ircv3.net, but if they don't, that's the assumed default
3. Servers MUST accept text.ircv3.net if it is present
4. binary.ircv3.net is no longer mentioned

I don't think this would create significant incompatibilities with any implementations that currently exist.

[SadieCat]

Can we please not relitigate this stupid pointless debate please @slingamn. The only thing the compromise is blocked on currently is Libera determining how much traffic is non UTF-8.

[slingamn]

I'm clearly misunderstanding something because I don't see why that would block the proposed compromise, which AIUI was merged as a draft with approval from Libera's representatives?

[realJoshByrnes]

I know this is an old issue (although still a proposal), but I was writing a small web client and encountered an issue...

Some servers send multiple messages in a websocket message (terminated by \r\n), some don't.

In an ideal world, we could remove the \r\n - but if you use a websocket proxy for an older IRCd, this won't happen as it's streaming the data between ws/tcp.

From a server side, removing \r\n works fine for clients sending \r\n and clients not sending it.

From the client's perspective, it's a bit trickier - When waiting for data from a server, do we proceed after receiving a message, or wait for the \r\n?

If the client waits for the \r\n we may never receive a valid message if the server isn't sending them. If the client doesn't wait, we may miss the rest of the message.

IMHO if the client hasn't specified a protocol, it would be better to send/receive the \r\n - and if it requests text.ircv3.net or binary.ircv3.net then we can safely remove it.

From what I can see, there is one WS-IRC gateway that will resolve this, but requires every single legacy server to use this to enable websockets. Existing WS-TCP proxies can stream data without inspecting/manipulating the data.

[SadieCat]

Some servers send multiple messages in a websocket message (terminated by \r\n), some don't.

Which servers? Because any server which is doing this is violating the spec and has been for quite a few years now. They should be sending one IRC message per WebSocket message.

Existing WS-TCP proxies can stream data without inspecting/manipulating the data.

Given most servers require you to explicitly whitelist your WebSocket clients I'm not sure arbitrary WebSocket to TCP proxies not intended for IRC are something we should be concerned about.

[realJoshByrnes]

You're right, it does break the work-in-progress spec (which is not recommended to be implemented), but that spec also breaks the "We intend for all the specs we put out to be backwards-compatible" mantra from IRCv3.

In-fact, removing the \r\n breaks the message format specified by IRCv3 itself.

I'm on two servers that send the messages with \r\n, although they are not IRCv3 servers. I was pointing out that the proposed spec breaks backwards compatibility when someone adds WS to IRC server.

From RFC1459:

The IRC protocol has been developed on systems using the TCP/IP network protocol, although there is no requirement that this remain the only sphere in which it operates.

and moving it to WS seems like a natural thing to do, but changing the messaging format breaks backwards compatibility (non-IRC).

Given most servers require you to explicitly whitelist your WebSocket clients

I've just tried a few random/popular ones, and they seem to accept my connection without being whitelisted?

I'm not sure arbitrary WebSocket to TCP proxies not intended for IRC are something we should be concerned about.

If IRCv3 is attempting to create a spec for WebSocket to IRC, of course it should be a consideration what should happen when a client expects messages (as defined in the RFCs).

[SadieCat]

You're right, it does break the work-in-progress spec (which is not recommended to be implemented), but that spec also breaks the "We intend for all the specs we put out to be backwards-compatible" mantra from IRCv3.

IRC WebSocket implementations existed in the wild before the IRCv3 spec was even written and every major client and server implementation uses the current specified behaviour. Even if the spec isn't finalised (which it kinda is, we're just awaiting ratification) breaking compatibility with every IRC WebSocket implementation in the wild is more of a concern than adding support for arbitrary WebSocket gateways which were never intended to be used for IRC.

In-fact, removing the \r\n breaks the message format specified by IRCv3 itself.

There's nothing in here that prohibiting changing the wire format as long as they don't break non-supporting IRC implementations. Implementations have already had to do this once before to add tag support. WebSockets are a new feature so nothing is being broken by requiring that IRC messages are encapsulated within WebSocket frames instead of being CR+LF terminated messages. Doing it this way was selected as it makes parsing easier for web clients.

I've just tried a few random/popular ones, and they seem to accept my connection without being whitelisted?

Which? By not implementing the origin whitelisting for in-browser WebSocket connections these networks are ignoring the security recommendations in the IETF WebSocket spec and are putting themselves at significant risk of drive-by spam attacks. Non-browser connections don't need origin whitelisting but they're not a significant use case for WebSockets as they can make normal TCP connections.

[realJoshByrnes]

IRC WebSocket implementations existed in the wild before the IRCv3 spec was even written and every major client and server implementation uses the current specified behaviour.

Perhaps now because of this draft, this is mostly the case. A quick google search for Websocket IRC (or similar) returns a bunch of results where people are suggested to use WS->TCP gateways.

Even if the spec isn't finalised (which it kinda is, we're just awaiting ratification)

It either is, or isn't.

breaking compatibility with every IRC WebSocket implementation in the wild is more of a concern than adding support for arbitrary WebSocket gateways which were never intended to be used for IRC.

Which servers/clients would break by sending a \r\n where a protocol wasn't negotiated (as per the suggestion)? It certainly shouldn't break ANY that have implemented the current draft.

[realJoshByrnes]

There's nothing in here that prohibiting changing the wire format as long as they don't break non-supporting IRC implementations. Implementations have already had to do this once before to add tag support.

I forgot to reply to this one. Tag support isn't enabled unless there's a CAP negotiation, and the client specifically requests this.

Which? By not implementing the origin whitelisting for in-browser WebSocket connections these networks are ignoring the security recommendations in the IETF WebSocket spec and are putting themselves at significant risk of drive-by spam attacks. Non-browser connections don't need origin whitelisting but they're not a significant use case for WebSockets as they can make normal TCP connections.

This is way off-topic. Servers can choose who to allow or not allow.

I get that you can't be bothered changing anything, and that's understandable. What is the point of having an open group for discussion if rare use cases just get shot down without any consideration? It just means everything gets done independently and nothing becomes a standard.

https://xkcd.com/927/

[progval]

@realJoshByrnes We are talking about three different protocols here:

1. normal IRC over TCP
2. WebSocket IRC, which is defined by our draft
3. IRC over TCP over Websocket, which is what you describe

In-fact, removing the \r\n breaks the message format specified by IRCv3 itself.

it does not, because the message format is for protocol 1, and our spec draft for protocol 2 explains that it uses a slightly different format

and moving it to WS seems like a natural thing to do, but changing the messaging format breaks backwards compatibility (non-IRC).

it does not, because servers implementing protocol 2 would reject clients that don't support the binary.ircv3.net or text.ircv3.net subprotocols.

Existing implementation of protocol 3, both clients and servers, can keep doing what they are doing by not using either of these subprotocols. It's still as non-standard as it ever was, but will keep working. It's also pointless for IRCv3 to write a spec intended for generic WS-TCP gateways because they would never implement it.

Servers could even implement both protocols 2 and 3 if they wanted to, switching between the two based on subprotocols.

Does this answer your concerns?

[SadieCat]

Tag support isn't enabled unless there's a CAP negotiation, and the client specifically requests this.

This specification changes nothing for existing IRC software which will still send/receive CR+LF terminated lines over TCP as previously specified. An existing non-supporting client won't be connecting to an IRCv3 WebSocket server endpoint unless it specifically adds support for the spec. This is entirely backwards compatible with all mainstream IRC software.

What is the point of having an open group for discussion if rare use cases just get shot down without any consideration?

This has been discussed at length previously in other issues and on IRC which is how we ended up with the current specification. You're free to go and read those discussions if you want to learn how we ended up with one WebSocket message containing one #ircv3 IRC message.

It just means everything gets done independently and nothing becomes a standard.

Every mainstream IRC implementation follows the spec as written so I would say we are doing pretty good when it comes to avoiding incompatible implementations.

I don't think we should be concerned about arbitrary WebSocket->TCP gateways which are not intended for IRC use because the IRCv3 WebSocket specification is not a protocol-agnostic transport system so any non-IRC WebSocket to TCP translation software is irrelevant in the context of this specification (and even they weren't incompatible using them for IRC is would be problematic as a translation gateway needs support for IRC specific functionality like the WEBIRC spec in order to be useful).