Suggestion: standardize the startup (001-005) with addition to specifications, and/or some extensions

[ghost]

Edit: to keep this updated a bit with the discussion, my initial cap-099-draft has been somewhat superseded by various suggestions, so let me put here what the discussion below yielded:

Updated suggestion

There is a de-facto start sequence as best described by @DanielOaks here: https://github.com/ircv3/ircv3-ideas/issues/3#issuecomment-243434145

I think this sequence should be put into writing as part of IRCv3 (but not necessarily associated with any new CAP), and numerics 002-004 (specified in RFC2812 which currently isn't considered part of IRCv3, while RFC1459 is) as well as numeric 005 (RPL_ISUPPORT, which isn't currently officially standardized at all) should be put down as official standardized part of IRCv3 as well - including a mention that RPL_BOUNCE is officially no longer used and RPL_ISUPPORT replaces it with no exceptions.

More elaborate solutions aren't really required because this is already de-facto standard and pretty much nobody violates it, so all it would do is give a much-needed spec to actually rely on instead of hoping that all server devs will indeed stick to the de-facto in the future (why this is a problem and what happens if they don't is described in the discussion below).

---

Original suggestion:

I'm suggesting a draft for a new capability. I attached the full specification draft as text to the ideas repo since there is currently no implementation of it. First, let me explain what this is about though:

Situation

Currently, valid IRC servers can do one of the following:

• Send 001 only as in basic RFC1459
• Send 001-004 only (no RPL_ISUPPORT), and at least twitch appears to do that in practice
• Send 001-005

Also, RFC1459 specifies that a MOTD may be sent, and offers an error message if there is no MOTD - but it doesn't seem to require that either a MOTD or a MOTD error is sent.

Resulting problem

Therefore, a fully compliant IRC client which wants to support legacy servers will need to do some timed wait for numerics 004 and 005 or assume that there will always be a proper MOTD notification. This seems a bit brittle, also because timed wait can lead to random failures to recognize 004/005 on modern servers due to connection slowdowns and similar.

Proposed solution

Therefore, I'm proposing a new mechanism offered by new servers to indicate that they're not legacy servers, which allows a client to have an expected "end of startup" point to wait for safely. Any modern server announcing this can then be sure that any client who supports this mechanism will parse all its startup numerics even when a shaky connection is involved and the client wants to keep timing heuristics for legacy server support.

draft-testing-cap-099.txt

[ghost]

Edit: I updated what I currently consider the best suggestion in the first post above

A small correction (I'll upload a corrected file later since I somewhat expect that some of you will also find some mistakes which I need to fix):

A server providing this
capability MUST NOT send any PRIVMSG, NOTICE, JOIN, PART, QUIT events

should be

A server providing this
capability MUST NOT send any PRIVMSG, NOTICE, JOIN, PART, MODE, QUIT events

(since correct MODE parsing requires the 004)

.. and the according client part should be [...] explicit JOIN, PRIVMSG, MODE or NOTICE command

I guess an argument could be made for a sort of whitelist approach, but I just wanted to provide a baseline of common sense and not forbid all sorts of extra communication. If someone has a better idea tho, I'm all ears

[ghost]

Edit: I updated what I currently consider the best suggestion in the first post above

The following alternative was just suggested to me which I think isn't that bad as well:

Make a new extension to the standard which specifies the following:

• 001-005 must all be sent after logging in, and they must all be sent in order, and no other sort of event must be sent to the client before that
• RPL_ISUPPORT is properly officially specified (and not just a draft)
• RPL_BOUNCE is declared as "not a thing"
• At the end of the login sequence, either a MOTD numeric or the error numeric for no MOTD file being found must be sent
• Possibly other things that are de-facto but haven't been properly declared which still need to be put into a proper specification

.. and in the end, add a read-only CAP which advertises this declaration as being followed. Like, "ircv3-super-substandard" or "irc3-v.34" or something. Therefore a client can do "CAP LS" right at the start, and then it will know it is on a server that will adhere to all those things properly, without the need of any guessing.

Rationale: it would avoid assigning a single CAP just for this tiny problem or even introducing any numerics, and combine all those things for which a single readonly-CAP would be somewhat overkill into a single package.

[DanielOaks]

I've said this in IRC, but I'll throw it here anyways.

In my opinion, this is not necessary. The current de-facto standard, as far as I know and have seen, is this:

1. Registration is completed.
2. Server sends 001, 002, 003, 004, 005 (multiple times).
3. Server sends whatever else it wants to.
4. Server sends either RPL_ENDOFMOTD or ERR_NOMOTD.

As far as I'm concerned, this is the current default, de-facto standard. I have not encountered a server in real-world use that does not already do this. Everything has support for RPL_ISUPPORT.

Specifying RPL_ISUPPORT is a separate discussion and would happen in a separate document.

If we need to send any other cap-specific numerics before registration is completed, we can specify in that document that those numerics are to be sent before the initial RPL_ENDOFMOTD / ERR_NOMOTD burst has completed.

I think that this use of the current de-facto standard covers the use cases around this and that introducing a whole new cap, numeric, or etc is not useful. Even if it was introduced, clients would fall back to this de-facto standard behaviour since every single server that does not support this brand-new mechanism (i.e. most of them for years) would still support the existing, totally-fine method.

If there are servers in use or networks which do not support the de-facto standard I've described here, please let me know so I can update documentation, but I think that this is the best method for going forward with this. No new behaviour needs to be specified and implemented.

In my eyes, relying on the RFCs to the letter and trying to 'patch this one hole' in this case is unnecessary. The newest RFC describing this part of the protocol is over 15 years old at this point. The current de-facto standard behaviour which (as far as I'm aware) everything does holds more weight than a document that was written so long ago and does not fully describe how the protocol is used today.

[ghost]

Well there is twitch, although I have been told it's generally not really a correct implementation anyway. I'm not sure about others. Anyway, maybe you are right and specifying this is enough and then any client who notices CAP LS to be working can simply assume it is being followed.

However, wouldn't it be a bit silly to require the 001-005 but not having any sort of official specification of 005? I think the discussions are at least slightly related, although there's no need to put it in the same document in the end result.

[DanielOaks]

Twitch is weird and does things like requiring caps for clients to be sent standard join/part/etc notices. I would not consider them in a real, major way in this specific discussion around IRC standards.

They are a useful datapoint, but I don't think they're useful to consider in this case.

[ghost]

Alright. Well I would certainly be happy if the de-facto order as you described it would be properly set in stone for anyone supporting IRCv3, and the 005 / RPL_ISUPPORT properly specified (which I think should really include a remark that RPL_BOUNCE is explicitly deprecated/removed)

I agree that anything beyond that is probably not strictly necessary. Maybe I was reaching a bit too far :smile:

[jevolk]

@JonasT This is a common misconception, honestly, one that I had myself when I was writing a more involved bot which sought to apply the startup data to its operation. The solution is to reason about it (and thus program about it) from a different perspective- don't actually require the startup sequence for its own sake. You can just send whatever you need to accomplish from the point of connection establishment, and apply the data from the startup sequence as you acquire it. If you truly can't send a request because, say your RPL_ISUPPORT map is still empty and you wouldn't make the request if something specific wasn't supported, queue the request or block the program until that piece of data is received. Otherwise, in reality, there is no true "startup sequence;" it's just an illusion.

[ghost]

@jevolk what you are saying doesn't make much sense. Of course there is no formal startup sequence, which is why there needs to be one. And obviously, RPL_ISUPPORT is vital and you must wait for it to operate properly, because you can't even tell what a valid channel name to join is without it, or what is a status symbol in front and what is part of the nick in NAMES, and you can't parse any complex MODE events properly without 004. This is why we are discussing this issue.

If you have a client that can join channels and operate properly without waiting for 004, 005, then it lets either the user manually read most things instead of handling them in a more highlevel fashion, or it doesn't operate properly on more unusual (but following the specs) IRC servers.

You can't make a conforming client for all complex situations without waiting for the 004, 005 data. Which can be a problem if you don't know if the server sends any of 004, 005, since it doesn't need to according to the current specs. So yes, you do require the startup sequence for its own sake, so you know whether something will still be sent or not, and when you're done getting all the things. Hence this issue.

Currently, client devs often wait for MOTD or the "error no MOTD file" message and assume it is done. My last suggestion is simply to formalize this, since the current specs don't require the servers to adhere to this exact sequence, even though right now most (all?) do.

EDIT: with current specs I mean RFC1459 + all IRCv3, since that is what the main site currently says compromises the specs. This does for example not include 002-004, which you really want to get as a client because of MODE parsing info in the 004.

[jevolk]

Standardizing the transmission of certain data after registration is a good idea. What isn't a good thing to standardize is indicating the end of that data, or that the developer should take comfort that it cannot be updated. That is the part which sets a sweeping precedent to take the protocol in a certain direction it does not necessarily have to go at this time.

In fact, the standard should seize the ambiguity here to make several advances in a more flexible direction:

• Ensure there is at least some way to query the server at any time for all of the low-numbered information and expect a response.
• Imply a certain set of said queries occurred when registering (present behavior).
• Establish a server-to-client command set to allow safe updates of live changes pushed by the server. Think: there is never a reason to reconnect, like, indefinitely.
• Establish a robust way for the server to indicate errors associated unambiguously to specific messages, which the client can adapt to a la carte as it attempts to do things in the blind.

[ghost]

@jevolk why is specifying that the MOTD must come after all startup numerics that are considered relevant to parsing for the client, and specifying that either a MOTD or MOTD error message must be sent bad? (and also that all of 001-005 must be sent, which is also already de-facto standard. Nobody suggests limiting the startup sequence to that for the foreseeable future) It would pretty much have absolutely no restrictions to expanding on the initial numerics or anything, just guarantee that you know when they are over, and that you'll also actually get the 004 and 005 on any reasonably modern server.

That is the part which sets a sweeping precedent to take the protocol in a certain direction it does not necessarily have to go at this time.

Well, but it does have to go there and de-facto it already went, since there is already a de-facto end and as I explained in my previous post/comment that is absolutely important for parsing and it should be possible to rely on this (most clients already do without this being guaranteed by the specs right now).

You will potentially break a lot of clients if you send MOTD before the 005, or not at all after the 005 - at least if the comments in the #ircv3 irc channel are of any indication, where this was by far the most suggested solution on how to deal with this problem. So how is it a bad idea to actually write down that things are supposed to stay this way?

Also a good argument was made on IRC not to make it too flexible, because it would encourage server owners to actually derivate from the current de-facto (which is currently not happening, most servers seem to stick to it although they wouldn't need to) and this might break a lot of clients which would definitely be an undesired outcome of this.

[ghost]

Although to be clear here, with a CAP to enable it there's nothing wrong with making some more flexible startup sequence.

However, my intention was to fix the current situation first. (because there is a de-facto standard which clients rely on, which servers could easily not adhere to while still adhering to the specs which is bad) That I did initially suggest to do this with a new 099 numeric was a bit misguided, I do think right now that simply not changing the initial flow at all and simply speccing the current de-facto more tightly and maybe add a read-only CAP to indicate this is the best way.

Beyond that, a new CAP that needs to be actively REQ'd for a more flexible, possibly different protocol start might be neat to have, but it's beyond the scope of what I wanted to propose here.

[jevolk]

@JonasT

Well, but it does have to go there and de-facto it already went, since there is already a de-facto end

Perhaps it's just a personal position but I don't think a standard should codify limitations based on the bad practices of the given day. Let the server developers worry about not crashing their clients; let the client developers benefit from the ambiguity and learn how to write more robust implementations. One or both will happen naturally, the standard doesn't need to be involved. If anything it should offer a progression -- a way out of the issue -- shine the light ahead for everyone to see, rather than regress with some kind of collective punishment for implementations.

why is ... specifying that the MOTD must come after all startup numerics ... bad

The engineering short answer is that it leads to interrupting clients, servers, or connections with cold restarts, rather than smooth running state transitions modular software is capable of. Of course, as you said, it would be de-facto irresponsible for any server to start pushing random RPL_ISUPPORT's right now and expecting clients to play along with updating themselves. Obviously a CAP is inevitable.

That doesn't mean it's justified to standardize The Wrong. On a philosophical level it sends a bad message to developers and encourages less robust, less modifiable implementations that future standards will have to deal with. Right now if a majority of the implementations are flexible and could easily handle (or be lightly modified to handle) random 00x's shoved to them changing to their behavior, when you lay down a standard like this one, that statistic will asymptote down to the big fat donut.

[ghost]

learn how to write more robust implementations.

Well, you can't do that right now! Unless you like undefined behavior.

I somehow got the impression that you didn't fully understand the underlying problem. (which is unrelated to updating RPL_ISUPPORT with later messages) The problem is simply that right now, a server can choose not to send 005, and before the 005 arrives you cannot tell if you are going to get it, so you cannot implement any sort of predictable behavior as a client if you want to wait for it for robust parsing.

I think making a super-flexible super-enginered special CAP start is good, but you could still easily do that after just specifying the de-facto, to ensure clients can indeed reliably wait for the 005 or knowing when to stop waiting (which is when the MOTD arrives, but this is just de-facto behavior and not strictly required which is the underlying problem) There is no modern server which DOESN'T actually send it, so this isn't "punishing" any implementations. Also you can always send an RPL_ISUPPORT with no contents, so this is absolutely trivial to follow for anyone even in the future.

Again, your argument of "learning to make robust implementations" is somewhat irrelevant, since the problem is that with the current protocol specs + legacy servers that is impossible unless you like timers, timed waits and other unpredictable heuristic behavior that can be easily broken by random circumstances like connection delays, which isn't something anybody can possibly want.

---

Also can you be more clear what you think is The Wrong? I think nobody will care if you tell them the current de-facto is wrong, and in fact the current de-facto startup sequence (as specified a few posts above) doesn't even prevent you from making a new additional 006 or sending RPL_ISUPPORT for changes later - so I really don't get your point here about that being limiting for the future.

If you think it just needs to be a bit prettier and even more flexible than that, then I suggest to make a separate proposal which could be easily implemented after this one, since it does in no way prevent further expansions that are explicitly requested with a CAP by a modern client e.g. for live-updating RPL_ISUPPORT which you seem to think would be a good idea.

I want to make the live easier for current, existing clients, and if you think some major changes are in order for future clients then that's nice, but that isn't going to fix the situation for the existing implementations relying on something that isn't actually standardized (while in all other regards it works just fine - it just needs to be written down). As you already noticed yourself this is going to need a new CAP that is explicitly REQ'd by the client anyway while my current newest proposal isn't, so it can be trivially put on top if there is a consensus that it is indeed a new awesome thing to have.

[ghost]

I updated the suggestion in the first post to make it super clear what I think the current best solution is. (mainly based on @DanielOaks 's suggestion) The old one is still present below for historical purposes.

Just to emphasize this, I think advanced suggestions like those of @jevolk are worth considering as well, but I think they should be on top of this basic suggestion on solidifying what everyone is already relying on. (with no robust alternative fallback, since that is technically pretty much impossible without heuristics, timers and other unreliable workarounds - which is why I think it's so important to specify the current de-facto situation properly before advancing to extended new solutions activated with a CAP)

[jevolk]

Also can you be more clear what you think is The Wrong?

The reason clients get away with being dumb at present is because the server sends the post-registration replies first, because you registered first, before doing anything else like joining channels. This allows those replies to "program" the client with how to parse mode prefixes which will be received before channel name lists because IRC benefits from strongly ordered transports.

You're making good points that the server can decide to send you 004/005 like a year or two after registering and nothing says that's wrong. It would be great if something said that was wrong, so I support what you're trying to do here. The proper way for a robust client to deal with that is to set default, reasonably expected values. PREFIX=(ov)@+ is a reasonable default value. There's an overwhelming probability the default will be written over by an 005 before it is ever applied to a name list.

If you connect to a server which has PREFIX=(pq)^~ but never sends you an 005 after registration telling you of that: it is absolutely, unquestionably being stupid on purpose. Disconnect. Standards won't help here. The hand of god won't reach down from the sky and make them follow it.

Going back to my original point again, if you view the protocol through the prism of two independent (duplex) strongly ordered streams, you can send NICK x\r\nUSER x x x x :x\r\nJOIN #mychannel\r\n all in a single frame immediately on connect. You don't need 004/005 to do that. You don't need to know if '#' is a valid CHANTYPE because you know about #mychannel in the first place. You don't need to wait for the MOTD to finish to do that, or a lot of other things either. The response to USER will come before the response to everything following it, setting your client up with the proper casemappings and whatever else.

Hence, there is no gap between 'startup' and 'the rest of the session.' It's all one piece. Hit the ground running and do what you need to do.

[ghost]

To do what you proposed, you need to:

• Assume a channel name is valid which you can't without the 004 / 005, or alternatively do some nonsense action like ping yourself which should be guaranteed safe. (just because the user entered the name doesn't mean it's actually valid)
• Assume there is strong ordering for 001-005 and followup events, which you are right is somewhat nonsensical not to assume but still it's not strictly in the specs that there can't be e.g. no 004-005 at first and then those are sent a bit later after other events (which should be forbidden by the specs but isn't)
• Forget about the fact that 002-005 aren't even part of IRCv3, because they are in RFC2812 and a draft both of which weren't actually formally included into IRCv3

Almost anything anyone ever suggested has weird guesstimates and in-depth implementation knowledge somewhere, and this is just silly. There should be a documented startup way that guaranteed works that is in the specs which allows parsing 004-005 on all modern servers safely. (My recommendation is still the one now in the original post suggested by @DanielOaks which guarantees MOTD / end of MOTD at the end of 001-005 and possibly other numerics, and guarantees no events before all of that went down)

As for the argument that RFC1459 is so old and therefore it obviously can't be 100% up to modern implementations, even more reason for adding a small update informational document to IRCv3 which cleans up some of the outdated stuff.

---

Or in a more practical engineering perspective: right now it's rather easy to write a client that works and parses 004-005 by just looking at the protocol flowing by and guessing what to do, but when you then look at the specs and see if you just got lucky or actually implemented it correctly in a way that will work everywhere, it's going to be really hard to find out. You might even discover that it is most likely not guaranteed to work everywhere, and then it is probably really complicated to figure out what is actually guaranteed to work on all compliant servers. It would be neat if that was different.

[ghost]

So is there any interest in me writing up a draft text that specifies this? (login as it is already done in practice in a standardized form that servers are supposed to adhere to for the future, without any special CAPs or anything)

I'm getting a bit of mixed feedback here, and I don't necessarily feel the most qualified to start a good text on this. However, if you want me to I'll take a shot at it.

EDIT: Also what about formally declaring RPL_ISUPPORT more than a draft? Since that is absolutely needed for a modern IRCv3 startup sequence specification.

[progval]

@JonasT There is some work by @DanielOaks here: http://modern.ircdocs.horse/#rplisupport-005

[ghost]

@DanielOaks that looks very promising! If I may suggest a few additions for clarifications:

1. Clarification of exact order of server messages:

"Upon successful completion of the registration process, the server MUST send the RPL_WELCOME, RPL_YOURHOST (002) [..]" -> "Upon successful completion of the registration process, the server MUST send in this exact order the RPL_WELCOME (001), then the RPL_YOURHOST (002) [..]"

2. Clarification of nothing else may be sent before login sequence is completed:

"The server MUST then respond as though the client sent it the MOTD command, i.e. it must send either the successful Message of the Day numerics or the ERR_NOMOTD numeric." -> "The server MUST then respond as though the client sent it the MOTD command, i.e. it must send either the successful Message of the Day numerics or the ERR_NOMOTD numeric. The server MUST NOT send any JOIN, PRIVMSG, MODE or any other not login-related event before this entire login sequence is complete and everything up to the MOTD or ERR_NOMOTD has been sent."

3. Maybe a note should be added for future numerics (since this discussion also was about some future flexibility for new implementation additions and protocol updates), mainly as a heads up to client devs to not make things explode:

"... and at least one RPL_ISUPPORT (005) numeric to the client. The server SHOULD then respond [...]" -> "... and at least one RPL_ISUPPORT (005) numeric to the client. Now, any additional numerics for additional login and server information MAY be sent if supported by this specific server only, although it is not recommended to server authors to send any such additional non-standard numerics unless they were enabled by a related CAP request of the client. The server SHOULD then respond [...]"

Are there any plans to make this new specification part of IRCv3? (preferably in a way that will be eventually more than just a draft)

[DanielOaks]

@JonasT Thanks, I've integrated some of your feedback and put the rest into an issue so I can look into it more! Definitely some good ideas, I need to look into implementations and other comments/thoughts around this beforehand though.

wrt integrating it into IRCv3, maybe at some point but I don't plan on it anytime soon. I go over my thoughts on this a bit more in the sections here: http://modern.ircdocs.horse/about.html

[ghost]

I am very happy with how that modern IRC docs rewrite looks like. @DanielOaks thanks a lot for putting that much work into it!

I think the best course would be to at least eventually consider making an RFC out of this, or at the very least put a big fat note on the IRCv3 home page recommending this as recommended main source for new readers.

E.g. something like:

"Right now the IRCv3 specification is distributed as a series of extension specifications to IRC protocol version 2.7, also known as RFC1459. To understand the basis of the IRC version 3 protocol, please read RFC1459 followed by the extension specifications." -> "Right now the IRCv3 specification is distributed as a series of extension specifications to IRC protocol version 2.7, also known as RFC1459. However, RFC1459 is somewhat superseded by various practical changes implemented by modern IRC software. Therefore, we recommend reading http://modern.ircdocs.horse as an up-to-date introduction (warning: still a draft) and RFC1459 in addition for knowledge of the original historical specifications, followed by the extension specifications on this page."

I know it's probably too early to do this already right now, I'm just throwing it out there for discussion.

[DanielOaks]

Thanks, much appreciated. I'd consider RFC/etc a possibility way later down the line possibly.

With regards to throwing it up on IRCv3, eh – I'll leave that for others to decide. The diff. in processes and all, i.e. that IRCv3 stuff is signed-off by multiple parties where that document is explicitly edited by me ([ref]), also needs to be taken into account there.

[ghost]

@DanielOaks sure, but maybe one early version could be signed off by multiple parties and linked as a draft at least. But I guess then we could also just wait to do a proper RFC...

It's just a bit unfortunate that such a gem of modernized docs exists and IRCv3 does currently not link it at all, but instead points people only to the outdated RFC1459.

[DanielOaks]

Threw in a PR that rewrites that whole intro to that page, since it's pretty crufty. We'll see what happens and what people think.