API: How to specify idempotent data?

[csperkins]

How is idempotent data, to be send in the zero-RTT connection establishment, specified?

The Use 0-RTT session establishment with an idempotent Message property "specifies whether an application would like to supply a Message to the transport protocol before Connection establishment" which suggests the idempotent message is set on the Preconnection and used later when the Connection is established, but it might be more natural to specify the idempotent data as a parameter to Initiate()/Rendezvous()?

[csperkins]

This also relates to Section 7, which says:

If Send is called on a Connection which has not yet been established, an
Initiate action will be implicitly performed simultaneously with the Send.
Used together with the Idempotent property (see {{send-idempotent}}), this can
be used to send data during establishment for 0-RTT session resumption on
Protocol Stacks that support it.

This seems incompatible with the Preconnection/Connection split, since the only way to get a Connection on which to call send() is by calling Initiate(), Listen(), or Rendezvous(), which starts the process of establishing the connection.

(Edit: and section 7.1.1.4)

[britram]

paging @tfpauly to discuss the pros and cons of the Apple and Linux approaches to TFO...

[tfpauly]

There are three top-level approaches that we’ve used/tried:

1. Mark fast open in parameters (Preconnection) such that connect() (Initiate) becomes a fake action, and you then send after that, and the first send kicks off the real Initiate.
2. Add your initial fast-open data to the parameters (Preconnection), such that it’s already known by the time Initiate is called.
3. Allow marking sends as idempotent, and allow enqueuing sends after create and before start.

(1) gets tricky if you want to monitor when connections actually complete, because they’re marked as Connected/Ready too early. This is a bad application contract, especially when there’s racing involved.

(2) is not great since it requires an entirely new way of specifying data that isn’t Send().

Neither (1) nor (2) clearly support sending multiple chunks of idempotent data into the stack as fast open data.

Thus, we chose (3).

However, Preconnection has certainly messed this up a bit. My suggestion is to no longer take Preconnection as an argument to Initiate(), but instead use a NewConnection(Preconnection) -> Initiate() pattern, which you can insert Sends in between. There are many other reasons to do this as well—if I want to set my callbacks for events for this connection, etc, in languages like C that’s a lot easier if I can call create() and then set my event handlers, and then start once those are set.

Note: I already removed the text in the architecture that referred to Preconnection being passed to calls like Initiate and Listen, since that was API, not architecture, and since I didn’t quite agree :)

[csperkins]

So you create a Connection from a Preconnection, then call Initiate() on the Connection? That would work, provided we're clear when the parameters of that connection lock in (when it's created or when it's initiated)

[tfpauly]

Yes, the parameters from the Preconnection are immutable once the Connection is created.

[csperkins]

Okay, makes sense. I'll put a pull request together to address this (although it'll most likely be tomorrow now)

[csperkins]

I created a branch to implement the change to create a Connection from a Preconnection, then call Initiate() on the Connection, rather than Preconnection.Initiate() -> Connection. However, I'm increasingly unconvinced this is the right approach. The Preconnection.Initiate() approach:

• Gives a much clearer split between the phases
• Makes it clearer what parameters affect connection establishment and which affect ongoing connections (i.e., we don't have to specify the effect of setting connection properties on a connection that has been created but not established)
• Makes it clearer how Rendezvous() works, in particular around resolving candidates
• Prevents the NewConnection(), send(), Listen() error case, since you can't call Send() on a Preconnection
• And, in strongly typed implementations, can prevent errors due to trying to call Initiate(), Listen(), or Rendezvous() more than once on a Connection, forgetting to call them, or calling Send()/Receive() on an unestablished connection – having taught introductory networking recently, these are all common bugs, and can be made compile time errors if we get the Preconnection/Connection split right.

So, please review the branch, but I don't think we should merge it. Rather, I think the right way to handle idempotent data is to add an optional idempotent Message to Initiate() and Rendezvous(). Event handlers seem like something that should be specified on initiate, etc., too.

[tfpauly]

@csperkins That's not exactly how I imagined the flow being allowed—I don't think we should allow Connection to turn into a Listen or Rendezvous, etc. A listener vends Connections, for example, and isn't a Listen itself.

I was thinking that with the current terminology, Preconnection could Create an outbound connection, which could be initiated later. The interaction with listen taking a preconnection would be the same as it is today.

Perhaps instead we can just change the terms here, and add an equivalent of a "start"/"resume" for the objects. Thus, we would have:

Preconnection.Initiate() -> Connection (not started) Connection.SetEventHandler(...) Connection.Send(Message) Connection.Start()

And for listening,

Preconnection.Listen() -> Listener (not started) Listener.SetNewConnectionHandler(...) Listener.Start()

This is pretty much how our implementation works today.

[tfpauly]

@csperkins I really don't like the model of adding many optional parameters to Initiate to handle the various callbacks or extra data that might need to be sent—this list will continually grow and shrink, and most languages don't cleanly support having a single symbol/function/method support an extensible list of parameters. These items also don't belong on the preconnection, since I should be able to have the same Preconnection across many connections to make them look similar, without requiring the same part of the app to handle their events, and to have the same initial data.

[britram]

@tfpauly hm. from an API expressiveness standpoint, I'm not wild about this.

Metacomment: I think we need to meet in person to converge on this. Can we stay with what we have (which gets a little twitchy with more than one Message on 0RTT), keep @csperkins' branch, and defer to post-London?

[tfpauly]

@britram Sure, I'm fine to defer to converge more. Specifically, what's the issue from the API expressiveness issue? Is it that you'd prefer to have a single call that takes all possible parameters and initial work to schedule?

We actually did start with that for our APIs in Objective-C, but moved to the delayed start model to better support languages like C, and to make it cleaner to be clear about which events you want to handle, and who handles them. It allows someone to create a Connection from a Preconnection with certain setup, and then hand it to another part of the app/framework to in turn use it and schedule the event handlers on it. If you require all of that to be fore-known (both the properties of the connection AND the runtime "who gets this callback" state), it's more limiting, since where the events get delivered is not really a fundamental property. This isn't an issue in languages that support an arbitrary number of event watchers without registering, etc, but that's not true across all languages.

[csperkins]

I updated the branch, to sketch out how this might look. I don't think it's too bad now.

[britram]

i should have said elegance instead of expressiveness; it's late. I really don't like the idea, aesthetically, that every connection has a two-phase startup preconnection.Initiate().Start(), more or less just to support the (uncommon) case where you want to send more than one idempotent message per RTT.

It seems to me we have two overarching design goals (at least, I do):

• The common case ("I don't care, just send my damn packets") should be as easy or easier to deal with in the transport services API as it is in the sockets API.
• Code written to the common case should benefit from advanced features in shiny new protocols (e.g. 0RTT) by default if possible.

0RTT is a special snowflake here, because it actually changes the order of operations at startup, and maybe we can't get away from leaking that complexity into the don't care case. But it'd be neat if we could find a way to make send-on-initiate work via a protocol (selection) property anyway.

[britram]

@csperkins I opened this as #124 so we can comment on the details of the wording over there.

[britram]

discussion continues in #124, propose to defer after -00 and tag this post-London if no objections...

[britram]

Expanding from my last thought on #124, here's what I think my current proposal for this is, designed to let application developers not really care all that much about 0RTT but still get benefits from it when available:

• Setting a protocol selection property that prefers 0RTT-capable transports changes the semantics of Initiate() and Send(): an Initiate() will be delayed until either:
  • an initial Send() call, in order to allow idempotent send on startup; or
  • a (fairly aggressively short) timeout, selected such that if the Send() doesn't happen until after the timeout, then the sender is so overloaded or otherwise slow that it wouldn't realize any material speedup from cutting an RTT out of establishment.
• This allows 0RTT for one message. An application that wants to bundle multiple Messages into a single 0RTT flight can set a new third state on the Immediate send parameter (or a new send parameter that is mutually exclusive with Immediate, doesn't matter). When !Immediate is set, a Message will be held until the following Message on send; during 0RTT this will mean (explicitly) that the Initiate should also be delayed until the end of the !Immediate run.
• All this behavior can be overridden with an InitiateNow() call, or with an Immediate parameter on Initiate() (which would default to false).

I can write this up in more detail in an alternate PR (after submitting -00) if y'all don't think it's insane.

[csperkins]

That could work for Initiate(), although I'm not sure I like the invisible change in semantics, but I don't think it works for Rendezvous(). The approach in #124 has the advantage of having very similar APIs for zero-RTT data for both Initiate() and Rendezvous().

[britram]

Can you walk me through why it doesn't work for Rendezvous()? (Are you somehow callable at the moment?)

[csperkins]

I don't think it works for Rendezvous() because there's no Connection to call Send() on until the rendezvous has completed, and by then it's too late for zero-RTT.

(At home due to a snow day, but Skype me if a call is useful)

[britram]

(for posterity, I'm going to go off and think about how my suggestion above might work in a non ugly way with Rendezvous, and if I get time before London, write that up in another PR targeted to land in this branch)

[britram]

For in-person discussion in Montreal; we should turn the suggestions in this issue and in #124 into a message to the list.

[britram]

@csperkins and @britram go into the thunderdome first week of June.

[britram]

this was closed by #214, bikeshedding is now in #224