ietf-tapswg / api-drafts

Architecture, interface, and implementation drafts for the definition of an abstract API for IETF TAPS
Other
23 stars 15 forks source link

Message Context Parameters: Ordered and Final #220

Closed mirjak closed 5 years ago

mirjak commented 6 years ago

I would like to discuss the message context parameters "ordered" and "final" a bit more. I've put this into the same issue because I think there might be some relationship here.

First I find "ordered" confusing because if you send a first message that is ordered=false and then the next one that is order=true that actually means that both messages together have a defined order, so setting it to false on the first one seems confusing. In post socket we called this dependency (and probably left the exact implementation a bit more open). I think that is actually more want we want to say: This message has to be delivered before/after the other message, however, this does not imply that both messages have to be provided to the transport right after each other. Thinking further about this, and maybe similar as we have connection groups, we may want to have message groups which basically only say that all messages belonging to the same group should be delivered in the order they have been given to the transport layer. Just an idea for now; did think this completely thought yet but maybe others have thoughts...?

However, at this point I got reminded of partial messages, which I think always imply in-order delivery...? But to be honest I have no idea why we need partial messages at the sender-side. I think what we want is the grouping as described above (to make sure we put on those messages on the same stream in the right order) and some indication at the receiver-side that the transport has no idea about the message boundaries and just delivers a random piece of data that it received. However, these seem to be two independent concepts which should be reflected by different parameters.

tfpauly commented 6 years ago

I agree that "Ordered" has a very confusing contract as its written here. I think that all of the functionality that is beneficial from "Ordered" can also be expressed via the post-sockets previous concept of marking an "Antecedent" to describe an explicit dependency relationship between two messages. This has the advantage of being very clear about which messages relate to one another in a situation in which reordering is possible.

The handling of partial messages do assume that all data for a single message is indeed sent and received in-order. I think this is a reasonable constraint to put on the API. If you really want to be able to receive chunks of a message out of order, then perhaps you should be treating those chunks as the "messages", right?

gorryfair commented 6 years ago

On 05/09/2018, 18:25, mirjak wrote:

I would like to discuss the message context parameters "ordered" and "final" a bit more. I've put this into the same issue because I think there might be some relationship here.

First I find "ordered" confusing because if you send a first message that is ordered=false and then the next one that is order=true that actually means that both messages together have a defined order, so setting it to false on the first one seems confusing.

I don't see that as confusing at all, the second is ordered with respect to the first, that reads perfect to me.

In post socket we called this dependency (and probably left the exact implementation a bit more open). I think that is actually more want we want to say: This message has to be delivered before/after the other message,however, this does not imply that both messages have to be provided to the transport right after each other. Thinking further about this, and maybe similar as we have connection groups, we may want to have message groups which basically only say that all messages belonging to the same group should be delivered in the order they have been given to the transport layer. Just an idea for now; did think this completely thought yet but maybe others have thoughts...?

I'm less sure here - I realise postsockets chose this, and NEAT chose not. I am not sure having just ordered and datagram suffers hugely. I can see how one can construct such a dependency beteween messages, but the actual benefit of further reducing head-of-line blocking can often be small except in corner cases, and I suspect this actually does not add a huge value.

However, at this point I got reminded of partial messages, which I think always imply in-order delivery...? But to be honest I have no idea why we need partial messages at the sender-side.

Is this what you mean by sender side: The sender starts transmission of a (longish) message. The interface blocks transmission for whatever reason after it has started sending, the sender gives up transmission. The last part is never sent. It's a form of late decision.

I think what we want is the grouping as described above (to make sure we put on those messages on the same stream in the right order) and some indication at the receiver-side that the transport has no idea about the message boundaries and just delivers a random piece of data that it received. However, these seem to be two independent concepts which should be reflected by different parameters.

Unsure.

Gorry

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/taps-api/drafts/issues/220, or mute the thread https://github.com/notifications/unsubscribe-auth/AHC1ktgQX3CteT37fTJxZOSXSuqWGuZRks5uYAkSgaJpZM4WbWv8.

mwelzl commented 6 years ago

I also didn't find the current description confusing, and I'm afraid that going into defining dependencies and message groups is just making it all unnecessarily complex. I think Mirja's issue is mainly captured by this statement from her:

This message has to be delivered before/after the other message, however, this does not imply that both messages have to be provided to the transport right after each other.

I think that statement is wrong because our text says "...which was passed to the same Connection..." - and so, within the same Connection, "ordered" really does mean "right after each other". Also, obviously, this has to be ignored on the first message. I suggest to simply include the following sentence in the text: "The Ordered property is ignored on the first Message that is sent on a Connection." ...or what am I missing?

In line with what Gorry wrote, I can understand that defining dependencies can be more "powerful", but I also have doubts about the added value. Concretely, what this can give the transport system is the ability to better understand which messages can be dropped from the buffer when one of them (upon which the others depend) is dropped. However, alternatively, applications needing this can also just try to keep the buffer small and maintain control over their data as long as possible, handling such dependencies themselves. Moving this functionality into the transport system then has the advantage of making it general, applying across applications, but it comes at the cost of 1) making the API more complex, and 2) perhaps discouraging application programmers from keeping the buffer small and handling things inside their apps (where such dependencies can take various application-specific forms, dynamically changing depending on various conditions, etc.). In conclusion, I don't think it's worth it.

About partial messages, +1 to Tommy's response.

mirjak commented 6 years ago

Micheal, my point with that sentence was that, yes currently you have to send them right after each other, however that's an unnecessary constraint of the current API definition because the feature we want does not requirement that.

Not sure if Gorry's maybe is that we don't need that feature at all...?

gorryfair commented 6 years ago

On 06/09/2018, 10:29, mirjak wrote:

Micheal, my point with that sentence was that, yes currently you have to send them right after each other, however that's an unnecessary constraint of the current API definition because the feature we want does not requirement that.

Not sure if Gorry's maybe is that we don't need that feature at all...?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/taps-api/drafts/issues/220#issuecomment-419028244, or mute the thread https://github.com/notifications/unsubscribe-auth/AHC1kv7AxWu0gISuWgEvY1Inv6267Zu8ks5uYOrvgaJpZM4WbWv8.

I haven't seen a case where we need this in the API that knows its internal dependencies. I used to hold passionately to the idea of sophisticated buffer control, but then I discovered multiple ways apps can use this... and realised if the API could allow the app to keep a short buffer at the sender and multiple streams, the app can mostly do the correct thing. Offloading can also make this very hard to achieve what you promise with deep buffers, unless the offload does what you expect :-0.

I say mostly, because I could see a case where an App may need to abort a long message that was not sent within a deadline - e.g. due to alink down or QoS-related issue, and then it may be better to give-up a particular message.

Gorry

mwelzl commented 6 years ago

@mirjak I'm lost... maybe the disconnect is about sending vs. receiving? I think that clearly, we want to be able to say that messages (on a Connection) must be delivered to the receiver directly in order? Or maybe you're thinking of streams - then it depends on how Connections are mapped onto streams. For now, I think we work with the assumption that 1 Connection can be 1 stream and you won't have multiple streams "inside" 1 Connection (no other form of multi-streaming is currently exposed in the draft).

mwelzl commented 6 years ago

@Gorry - when you say:

I haven't seen a case where we need this in the API that knows its internal dependencies.

... what do you mean with "this" ? Ordering altogether (as we have it now), or being able to define a string of dependencies?

gorryfair commented 6 years ago

On 06/09/2018, 10:42, mwelzl wrote:

@gorry https://github.com/gorry - when you say:

I haven't seen a case where we need this in the API that knows its
internal dependencies.

... what do you mean with "this" ? Ordering altogether (as we have it now), or being able to define a string of dependencies?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/taps-api/drafts/issues/220#issuecomment-419032209, or mute the thread https://github.com/notifications/unsubscribe-auth/AHC1kloQCnFQP1CEYISLlH9dbZchXjmQks5uYO4YgaJpZM4WbWv8.

Sorry, could have been clearer:

I see two important and useful styles:

I know apps - such as sophisticated web browsers (and RTP media servers) build a string of dependencies and use this to control what is emitted on the wire, however these apps often want tight control to do very specific things to control what they send when, I don't see this as easy or useful to generalise.

Someone building their own sophisticated app will have their own incentives. A short buffer seems sufficient to me (with ordered/unordered) delivery. Ability to abort a partially sent message seems a reasonable optimisation to help avoid sender-side blocking.

Gorry

mirjak commented 6 years ago

Gorry, I think what you are talking about is a connection property (not a message property). That's a different thing we should not mix up (and as such all not name the same way).

Also aborting a message is done based on the message lifetime and therefore also a different feature.

gorryfair commented 6 years ago

On 06/09/2018, 11:55, mirjak wrote:

Gorry, I think what you are talking about is a connection property (not a message property). That's a different thing we should not mix up (and as such all not name the same way).

Indeed, this is sufficient.

I don't think we need to group messages within this level of the API.

Also aborting a message is done based on the message lifetime and therefore also a different feature.

Yes, that's fine as a message property.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/taps-api/drafts/issues/220#issuecomment-419051900, or mute the thread https://github.com/notifications/unsubscribe-auth/AHC1kvMevlSROMycJkR6LWoEPEcakcidks5uYP8igaJpZM4WbWv8.

Gorry

mwelzl commented 6 years ago

This is getting interesting. SCTP has per-message ordering as a flag, but I must admit that I always asked myself why this is needed. I also think that a per-connection property should be enough.

mirjak commented 6 years ago

I think the use case is that you have a multi streaming protocol but don't expose the multiple stream to the application (but still want to make use of them in a sensible way). Not sure if we want to support this use case though or at least if there is maybe a better way to support it.

mwelzl commented 6 years ago

That's not the use case in SCTP because in SCTP it applies per stream only, and streams are exposed. So, I still don't know why SCTP has that... then again, I'm not sure if there's much benefit to taking this (per-message) functionality away? At the end of the day, it's a simple thing, the receiver either looks at sequence numbers before handing over the data or not.

abrunstrom commented 6 years ago

I agree that the text is unclear, but the concept of ordered seems clear to me. An ordered message is only ordered with respect to other ordered messages within the same connection. I suggest to clarify the text rather than introducing new more complicated concepts like message groups.

Whether the "ordered" property is needed or not can of course be discussed, but from an abstract view I guess it makes sense. It allows to send "out-of-band" data on a connection. How useful it is I am not sure, but as it is supported by at least SCTP I would leave it in for now.

csperkins commented 6 years ago

Dependencies seem more useful than ordered, since they also affect partial reliability. I do see use cases for this, and wouldn't want to drop the functionality.

abrunstrom commented 6 years ago

I think partial reliability is different, we have niceness and lifetime for that.

britram commented 6 years ago

So dependencies (through antecedents) were a key feature of Post (like much of Post, lifted shamelessly out of Minion and its, erm, antecedents), but AFAIR there were a few reasons we left them out of the TAPS API:

I think dependency management is a useful feature, but it might also be a good future feature. Yay registries! (?)

britram commented 6 years ago

all that said, I'd be okay with either per-message or per-connection ordering. If we do per-message ordering, I think that the (admittedly awkward) definition we have now is probably the most internally consistent...

csperkins commented 6 years ago

@abrunstrom but you need to know the dependencies to know what you can safely discard, and what you must keep even if it's exceeded its lifetime

csperkins commented 6 years ago

@britram agree with the reasons why dependencies are hard, but I do think they're important. If we leave them out of v1 we need a way to cleanly add them back later, but I'd prefer not to leave them out.

mirjak commented 6 years ago

@abrunstrom There should be nothing you should keep if the lifetime is exceeded. If that would be the case, the lifetime would have been set wrongly. I guess in many cases if you have should dependenciess, your lifetime would be infinite anyway.

@britram You don't have to refer future messages. The future message can just refer the already sent one.

I don't think the transmission scheduled is such an issue because if you only have one stream, you just send everything in the order you get it and setting a dependency has no effect. But if you have multiple streams you can send one message per stream expect there is a dependency then you want to use the same stream. However, maybe it is easier to either expose stream to the upper layer in this case, or only do this for connection where the connection property is unordered.

csperkins commented 6 years ago

@mirjak I'm not sure it's that simple – you might want to send data if its lifetime has been exceeded (its deadline has passed) because it's needed to allow future data, that hasn't yet missed its deadline, to be used. Dependencies allow you to express this.

mwelzl commented 6 years ago

The disadvantages mentioned by @britram all come on top of what I perceive to be the biggest disadvantage, already explained above. Again, rephrased shorter: this can only operate on what is in the send buffer of the transport system, yet we should probably try to keep this buffer short.

A way to keep this buffer short is to keep the data + operations on it inside the application as long as possible.

@csperkins - it seems to me that, if the lifetime has exceeded yet an application still wants to send it, the most straightforward way to handle this would be for the application to resend it into the transport system.

Moving all this logic into the transport system (potentially complex dependencies, allowing exceeded Messages to still be sent, ..) seems like pretty clumsy design to me, when we could just leave this all up to the application. What's the real benefit of moving it down below the API?

csperkins commented 6 years ago

@mwelzl of course the application can do any of this – but why would should we push that complexity on all the application developers, forcing them to keep reinventing the wheel, when we can build it as a reusable library? Isn't the point of TAPS to make the API more expressive, so the stack can help the application writers?

Short send buffers: for some applications, certainly we want to keep buffering low. For others, the data is available ahead of time, so why not pass it to the stack, and let it do something smart with how the data is transported?

mwelzl commented 6 years ago

@csperkins - I wanted us to consider the full set of trade-offs in this design decision, that's all. Having general functionality for all apps is a pro. What I said, and what Brian added to it, are the con's.

My vote is for simplicity... but sure, I see your point.

abrunstrom commented 6 years ago

I also think we should keep things simple.

@csperkins I am not fully up-to-date with post, so I am not sure what functionality want to provide here that is supported by the standard transport protocols and that can not be expressed with the basic primitives? In our current model, connections map to streams for protocols that support multi-streaming so can you not get the functionality you are after using a connection group or an unordered connection? For partial reliability, at least SCTP will not send a message after the lifetime has expired. What protocol supports this?

csperkins commented 6 years ago

I don't think a dependency (antecedent-based) API is complex, although an effective implementation of it might be. If we're going to support partial reliability and deadlines, then I think we really do need the ability to specify dependencies – or at least to cleanly add this to the API in a backwards compatible manner.

@abrunstrom our TCP Hollywood prototype supports this – yes, it's a research prototype rather than a widely deployed protocol, but surely the goal of this API is to be general enough to support upcoming protocols, not just what's deployed today?

mwelzl commented 6 years ago

I don't think there's any word of future protocols in the TAPS charter, and that's for a good reason: TAPS needs to stay easy enough to implement (and, of course, also use) to really have an impact. At least that should be the case for the mandatory-to-implement part.

We have an appendix with "Experimental Transport Properties", already containing properties that protocols don't support today, but future protocols might. This is where this could fit, IMO.

csperkins commented 6 years ago

Sorry, but I disagree on this – this is a fundamental property of partially reliable protocols, and it needs to be supported (or, at minimum, possible to add in a backwards compatible manner).

mwelzl commented 6 years ago

I'm still sceptical. At the end of the day, it's just about dropping things from a buffer; there can be 100 different ways to decide to drop stuff from a buffer - you say that this a fundamental property of partially reliable protocols, and I could answer that the fundamental properties of partially reliable protocols should include everything in RFC 7496 (time-based, priority-based, and # retransmission-based dropping). We can do all of that but I don't think that's helpful.

I don't argue the usefulness of the functionality that you propose (to me, it seems a lot more useful than priority-based or # retransmission based dropping), but I'm seriously worried about the API growing too large and the implementation growing too heavy. I think that what you want could be implemented quite nicely on top of a layer like we already have: when a PR transport drops a message, it should tell the application about it, and then the application can drop the messages that depend upon it from its own memory that it still has control over. This way, the application sees less buffering and has more control.

You argued for generalising functionality and not having to re-invent the wheel inside applications. About that, why not offer it in a shim that takes more data in its buffer than the TAPS API below it, and drops things from it based on the information it gets from the TAPS API (about packets dropped by the PR transport)?

csperkins commented 6 years ago

Designing a new, supposedly general-purpose, API, then immediately telling users of that API they have to implement a shim to fix its limitations is not exactly a vote of confidence in TAPS...

We have extensive experience with this feature as part of RTP, and the API additions are small, well-understood, and have been previously specified in Post Sockets. I'm hardly arguing for some unusual, complex, experimental API change.

britram commented 6 years ago

this seems like a good discussion to take to the interim if we have remaining time...

Here's where my thoughts are on this now (having just caught up on the thread):

I agree with @csperkins's basic argument that if we want TAPS to be the Transport Interface Of The Future™ that dependency tracking would be very good to have, if only to get application developers used to thinking in these terms. So IMO (now), if we can come up with a way to specify this and convince ourselves that it is easily correctly implementable, we should try to get it in.

mwelzl commented 6 years ago

Just an add-on to one of Brian's points here: "an implementation of a dependency API that actually gets a massive win requires a new transport and/or changes to the architecture. "

The biggest problem that I see with this is missing here: it also requires a large buffer.

So all in all, I'm not convinced, but that's just my view. I personally give a very high weight to the "KISS" side of things here. Maybe too high, what do I know. At the end of the day, where I place myself in this functionality-vs-complexity trade-off is just my gut feeling.

mwelzl commented 6 years ago

BTW, +1 to discussing this at the interim !

britram commented 6 years ago

it also requires a large buffer

Yes, it does (I see this as a detail hiding behind "somewhat tricky to get not-wrong", but that's just me).

My two-really-three questions on the buffer issue (and these are actual I-don't-know-the-answer questions):

  1. In the contexts in which people would actually implement dependency-aware transport (generally biggish media servers), does the size of the buffer matter?

  2. (Isn't the size of the buffer constant for this sort of thing anyway, and we're just arguing which side of the API boundary it has to live on?)

  3. Is a dumb implementation of a dependency API (i.e. one that providing dependency information as an application does not give you much win, but which does not otherwise penalize you for trying) that doesn't need a large buffer possible?

mwelzl commented 6 years ago

I believe I can answer these questions ... so here's a try:

  1. Yes absolutely: to stream non-live-content, the goal is typically to operate lossless and adjust the quality if needed to avoid buffer underruns. E.g., streaming over TCP works quite okay, as we know from various DASH applications and also e.g.: https://ieeexplore.ieee.org/document/5497172/?tp=&arnumber=5497172&filter%3DissueId+EQ+%225602023%22

... so the types of ideas where one throws away data because of dependencies are typically a matter of live content. I find it hard to imagine this being useful for anything else.

  1. I guess it can be constant, yes, and yes we're arguing which side of the API boundary it has to live on.

  2. Certainly possible, but it doesn't have much to operate on.

csperkins commented 5 years ago

It's interesting: I generally cite that paper as an explaining why streaming over TCP does not work especially well...

Yes, we're arguing which side of the API boundary this feature lies. Whether we continue to implement this in the applications, or whether we accept the long implementation experience as evidence that it's a useful feature, and provide dependency and timing information so newer transport protocols can be designed to help applications.

A dumb implementation: isn't that just to ignore the dependency information? It offers no benefit, but also has no real cost (the applications that could use this know the dependencies anyway, so the only cost is the additional, ignored, parameters passed to the send call).

britram commented 5 years ago

A dumb implementation: isn't that just to ignore the dependency information?

well, maybe. It depends on the details of the contract on the dependency information. If the contract is "X depends on Y means X will not be sent until Y is acked", then the TAPS sender (behing the API) has to track it. If the contract is "X depends on Y means X will not be sent until Y is sent", then the TAPS sender can trivially compile deps down to an ordered message buffer list. If the contract is "X depends on Y means that the transport will take that under advisement and try to do the right thing", then the TAPS sender can completely ignore.

csperkins commented 5 years ago

@britram we'd need to specify an appropriate API contract, to be sure, but none of these, or the other variants I might imagine, seem difficult to implement. I'd tend to opt for a contract of the "hint that the transport might find useful to optimise things" form, but the details of the contract are for a different discussion.

gorryfair commented 5 years ago

See how much you hate or love this:

mwelzl commented 5 years ago

@csperkins - side note, about the paper: I guess it's possible to use their model to argue in this or that direction; sure, there are limits, and they are quantified in this paper. The authors say that "Our research reveals that real-time application performance over TCP may not be as delay-unfriendly as is commonly believed", which is why is tend to use it like I do. But, it depends on what one considers "delay-unfriendly", of course.

Anyway, back to the point. I'd strongly argue for a dumb implementation not doing anything, on the basis of the API contract just being a hint from the application with no promises made, exactly as suggested by @csperkins. Anything more gets in the way of fall-backs which defeats the purpose of TAPS (because it requires fall-back code inside the application). For anyone who might not like that line of reasoning, let me try this: stricter API contracts have a rotten QoS smell from the 90's.

About Gorry's message, I have been arguing that this shouldn't be mandatory-to-implement functionality before... so I obviously agree. Can't we just say that this, like other mechanisms that don't speak to what protocols are offering today (i.e. beyond minset - e.g., the "traffic category" or "size to be sent or received" intents), belongs in an appendix?

csperkins commented 5 years ago

@gorryfair I tend to think this is pretty fundamental for real-time applications, so I think we do need to either include it in the first version, or provide a backwards compatible way of adding it. I think it's one of the simpler parts of the API to implement, so I'm not understanding the push-back.

@mwelzl "like other mechanisms that don't speak to what protocols are offering today" – people have been implementing this in RTP for many years! It's not a novel feature. Yes, it's beyond the minset, but restricting TAPS to the minset makes it somewhat backward looking and uninteresting, surely?

mwelzl commented 5 years ago

About "what protocols are offering today", I meant transports - to draw a line between 1) functionality that's almost only a hand-over to the protocols below (with happy eyeballing and all that), which we definitely need; and 2) functionality that adds something on top (which can be useful, but could just as well include 15 more RTP things, FWIW).

I'm not at all saying 2) should be out, just that, in general, things in category 2 should be optional to implement. What I'm going to say below is: I think I AM in fact convinced now, but for other reasons - the "RTP has done this for many years" argument is not convincing me at all.

As I said earlier, that this should not be mandatory is just based on a "gut feeling" in the trade-off between added benefit and added complexity. I tend to lean heavy on the "no added complexity" side because I'm keen on seeing something that would really be picked up by implementers. If we add so many bells and whistles that the sheer length of the document scares people away, we haven't achieved anything. However, several points have made my gut feel better about this: 1) your statement that it's an easy API update; 2) the agreement (which I hope we have?) that this is just a hint from the application, nothing binding, i.e. a simple implementation could just ignore it; 3) Tommy's elaborations at the virtual interim on how they could implement things more efficiently because this buffer is in the kernel in their case.

The latter point is particularly strong. If I think of the NEAT library in user space, I'd wonder, why does NEAT have to do that - couldn't RTP-over-NEAT do it? But the thing with buffers is, if you have poorly managed buffers below whatever you invent, you're still lost. It seems to me that we've had quite a long history of poor buffer management with the sockets API - consider how late TCP_NOTSENT_LOWAT came about, and how big a deal this very TCP-specific hack can be.

So, being able to control the buffer better really does strike me as key functionality and something that could convince folks that this API makes a difference ("better buffer management than with the sockets API" is a convincing statement by itself, I think). So yes, I guess I'm convinced by now.

In more general terms, maybe we could say that buffer management mechanisms should be excluded from my category 2), and they can be mandatory (of course only if the API change is minor and the implementation can be easy).

gorryfair commented 5 years ago

I agree: "people have been implementing this in RTP for many years! It's not a novel feature", I said this on the interim call, I still see that on this, but not quite the same. I think we've reached something that I I would appreciate a little time for someone explain face-to-face, because I may have missed something.

csperkins commented 5 years ago

@mwelzl "I meant transports" – RTP is is a transport protocol! Yes, it has some non-transport features, but I'd strongly argue that deciding what packets to send is very much transport-related.

mwelzl commented 5 years ago

@csperkins - yes I'm aware you're from the RTP-is-a-transport planet :-) :-) but RTP does so much more than just deciding what packets to send...

mwelzl commented 5 years ago

Why, oh why, did I have to do this. Sometimes I think I'll never learn. This is not the time and place for this discussion. We're discussing about what I think is a quite fuzzy boundary, and we will not be able to quickly agree on anything. Yet, this discussion has already converged - if you want your API change in, write text. I'll accept that RTP is a transport or whatever.

britram commented 5 years ago

hey @mwelzl are you ok? closing.