Send request body when requesting Block2 messages

[chrysn]

Currently, when a request with a payload that is not block1wise'd has a response that is block2wise'd, the request payload is not repeated. This is efficient in terms of traffic, but means that the common stateless optimization for GET (where the response is calculated internally on each request, but only the slice requested in block2 is sent) is impossible for FETCH.

If we do any alterations around block-wise, I suggest we reconsider that choice.

(Also, this is what is making it necessary to fix 9175 for small-request-payload-and-then-block2 cases, but I for one am OK with using Request-Tag there, as in OSCORE the empty request tag can be recycled indefinitely; DTLS users may not be so lucky).

[chrysn]

CC'ing @mrdeep1 whose review pointed me to this. (In the context of today's meeting, this item is not one of your conclusion items, but just my personal take on how the big issue should have been handled in the first place).

[mrdeep1]

To add to this, I have just stumbled over RFC8132 2.3.2 The ETag option

              The FETCH payload is input to
   that selection process and therefore needs to be part of the cache
   key.

which means to me that the payload has to be a part of a FETCH (even if split over many blocks), even when asking for the next Block2 response, so I now have a conflict over my conclusions at today's meeting.

[boaks]

The issues seems to have reached Eclipse/Californium.

Is there already an conclusion, if FETCH SHOULD/MUST send the payload in every request? Especially #28 would raise questions about that,

Maybe adding a special Response Code in an (updated) RFC similar to 2.31 Continue helps to indicate, what a client is expected to send.

[sbernard31]

Just to be sure, payload would be needed in all block2 requests only to support stateless implementation, right ? (or there is something more ?)

Stateless implementation is not possible if FETCH is using block1 and block2 ? because you need to store the FETCH payload somewhere (which is a state), right ?

If that true, I ask myself if this is a good idea to repeat the payload for each block just for this use case. :thinking:

[chrysn]

I don't see a conclusion yet, but maybe we can come to one. Eventually, the WG will need to decide whether we can just change the behavior (by a new document or by just stating that we're now interpreting things this way) or whether Block2 means no-payload-in-later-requests and a new option has to be introduced if we want to change anything. The WG will likely come to implementers about how their impls behave, and here we are.

If we can agree that (even if nothing else) our implementations do tolerate the repeated payload in the request, that'd be a good step toward eventually placing a SHOULD or MUST on the request payloads.

I don't think a new response code would be called for (as to some extent they are the last resort), but a response option would be a way out if we need it.

One design of that option that I could get behind is following a proposal of @mrdeep1 around the Request-Tag topic that triggered this discussion in the first place; this is assuming that we can just start sending request payloads on Block2 requests. If we introduced a response option a la "Your-Request-Tag" (whether or not that would be in the same space as Request-Tag is a detail for later), then the server could indicate (in its first response that had Block2:0/1/x) that it does keep state. Not only would the client then be free to not send the request body in later request, but it may even elide other options (say, Uri-Path) depending on the semantics of that "Your-Request-Tag" option.

payload would be needed in all block2 requests only to support stateless implementation, right ?

Mainly, yes. Having the payload also helps with a possible attack scenario, but that's more about how to cleanly specify how that's handled; the main purpose of repeating the request payload in Block2 requests is to allow stateless FETCH.

As you say, yes, state is required once there is Block1. But for Block2 it really depends on the implementation, and for most FETCH cases (where the response is generated by some kind of stable algorithm), statelessness would be possible quite easily, provided we do repeat the payload.

[boaks]

@chrysn

Thanks for your answer and update.

I don't think a new response code would be called for (as to some extent they are the last resort), but a response option would be a way out if we need it.

I choose the response code, because it is already used in blockwise for a similar function in block1. My assumption is, that this would make it easier to implement it, because the means should be already in place.

If backwards compatibility must be considered, I agree, that a new response code can't help and we the need an Option.

If we can agree that (even if nothing else) our implementations do tolerate the repeated payload in the request, that'd be a good step toward eventually placing a SHOULD or MUST on the request payloads.

We would also need to decide, what should happen, if the payload is changing. I guess, this will be a new transfer.

That's also one pitfall for an implementation without payload in follow up requests, because that doesn't allow the resume such an transfer in the middle when the server may have timedout the transfer and removed the state.

[sbernard31]

that'd be a good step toward eventually placing a SHOULD or MUST on the request payloads.

If this is a MUST, it will not work with block1. That's sounds a bit strange to specify that block2 MUST repeat payload (except if block 1 is also used)

If this is a SHOULD then clients which want to increase interoperability will need to send payload by default. We get same issue as above :point_up: and this also increase traffic just to support specific stateless implementation.

Note that maybe I'm biased because it's hard to me to imagine how purely stateless implementation could work. (I mean what happens if state change between 2 blocks request), so I maybe overlook the stateless implementation use case.

[boaks]

what happens if state change between 2 blocks request

That's also a question for GET and using a ETAG is the answer. If the state changes, a well constructed ETAG is changing. And a compliant client will withdrawn the transfer, maybe starting over a new transfer.

[mrdeep1]

Should Request-Tag be used here to differentiate which response data chunk needs to get sent back instead of repeating data payload?

[sbernard31]

Should Request-Tag be used here to differentiate which response data chunk needs to get sent back instead of repeating data payload?

That would be used to say : "I don't want you to repeat the payload" and so the default behavior will be "always send the payload" ?

It could work but I feel that default behavior should rather be to not send the payload.

[mrdeep1]

That would be used to say : "I don't want you to repeat the payload" and so the default behavior will be "always send the payload" ?

Request-Tag is only allowed to be in a request, not a response. So it cannot be used as a "hint" by the server. I think I was more thinking of stateful servers here where Request-Tag would be sufficient as an alternative to the full FETCH payload.

[chrysn]

As much as I'd like to leave the discussion of the response-option-that-guides-later-requests to a separate thread, we'll need a bit of that to carve out the option space. Likewise, I'd have hoped to leave @mrdeep1's attack out, but that too is part of the outcome and thus needs to be considered.

As I see it, options around a situation with a short request payload and a long response payload are

1. Do nothing.
   • Stateless FETCH stays impossible.
   • We'll need to think hard about how to defeat the attack. Possible outcomes are "the server must be extra careful with its ETags" and "the client must send Request-Tag quite often".
2. Add an option in which the server requests repetition of the payload.
   • This allows stateless FETCH (when supported by the client, but that's the case for all solutions).
   • It doesn't help with the attack; items of 1 still apply.
3. Alter 7959 in short-request-long-response mode to send the payload by default This requires a "breaking" update of 7959. I put breaking in quotes because AIU implementations do different things anyway. (Or a variant that @cabo suggested in today's interim, possibly just the rules for block-wise FETCH, although I'd rather use a code independent model, also due to the attack; that'd only break 8132).
   • Stateless FETCH becomes possible; stateful FETCH has now more overhead.
   • The attack is practically defeated (as long as the client applies the rules of 9175 on when to set a Request-Tag, and the server the rules for when to use an ETag). Possible mitigations for the increased overhead of FETCH:
     1. The client can minimize the overhead by decreasing the block1 size to split the request payload in two parts, effectively creating a long-request-long-response situation. This adds a single round-trip, and works for requests > 16 bytes (but for the smaller ones, the repetition is not too bad). But it's weird to implement.
     2. The server can tell the client that it is stateful, similar to how in a long-request situation it says 2.31 Continue. That option would be elective, stating that the client is free to just send the request payload all over again (as well as all options). If the client supports the option, it sends later requests only with Block2, the reflected option value, and the payload. This even saves data compared to the original scenario, because other request options can be elided as well (precise rules TBD; I think we better pick a different option to Request-Tag as otherwise we get mixups as to who is responsible for the namespace). The breakage would be visible to clients that do not know the updated behavior (they don't need to know the new option). If the server sends a 4.08 Request Entity Incomplete for any later block request without a request payload, it's even kind of within spec (the server just has a very short timeout...). Servers that do have state may support the old behavior if they can be sure to not be affected by the attack. Servers that are stateless didn't work previously (or were just implemented assuming the new behavior), so they don't become broken by this change.

I think that 3.ii is a good way to go.

If that's a direction we can converge on, the next question would be to ask implementers what their servers are doing -- would servers out there tolerate clients sending additional payloads? (For that is the case where actual breakage can occur -- in the other direction, there's the option "if they can be sure to not be affected" above).

For aiocoap (a stateful server), I can report that it ignores any request payload sent in later Block2 requests. This is compatible behavior. (It will break if clients start randomly accessing FETCH responses, but that is the essence of the attack, so I argue it's already broken).

[boaks]

Californium has two modes:

• transparent mode: the blockwise transfer is handled by the coap-stack. This uses a stateful implementation. Payload in follow up request are not sent by the client and ignored by the server. That's the option, which is mainly used.
• random mode: the blockwise requests are forwarded to the application layer. There it depends on the application specific implementation. There it is also possible, that the application ignores the block option and simply returns the full response, in that case the stack handles to cut the block. AFAIK, the random-mode is only rarely used.

Sometimes a client may resume such a blockwise transfer in the middle. Californium handles that as "random mode". If the client provides the payload, then the server is able to resume. If it doesn't add the payload on resume, the outcome is for now undefined.

If the payload gets added for the follow up requests, then it may help to be clear, if a FETCH (block2 0) will be valid even without payload. If FETCH without payload is not valid, a server may be able to detect the mode and may the either perform the transfer or fail.

If a FETCH without payload is invalid, that would also enable Californium to be strict on resuming a FETCH, because an empty payload will then be clearly and error.

In my sum up:

• there are many protocols for different complex applications.
• the domain of coap is in my opinion the more simple ones. So leave the complex stuff to other protocols.
• I don't see a too big benefit in trying to support a "stateless" FETCH. Sure there are very tiny peers, but there are also a lot devices, which ore not that limited. If it gets complex, I see more the bigger devices and there the state is not a that big issue.
• resuming a transfer is for me more attractive to be considered than a "stateless" transfer.

[mrdeep1]

From libcoap's perspective, the server does not care whether the data is there or not for FETCH Block2 NUM != 0, handling multiple Block1 for large payload if needed as it is operating in stateful mode, assuming the coap stack is enabled to request/ send the individual blocks rather than the application.

[mrdeep1]

I think that 3.ii is a good way to go.

I'm inclined to agree. Only comment is that the new option should indicate stateless support (or stateful = yes/no) to be consistent with previous behavior (for FETCH in particular).

How far does the server have to go in being stateless?

• State not required for Block2 transfers
• State not required for Block1 transfers (Something needs to assemble data payload or stream it into the app)
• State not required for NonSec UDP requests (remote IP/port tracked during processing of request)
• State not required for DTLS
• State not required for TCP based connections (PDU may need to be reassembled from multiple packets)

[boaks]

How far does the server have to go in being stateless?

That's a good question. "state" is usually not "every state". e.g. the DTLS state may be lost and reestablished using an fresh handshake. That doesn't really affect the blockwise transfer, but requires to authorize the follow up requests again. For a single request response pair my interpretation of RFC 7252 is more strict and requires to use a new request response pair (means request - handshake - response isn't intended to work).

With that, I consider "State not required for Block2 transfers" and "State not required for Block1 transfers". But as I wrote, for more complex use-cases as a FETCH, I don't see this too important.