mnot
commented
11 years ago fielding@gbiv.com commented:
From 2332:
Explain in more detail the conformance requirements related to specific roles, recipients, and workarounds; addresses #484
Closed mnot closed 3 years ago
mnot
commented
11 years ago fielding@gbiv.com commented:
From 2332:
Explain in more detail the conformance requirements related to specific roles, recipients, and workarounds; addresses #484
mnot
commented
11 years ago fielding@gbiv.com commented:
A sender MUST NOT generate protocol elements that do not match the grammar defined by the ABNF rules for those protocol elements that are applicable to the sender's role.
The "for those protocol elements..." part should be dropped IMO. A sender MUST NOT generate invalid protocol elements even if they are not applicable to the sender's role. Note that we are talking about generation and not forwarding here.
That isn't why it is being described. There are ABNF rules that define various alternative syntax to be generated based on the role of the sender (and, in some cases, based on the role of the recipient). It is hard to capture in a small number of words.
If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules
"processed" seems too broad because simply buffering a header may be called "processing". "Interpreted" may be better. Or did I miss the definition of "process" that clarifies this?
Actually, no, simply buffering a header is not called processing. "Movement of data or material towards a known goal or end result, by passing it through a series of stages or a sequence of actions." However, I can rephrase it.
If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.
The "excluding only those rules not applicable..." part seems to contradict the "processed" verb. Why would a recipient want to process something inapplicable? Perhaps this is related to the "process" versus "interpreted" issue mentioned above.
A client does not need to parse syntax that is only sent to servers. An origin server does not need to parse syntax that is only sent by a server. ...
the recipient MUST be
able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.
Please rephrase to avoid double negation in "excluding not applicable". For example: "the recipient MUST be able to parse any value matching the corresponding ABNF protocol element rules applicable to the recipient's role"
Okay.
One more attempt at describing the overall conformance requirements has been committed in 2332 to address this part of #484 (and related concerns).
mnot
commented
11 years ago fielding@gbiv.com commented:
From 2334:
Many changes to properly target MUST requirements by role; addresses #484
mnot
commented
11 years ago The following was included in 2334:
Senders MUST exclude the userinfo subcomponent (and its "@" delimiter) when an "http" URI is transmitted within a message as a request target or header field value.
The above MUST should not apply to proxies, right? Policing forwarded URLs will break applications and policing forwarded extension header fields is not possible at all and would violate the "MUST forward" rule. How about "senders MUST NOT generate"?
Right, fixed.
A server MUST be prepared to receive URIs of unbounded length
This MUST may be demoted to "ought" because "be prepared" is too vague (but see below for a related missing MUST).
Fixed.
A server MUST be prepared to receive URIs of unbounded length and respond with the 414
Please insert a second MUST after "and": "and MUST respond".
Fixed.
Multiple header fields with the same field name can be combined into one "field-name: field-value" pair
Should this be a MAY as in "The recipient MAY combine multiple ...". As worded now, it is not clear whether a proxy is allowed to combine headers when forwarding them. Note that this affects extension and other headers that a proxy may not understand (but may still want to combine if allowed to do so).
Fixed.
A server MUST be prepared to receive request header fields of unbounded length and respond
Consider removing the above MUST but please add MUST after "and": A server ought to be prepared to receive ... and MUST respond ... See above for discussion of a similar MUST that applies to URIs of unbounded length.
Fixed.
A client MUST be prepared to receive response header fields of unbounded length.
Same here, except no new MUST is needed.
Fixed.
If chunked is applied to a payload body, the sender MUST NOT apply chunked more than once
The precondition is bogus: If chunked is NOT [yet?] applied to a payload body, the sender still MUST NOT apply chunked more than once!
Fixed.
the sender MUST NOT apply chunked more than once
This needs to be rephrased to make it clear that proxies are not responsible for dechunking multiple chunked encodings to make the forwarded message comply with this MUST. For example, we could say: "the sender MUST NOT generate messages with multiple chunked encodings".
Please note that both the proposed "multiple chunked encodings" and the existing "more than once" wordings imply that foo,chunked,bar,chunked combination is also not allowed.
Both are intentional. A proxy is required to fix framing problems in received messages (or reject them without forwarding). Such messages might be crafted to bypass security filters.
A server MUST send an empty trailer with the chunked transfer coding unless at least one of the following is true:
This should be relaxed to "A server MUST generate ..." because a proxy, in general, does not know whether bullet #2 ("the trailer fields consist entirely of optional metadata...") is true. Even though chunking is a hop-by-hop mechanism, proxies ought to forward Trailers whenever possible, right?
Fixed.
a client MUST send only the absolute path and query components of the target URI as the request-target
To allow for transition to the absolute-form for all requests in some future version of HTTP, HTTP/1.1 servers MUST accept the absolute-form in requests
Should the first "MUST send" be relaxed to "MUST generate" so that the proxies do not block the apparently anticipated "transition to the absolute-form for all requests" by stripping URIs as they forward them?
No, since then such proxies won't interoperate with older servers. I wish we could just delete the second requirement, since it has been proven to be unreliable in practice and wouldn't be reliable for any HTTP/1.x (it isn't needed for support of HTTP/2).
In order to avoid request loops, a proxy that forwards requests to=20 other proxies MUST be able to recognize and exclude all of its own server names
Several intermingled issues here:
1) The "other proxies" prerequisite is a red herring IMO. Any proxy should avoid request loops. If a proxy that is not configured to forward request to other proxies sends an "origin server" request to itself, such a request may still create a loop.
Well, if it sends an origin server request to itself, then as the recipient it takes on the role of an origin server or gateway, not a proxy, and the requirement (as stated) no longer applies ...
I think the following would be better: "In order to avoid request loops, a proxy MUST ..."
Since this section is about forwarding, I have changed it to
An intermediary MUST NOT forward a message to itself unless it is protected from an infinite request loop. In general, an intermediary ought to recognize its own server names, including any aliases, local variations, or literal IP addresses, and respond to such requests directly.
2) If we want the proxy to recognize and exclude, let's demand that instead of just demanding that the proxy is able to do that (but possibly does not do it): "a proxy ... MUST recognize and exclude ...".
Fixed.
3) This MUST may benefit from some polishing to clarify what "exclude" means. I think it means "reject" in this context.
Not necessarily.
4) The "recognize" part can be dropped because recognition is implied by the "exclude" requirement.
At the end, we may arrive at something like this:
"In order to avoid request loops, a proxy MUST reject requests for itself, including requests where the server address is formed using a proxy domain name, its aliases, local variations, or literal IP addresses."
It can handle the requests itself. I have rephrased it.
A client that does not support persistent connections MUST send the "close" connection option in every request message.
Including a CONNECT request message?
Yes (they are orthogonal).
A client that pipelines requests MUST be prepared to retry those requests
MUST be prepared to retry but does not have to retry? Or MUST retry?
Changed to "SHOULD retry unanswered requests".
A client that pipelines requests MUST be prepared to retry those requests if the connection closes before it receives all of the corresponding responses.
Please clarify that the client MUST retry unanswered requests and not all "those requests" it pipelined.
Fixed.
MUST NOT pipeline on a retry connection until it knows the connection is persistent.
Is it really possible to know that a connection is persistent?
Fixed, and rephrased to note that it is due to the TCP reset problem.
...
And here is a list of MUST-level requirements that are missing an explicit actor on which the requirement is placed. Most of these should be easy to rephrase to place the requirement on the intended actor (e.g., "A proxy MUST" instead of "header field MUST":
An unrecognized header field received by a proxy MUST be forwarded downstream
Fixed (and then removed as it was a redundant copy).
The host MUST NOT be empty; if an "http" URI is received with an empty host, then it MUST be rejected as invalid.
Fixed.
the TCP connection MUST be secured,
Fixed.
These special characters MUST be in a quoted string
Removed as it was redundant to ABNF.
the message framing is invalid and MUST be treated as an error
Fixed.
a response message received by a user agent, it MUST be treated as an error
Fixed.
The trailer MUST NOT contain fields
Fixed.
the Host field-value MUST be identical
Fixed.
the Host header field MUST be sent with an empty field-value.
Fixed.
The "Via" header field MUST be sent by a proxy
Fixed already for another ticket.
the connection MUST be closed after the current request/response is complete
Removed as redundant.
all messages on a connection MUST have a self-defined message length
Changed to "need to".
the first action after changing the protocol MUST be a response
Rephrased.
unassigned to 24incorporatednew to closedmnot
commented
11 years ago @mnot changed summary from MUSTs and other feedback to MUSTs and other feedback 5
[ see further discussion / proposals on list ]
The "for those protocol elements..." part should be dropped IMO. A sender MUST NOT generate invalid protocol elements even if they are not applicable to the sender's role. Note that we are talking about generation and not forwarding here.
"processed" seems too broad because simply buffering a header may be called "processing". "Interpreted" may be better. Or did I miss the definition of "process" that clarifies this?
The "excluding only those rules not applicable..." part seems to contradict the "processed" verb. Why would a recipient want to process something inapplicable? Perhaps this is related to the "process" versus "interpreted" issue mentioned above.
Please rephrase to avoid double negation in "excluding not applicable". For example: "the recipient MUST be able to parse any value matching the corresponding ABNF protocol element rules applicable to the recipient's role"
The above MUST should not apply to proxies, right? Policing forwarded URLs will break applications and policing forwarded extension header fields is not possible at all and would violate the "MUST forward" rule. How about "senders MUST NOT generate"?
This MUST may be demoted to "ought" because "be prepared" is too vague (but see below for a related missing MUST).
Please insert a second MUST after "and": "and MUST respond".
Should this be a MAY as in "The recipient MAY combine multiple ...". As worded now, it is not clear whether a proxy is allowed to combine headers when forwarding them. Note that this affects extension and other headers that a proxy may not understand (but may still want to combine if allowed to do so).
Consider removing the above MUST but please add MUST after "and": A server ought to be prepared to receive ... and MUST respond ... See above for discussion of a similar MUST that applies to URIs of unbounded length.
Same here, except no new MUST is needed.
The precondition is bogus: If chunked is NOT [yet?] applied to a payload body, the sender still MUST NOT apply chunked more than once!
This needs to be rephrased to make it clear that proxies are not responsible for dechunking multiple chunked encodings to make the forwarded message comply with this MUST. For example, we could say: "the sender MUST NOT generate messages with multiple chunked encodings".
Please note that both the proposed "multiple chunked encodings" and the existing "more than once" wordings imply that foo,chunked,bar,chunked combination is also not allowed.
This should be relaxed to "A server MUST generate ..." because a proxy, in general, does not know whether bullet #2 ("the trailer fields consist entirely of optional metadata...") is true. Even though chunking is a hop-by-hop mechanism, proxies ought to forward Trailers whenever possible, right?
Should the first "MUST send" be relaxed to "MUST generate" so that the proxies do not block the apparently anticipated "transition to the absolute-form for all requests" by stripping URIs as they forward them?
Several intermingled issues here:
1) The "other proxies" prerequisite is a red herring IMO. Any proxy should avoid request loops. If a proxy that is not configured to forward request to other proxies sends an "origin server" request to itself, such a request may still create a loop. I think the following would be better: "In order to avoid request loops, a proxy MUST ..."
2) If we want the proxy to recognize and exclude, let's demand that instead of just demanding that the proxy is able to do that (but possibly does not do it): "a proxy ... MUST recognize and exclude ...".
3) This MUST may benefit from some polishing to clarify what "exclude" means. I think it means "reject" in this context.
4) The "recognize" part can be dropped because recognition is implied by the "exclude" requirement.
At the end, we may arrive at something like this:
"In order to avoid request loops, a proxy MUST reject requests for itself, including requests where the server address is formed using a proxy domain name, its aliases, local variations, or literal IP addresses."
Including a CONNECT request message?
MUST be prepared to retry but does not have to retry? Or MUST retry?
Please clarify that the client MUST retry unanswered requests and not all "those requests" it pipelined.
Is it really possible to know that a connection is persistent? And what does that really mean for a connection to be persistent "now"? I think all it means is that the connection seems to be "open" -- that the sender has not received an error or connection close notification [yet].
It is possible to know that the connection was persistent (i.e., handled multiple messages), but since the server may close an idle (from the server point of view) connection at any time, I do not think it is possible to know that the next message will reach the server, unless the client and the server are coordinating out of band somehow.
What are we trying to achieve with this requirement? Avoid multiple and possibly endless retries? Minimize the chances that a retry will have to be retried? Perhaps the MUST can be relaxed or reworded to reflect the true intent?
And here is a list of MUST-level requirements that are missing an explicit actor on which the requirement is placed. Most of these should be easy to rephrase to place the requirement on the intended actor (e.g., "A proxy MUST" instead of "header field MUST":
Please be careful with "send" and "generate" when fixing the above actorless rules so that the proxies do not accidentally become responsible for policing traffic where unnecessary.
Reported by @mnot, migrated from https://trac.ietf.org/trac/httpbis/ticket/484