httpwg / http-core

Core HTTP Specifications
https://httpwg.org/http-core/
466 stars 43 forks source link

Ben Kaduk's COMMENT on semantics #914

Closed mnot closed 3 years ago

mnot commented 3 years ago

In Section 10.2 we had some good text cleanup (I think, prompted by one of my comments -- thank you!), but the removed text included a note about how the semantics of a response header field might be refined by the semantics of the request method and/or the response status code. That seems like it would be useful to have mentioned, and I'm not sure if this text was replicated elsewhere.

Editors: The problem with that wording was that a field has defined semantics that include variations based on the context in which it appears, and those variations are included as part of the field definition (not refined by other parts of the specification). Ultimately, we decided that this doesn't need to be said in the section intro.


This document updates RFC 3864, which is part of BCP 90. However, this document is targeting Proposed Standard status, which means it cannot become a part of BCP 90 as part of that update. Did we consider splitting out the RFC 3864 updates into a separate, BCP-level, document, that would become part of BCP 90?

Editors: The intended status for this document is full standard. The reason it updates RFC 3864 (which previously defined the registry for HTTP header fields as part of a registry for all application-level IMF-like protocols) is that IETF thinking has changed since 3864. Having a single IMF-wide definition of fields was unsuccessful and led to more confusion when fields diverged. Hence, this document is obsoleting only the HTTP parts of RFC 3864 by moving them back to the standards track. It is an update only because there is no status for partial obsoleting.


HTTP/2 ([RFC7540]) introduced a multiplexed session layer on top of the existing TLS and TCP protocols for exchanging concurrent HTTP messages with efficient field compression and server push. HTTP/3 ([HTTP3]) provides greater independence for concurrent messages by using QUIC as a secure multiplexed transport over UDP instead of TCP.

My understanding was that h2 and h3 also use non-text-based headers, in contrast to HTTP/1.1's "text-based messaging syntax" that we mention earlier. Is that non-text nature worth noting here?

Editors: No. This is not an overview of the differences between the protocols; just a brief introduction.


Proxies are often used to group an organization's HTTP requests through a common intermediary for the sake of security, annotation services, or shared caching. [...]

The term "security" can mean so many different things to different audiences that its meaning in isolation is pretty minimal. I suggest finding a more specific term for the intended usage, perhaps relating to an auditing, exfiltration protection, and/or content-filtering function.

Editors: That is often how proxy-based products are sold/positioned in the market.


For example, an interception proxy [RFC3040] (also commonly known as a transparent proxy [RFC1919]) differs from an HTTP proxy because it is not chosen by the client. Instead, an interception proxy filters or redirects outgoing TCP port 80 packets (and occasionally other common port traffic). Interception proxies are commonly found on public network access points, as a means of enforcing account subscription prior to allowing use of non-local Internet services, and within corporate firewalls to enforce network usage policies.

Is this text still accurate in the era of https-everywhere and Let's Encrypt?

Editors: They are still deployed, yes. On a public access network, the first TLS request will fail. User agents recognize such failures and fall back to a plain HTTP access to a common URL, which is then intercepted by the filter and the user agent is directed to login for Internet access. You can see this in every hotel, cafe, and convention center.


As Éric notes, OpenSSL 0.9.7l supports only SSL and TLSv1.0, which per RFC 8996 is no longer permitted -- I concur with his recommendation to update the example (potentially including Last-Modified).

Editors: already addressed.


The hierarchical path component and optional query component identify the target resource within that origin server's name space.

Would a BCP 190 reference be appropriate here (emphasizing that the name space belongs to the origin server)?

Editors: Not really. This section is defining what those components are for. BCP 190 is advice for specifications that assume certain hierarchies within applications. Most readers would find that to be an unnecessary distraction at best, or a circular down-reference at worst.


The "https" URI scheme is hereby defined for minting identifiers within the hierarchical namespace governed by a potential origin server listening for TCP connections on a given port and capable of establishing a TLS ([RFC8446]) connection that has been secured for HTTP communication. [...]

Is "capable" the correct prerequisite, or does the server need to actually do so on that port? (Given the following definition of "secured", though, the ability to successfully do so would seem to depend on the trust anchor configuration on the client, which is not really something the server can control...)

Editors: "capable" is correct. As stated above that in 4.2, the server does not need to exist.


Note, however, that the above is not the only means for obtaining an authoritative response, nor does it imply that an authoritative response is always necessary (see [Caching]).

Is it intentional that this paragraph diverges from the analogous content in §4.3.2 (which also mentions Alt-Svc and other protocols "outside the scope of this document")?

Editors: Yes, it is intentional. It isn't necessary to repeat the Alt-Svc example, and the last sentence (for alternative access to "http" resources) is encompassed by the definition of "https" authority by certificate match.


The reference seems to conclude only that the situation for "Set-Cookie" is underspecified, and doesn't really give me much guidance on what to do if I receive a message with multiple field lines for "Set-Cookie". (It does talk about the "Cookie" field and how semicolon is used to separate cookie values, which implies that "Cookie" would get special treatment to use semicolon to join field lines, but doesn't really give me the impression that "Set-Cookie" should also have such treatment.)

Editors: Handling for Set-Cookie and Cookie are not defined by this specification; this is just an informative note.


A client MAY discard or truncate received field lines that are larger than the client wishes to process if the field semantics are such that the dropped value(s) can be safely ignored without changing the message framing or response semantics.

Is it worth saying anything about fields that the client does not recognize? (Per the previous discussion, the server needs to either know that the client recognizes the field or only send fields that are safe to ignore if unrecognized, if I understand correctly...)

Editors: That's not relevant here.


In a response, the content's purpose is defined by both the request method and the response status code (Section 15). For example, the content of a 200 (OK) response to GET (Section 9.3.1) represents the current state of the target resource, as observed at the time of the message origination date (Section 10.2.2), whereas the content of the same status code in a response to POST might represent either the processing result or the new state of the target resource after applying the processing.

Doesn't the last clause mean that there is some additional (meta)data that can affect the content's purpose (e.g., a Content-Location field)? Or how else would one know if the 200 POST response is the processing result vs the new state? It seems incomplete to just say "is defined by both" and list only method and status code as the defining factors.

Editors: https://github.com/httpwg/http-core/pull/925


[I had the same question as Martin Duke about default TCP port, and the interaction with the scheme. I see that it has been answered since I initially drafted these notes, hooray.]


A proxy MUST NOT modify the "absolute-path" and "query" parts of the received target URI when forwarding it to the next inbound server, except as noted above to replace an empty path with "/" or "*".

I found where (in the discussion of normalization in §4.2.3) we say to replace the empty path with "/" for non-OPTIONS requests. I couldn't find anywhere "above" where it was noted to replace an empty path with "*" (presumably, for the OPTIONS requests), though.

Editors: #930


Implementers are encouraged to provide a means to disable such sniffing.

"encouraged to provide a means to disable" could be read as also encouraging implementation of the (sniffing) mechanism itself. Is it actually the case that we encourage implementation of MIME sniffing?

Editors: https://github.com/httpwg/http-core/commit/085352bac07be803a6f20f69f4a5e06557bfecc2


My understanding is that, e.g., a cryptographic hash over the representation and metadata would be intended to be a strong validator, but for such a construction the "unique" property can only be guaranteed probabilistically. Are we comfortable with this phrasing that implies an absolute requirement?

Editors: yes.


Section 8.8.4

I'm failing to come up with an explanation for why it's important to specifically call out the HTTP/1.0 origin server in the latter case. What's special about an HTTP/1.1 origin server that only provided a Last-Modified value and subrange cache validation requests that makes the MAY inapplicable? (What's the actual expected behavior for that situation?)

Editors: #933


A request method is considered idempotent if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request. [...]

I sometimes worry that a definition of idempotent like this hides the interaction of repeated idempotent requests with other requests modifying the same resource. A+A is equivalent to A, but A+B+A is often not equivalent to A+B...

Editors: The definition of idempotent is about a single user agent's intent being repeatable (automatically retried on failure). The user's intent does not depend on the resource state unless the user makes it so using the conditional request mechanism defined in Section 13. We could add a forward reference here, but it is already discussed in 8.8.


Likewise, other implementation aspects of a resource might need to be deactivated or archived as a result of a DELETE, such as database or gateway connections. In general, it is assumed that the origin server will only allow DELETE on resources for which it has a prescribed mechanism for accomplishing the deletion.

The specific phrasing of "only allow DELETE [...]" calls to mind (for me) an expectation of authorization checks as well. In some sense this is no different than for POST or PUT, and thus may not be worth particular mention here, but I thought I'd ask whether it makes sense to mention authorization (and authentication).

Editors: Not worth particular mention.


A client SHOULD NOT generate content in a DELETE request. Content received in a DELETE request has no defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request.

We had a similar paragraph earlier in the discussion of GET and HEAD, but those paragraphs included a clause about "close the connection because of its potential as a request smuggling attack" -- is DELETE not at risk of use for request smuggling?

Editors: this has been fixed in a prior issue


The referenced section seems to cover the "close" connection option, which is a positive signal of intent to close. Is the absence of that connection option to be interpreted as a positive signal of intent to continue reading the request content, or is there some other positive signal of such intent to continue reading?

Editors: It is version specific. For example, HTTP/1.1 is persistent by default, so the absence of close is a positive signal.


A server SHOULD NOT use the From header field for access control or authentication.

It seems that the level of security provided by the From header field is at most that of a bearer token, and that the natural choice of such token is easily guessable (though unguessable choices are possible). I'm having a hard time coming up with an IETF-consensus scenario where it would make sense to use From for access control or authentication (i.e., could this be MUST NOT instead?).

Editors: already discussed as part of Francesca's feedback.


Some servers use the Referer header field as a means of denying links from other sites (so-called "deep linking") or restricting cross-site request forgery (CSRF), but not all requests contain it.

I think we should say something about the effectiveness of Referer checks as a CSRF mitigation mechanism.

Editors: That's a moving target that is often browser or organization-dependent. If we could get anyone to agree on a common opinion, let alone a common implementation, it might make an interesting BCP.


Most general-purpose user agents do not send the Referer header field when the referring resource is a local "file" or "data" URI. A user agent SHOULD NOT send a Referer header field if

This seems like a curious statement. Are we expecting future general-purpose user agents to emulate this behavior? If so, then why not recommend it explicitly?

Editors: Referer policy is more under control of the W3C's WebAppSec WG; these are just general guidelines.


the referring resource was accessed with a secure protocol and the request target has an origin differing from that of the referring resource, unless the referring resource explicitly allows Referer to be sent. A user agent MUST NOT send a Referer header field in an

How does a referring resource indicate that Referer should be sent?

Editors: Out of scope for this document. W3C defines referer-policy, but that's browser-centric.


The TE field value consists of a list of tokens, each allowing for optional parameters (except for the special case "trailers").

Should the prose mention the 'weight' part of the t-codings construction (the "weight" production itself does not seem to be defined until §11.4.2)?

Editors: https://github.com/httpwg/http-core/pull/918


Please pick one of "message integrity check" and "signature" and use it consistently; these are both cryptographic terms of art (with different meanings).

Editors: https://github.com/httpwg/http-core/pull/919


Because the Trailer field is not removed by intermediaries, it can also be used by downstream recipients to discover when a trailer field has been removed from a message.

It seems that this usage is only possible if sending the Trailer field is a binding commitment to emit the relevant trailer fields; otherwise the recipient cannot distinguish between a removal by an intermediary and a sender declining to generate the trailer field.

Editors: 6ee7338d


A user agent SHOULD send a User-Agent header field in each request unless specifically configured not to do so.

(I assume that a reference to client-hints (or UA-CH) was considered and rejected.)

Editors: yes.


A user agent SHOULD NOT generate a User-Agent header field containing needlessly fine-grained detail and SHOULD limit the addition of subproducts by third parties. Overly long and detailed User-Agent field values increase request latency and the risk of a user being identified against their wishes ("fingerprinting").

client-hints might even be more appropriate as a reference here than it would be above...or just in §17.13.

Editors: CH is Experimental.


Section 10.2

It seems like it might be worth listing the fields already defined in the previous section (as request context fields) that can also appear as response context fields.

Editors: Good catch. Trailer and Date are bidirectional fields, so it would be better to make a separate section for them, which would be either 10.1 or 10.3 (depending on on references in HTTP/3). #934


I'm not sure that I understand how an HTTP extension would help specify a mechanism for automatic selection in reactive negotiation; isn't this just an implementation detail in the user-agent?

Editors: Perhaps we should just remove "as an extension", since this isn't specific to HTTP? The URI and Alternates fields were proposed long ago for that purpose but did not attain sufficient implementation to remain in the standard. It is commonly implemented today using JavaScript in non-uniform ways. Likewise, HTML was extended to include the srcset attribute on img.


This note seems like it would be more useful in the IANA media-types registry than "some random protocol specification that uses media types".

Editors: same as https://github.com/httpwg/http-core/issues/849#issuecomment-857623754


For example,

   Accept-Encoding: compress, gzip
   Accept-Encoding:
   Accept-Encoding: *
   Accept-Encoding: compress;q=0.5, gzip;q=1.0
   Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0

Are these supposed to be multiple standalone examples or one single example with multiple field lines? (I note that they appear in a single element in the XML source.) If they are supposed to be one single example, I would have expected some remark about the combination of "" and ";q=0" (my understanding is that the q=0 renders codings not listed as unacceptable, even despite the implicitly q=1 wildcard). It seems that in other instances where we provide multiple examples in a single artwork, the prefacing text is "Examples:" plural, that makes some effort to disambiguate.

Editors: https://github.com/httpwg/http-core/pull/920


This wording implies to me that there is a normative requirement somewhere else that qvalues cannot be used with x-gzip and x-compress, but I'm not sure where that would be. (It's also a bit hard to understand how x-gzip would be affected but not plain gzip, given that §18.6 lists it as an alias for gzip ... additional restrictions don't quite match up with an "alias" nature.)

Editors: This note reflects historical practice in 1996. Removed in https://github.com/httpwg/http-core/commit/082b619a2c62209f13e2ea771d525e248a64b777


It might be contrary to the privacy expectations of the user to send an Accept-Language header field with the complete linguistic preferences of the user in every request (Section 17.13).

This leaves me wondering how to improve on the situation and pick which subset of requests to send the header field in. I would expect that a blind random sampling approach would not yield privacy improvements over always sending them.

Editors: this comment does not appear actionable.


Section 12.5.5

An origin server SHOULD send a Vary header field when its algorithm for selecting a representation varies based on aspects of the request message other than the method and target URI, unless the variance cannot be crossed or the origin server has been deliberately configured to prevent cache transparency. [...]

I don't think I know what it means to "cross" a variance. The example (elided from this comment) about Authorization not needing to be included gives some hint as to what is meant, but I still don't have a clear picture.

Editors: #938


Section 13.2.2

  1. When the method is GET and both Range and If-Range are present, evaluate the If-Range precondition:

    • if the validator matches and the Range specification is applicable to the selected representation, respond 206 (Partial Content)
  2. Otherwise,

    • all conditions are met, so perform the requested action and respond according to its success or failure.

I think that if the If-Range doesn't match, we're supposed to ignore the Range header field when performing the requested action, which doesn't seem to match up with this unadorned directive to "perform the requested action" (which would include the Range header field). (We might also change point (5) to use the "if true" phrasing that the other items use in the context of evaluating the precondition.)

Editors: #940


This document obsoletes RFC 7538; if we believe that content is still useful we should probably consider incorporating it into this document.

Editors: nope. We already here again and again that the spec is too long. Readers who care about these deployment issues can easily navigate to the reference spec. It's not required to understand the protocol.


Field names are registered on the advice of a Designated Expert (appointed by the IESG or their delegate). Fields with the status 'permanent' are Specification Required ([RFC8126], Section 4.6).

I would have expected IANA to ask for the phrase "Expert Review" to be used for the general case (if they did not already), since that's the relevant registration policy defined in RFC 8126.

Editors: And yet, they did not.


Registration requests consist of at least the following information: [...] Specification document(s): Reference to the document that specifies the field, preferably

If the registration consists of "at least" a group of information that includes a specification document, doesn't that mean the policy is always "Specification Required", not just for permanent registrations?


Provisional entries can be removed by the Expert(s) if - in consultation with the community - the Expert(s) find that they are not in use. The Experts can change a provisional entry's status to permanent at any time.

(The ability to freely convert a provisional registration to permanent seems to also require a specification document to always be present, even for provisional registrations.)

Editors: no action evident.


A few potential considerations that don't seem to be mentioned in the subsections:

Editors: It is very late in the process to introduce such substantial text, especially when it would need additional review due to security impact. As this is a COMMENT, not a DISCUSS, we will not act upon this.


Unfortunately, communicating authority to users can be difficult. For example, phishing is an attack on the user's perception of authority, where that perception can be misled by presenting similar branding in hypertext, possibly aided by userinfo obfuscating the authority component (see Section 4.2.1). [...]

We might also mention "confusable" domain names here as well (which are possible even without resorting to IDNs).

Editors: Same as above.


Should we also discuss situations where there might be redundant lengths at different encoding layers (e.g., HTTP framing and MIME multipart boundaries), in a similar vein to https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34#section-10.8 ?

Editors: Same as above.


Authentication schemes that solely rely on the "realm" mechanism for establishing a protection space will expose credentials to all resources on an origin server. [...]

There's also not any clear authorization mechanism for the origin to claim use of a given realm value, which can lead to the client sending credentials for the claimed realm without knowing that the server should be receiving such credentials.

Editors: This doesn't appear to be actionable.


Should RFC 5322 be normative? We rely on it for, e.g., the "mailbox" ABNF construction.

Editors: https://github.com/httpwg/http-core/issues/924


[Just noting that I did not attempt to validate the ABNF, since the shepherd writeup notes that they have been validated]


Clarified that If-Unmodified-Since doesn't apply to a resource without a concept of modification time. (Section 13.1.4)

I couldn't really locate which text was supposed to be providing this clarification.

NITS


Most resources are identified by a Uniform Resource Identifier (URI), as described in Section 4. [...] HTTP relies upon the Uniform Resource Identifier (URI) standard [RFC3986] to indicate the target resource (Section 7.1) and relationships between resources.

Are these two statements compatible? (What is used for the non-URI resource identification scenarios?)

Editors: the "most" is referring to the fact that some resources don't have explicit identifiers (as explained elsewhere).


We seem to use the obs-text ABNF construction prior to its definition, which is in Section 5.6.4.

Editors: https://github.com/httpwg/http-core/pull/937


In any production that uses the list construct, a sender MUST NOT generate empty list elements. In other words, a sender MUST generate lists that satisfy the following syntax:

1#element => element *( OWS "," OWS element )

Are the two formulations equivalent without some restriction on 'element' itself?


I count two "target resource" and two "resource identified by the target URI". Is there an important distinction between those two phrasings or could we normalize on a single term?

Editors: c5db347


Section 7.3.3

If no proxy is applicable, a typical client will invoke a handler routine, usually specific to the target URI's scheme, to connect directly to an origin for the target resource. How that is accomplished is dependent on the target URI scheme and defined by its associated specification.

This document is the relevant specification for the "http" and "https" URI schemes; a section reference to the corresponding procedures might be in order.

Editors: #952


An origin server with a clock MUST NOT send a Last-Modified date that is later than the server's time of message origination (Date). If

I suspect some relevant details for this clock are covered in §10.2.2; maybe a forward reference would be useful.


The remaining response header fields provide more information about the target resource for potential use in later requests.

I didn't see a previous enumeration of fields such that "remaining" would have meaning. (Also, the whole toplevel section seems to contain multiple sentences that are nearly redundant.)

Editors: That text has been rewritten based upon other comments.


Section 10.2.2

A recipient with a clock that receives a response message without a [...] A recipient with a clock that receives a response with an invalid

Are we using "with a clock" as shorthand for "have a clock capable of providing a reasonable approximation of the current instant in Coordinated Universal Time"? It might be worth clarifying if this different phrasing than above is intended to convey different semantics.

Editors: #953


The Proxy-Authentication-Info response header field is equivalent to Authentication-Info, except [...]

Is it worth calling out again that it can be sent as a trailer field, in case someone specifically goes searching for trailer fields?

Editors: #946


Section 13.2.1

server that can provide a current representation. Likewise, a server MUST ignore the conditional request header fields defined by this specification when received with a request method that does not involve the selection or modification of a selected representation, such as CONNECT, OPTIONS, or TRACE.

We do say "can be used with any method" regarding If-Match, earlier, which is not very well aligned with this "MUST ignore".

Editors: #954


  1. If the request method has been changed to GET or HEAD, remove content-specific header fields, including (but not limited to) Content-Encoding, Content-Language, Content-Location, Content-Type, Content-Length, Digest, ETag, Last-Modified.

The discussion in §8.8.3 seems to indicate that ETag is only used in responses, not requests, so I'm not sure in what scenarios it would need to be removed from the redirected request.

Editors: #917


I had to read this text several times to find a way to understand it that seems to make sense to me (but might still be wrong!). I think part of my confusion is that the word "former" is being used in two different senses (the first of the two choices, and the historical/earlier version). Perhaps it's more clear to just talk about "method rewriting" (and not rewriting) instead of using the overloaded term.

Editors: #948

mnot commented 3 years ago

Re: Appendix B.4 -- This text originally referred to the resolution of #326, but that was subsequently overwritten when we aligned the way we specified conditionals. It probably needs to be re-introduced (and perhaps looked at for the other conditionals).

mnot commented 3 years ago

Closing, as all remaining issues have been split out.

kaduk commented 3 years ago

A server SHOULD NOT use the From header field for access control or authentication.

It seems that the level of security provided by the From header field is at most that of a bearer token, and that the natural choice of such token is easily guessable (though unguessable choices are possible). I'm having a hard time coming up with an IETF-consensus scenario where it would make sense to use From for access control or authentication (i.e., could this be MUST NOT instead?).

Editors: already discussed as part of Francesca's feedback.

Since I put the effort in to track it down, I'll note for posterity that Francesca's corresponding feedback was item 14 at https://github.com/httpwg/http-core/issues/849 and bde626d7cdfd2a6221e1d2f4ec6c87f40dd8c7be is how it got addressed. (I had gotten confused about the relative timing of the respective reviews which prompted me to actually look at the history.)