Decoding of "unknown" headers - RFC2047 vs raw

[brong]

From the Singapore notes:

Searching in and returning headers:

• lots of debate about decoding of unknown headers, where applying RFC2047 decoding may be incorrect, both for search and for display.
• header names must be ASCII, so lowercasing is safe there, it is well defined.
• there was no agreement in the room, we need to take this one to the list. Should "unknown" headers be returned as raw 7bit bytes, or decoded to characters that become UTF-8 JSON?
• likewise for search - does there need to a knob to do encoded/raw access to the header?

[brong]

A potential option is to provide rawheader and rawheader.$name fetch options as well.  It adds to server complexity, but not greatly, since the server will have to handle the data in that format anyway.

[cjnewman]

Access to rawheader is desirable if a DKIM-like signature mechanism that is designed to survive submission and final-delivery is invented. Note that rawheader may contain UTF-8 if the message is an RFC 6532 message.

IMAP requires servers to decode 2047; JMAP should do the same. The RFC 2047 encoding model can be used in parts of headers where RFC 2047 itself does not allow it. Implementations will decode it in those places if it works well in practice. We should neither require nor forbid it in that scenario; that can be a quality-of-implementation issue (no need for a standard to document every case). For attachment filename/name parameters, both RFC 2231 and misuse of RFC 2047 occur frequently in the field. That might be one place where we explicitly encourage support for RFC 2047 outside its standardized purpose just because attachment filenames interoperate better if both are supported. But RFC 2231 in filename content-disposition should take precedence over the deprecated name parameter.

Don't over-constrain search. Implementations will use whatever search/index technology they've been using regardless of what the specification says. IMAP's substring search semantics were over-specified and often don't work in practice on some implementations. For example, how many servers would match "bstring sear" to the previous sentence?

But requiring decoding of standard 2047 is good for interoperability (otherwise clients might try searching both encoded and un-encoded form which would create a mess).