Metadata violates standard and misses expectations

[jaraco]

In https://github.com/pypa/setuptools/issues/1390#issuecomment-762410744, I learned a few things:

• PEP 345 only indicates Description as supporting multiline values.
• PEP 345 indicates a pretty arbitrary prefix for continuation lines in multiline values (|) and alludes to that being RFC 822 compatible, but RFC 822 only expects a single space or tab for a continuation.
• Distutils violates the PEP spec with another arbitrary continuation ( ), additionally violating RFC 822.
• Users would like to include multi-line values in more than just the Description field.
• The RFC 822 long header fields spec isn't particularly well-designed for formats where the newlines are actually meaningful.
• PEP 566 describes a JSON-compatible format, but that format suggests to pass through the values still with continuation characters.

Here's what I propose:

• Make explicit the limits and divergence from RFC 822.
• Drop the | value and update the spec to honor the 8-space format.
• Allow any field to have newlines (from a metadata format perspective).
• Explicitly indicate which fields may or may not have newlines (similar to how each field indicates multi- or single-value).
• Libraries (like importlib.metadata) should provide the metadata with these encoding details (continuation characters) removed, either naturally or through the JSON form.

@pfmoore What do you think of the proposal? Would you expect a PEP for this proposal?

[KOLANICH]

Drop the | value and update the spec to honor the 8-space format.

distlib will be required to be fixed.

[pfmoore]

See here for a discussion on the matter. Your proposal sounds reasonable, but you should co-ordinate with the people in that discussion.

As this is a change that would affect software interoperability, it should technically require a PEP according to the documented process. I would not expect such a PEP to be large, or controversial, assuming there were no substantial objections during the discussion.

Personally, I think that having a PEP for every change like this might be too much, but that's a PyPA governance discussion, so rather than making a unilateral decision, I'll raise that on the Packaging discourse (Edit: Opened a discussion here).