search

luxas / deklarative

A library to work with serialized declarative configuration, e.g. Kubernetes-style YAML in a file
Apache License 2.0
0 stars 0 forks source link

Make `json-iter` decoder able to recognize duplicate fields #5

Open luxas opened 3 years ago

luxas commented 3 years ago

A common way for users to shoot themselves in the foot with YAML (unfortunately) is to misspell something (i.e. pointing to an "unknown field"), indent something at the wrong level (very hard to see this clearly in a large file) (again, becoming an "unknown field" at the wrong context) or register two duplicate fields.

As can be read in YAML 1.2 spec :

JSON's RFC4627 requires that mappings keys merely “SHOULD” be unique, while YAML insists they “MUST” be. Technically, YAML therefore complies with the JSON spec, choosing to treat duplicates as an error. In practice, since JSON is silent on the semantics of such duplicates, the only portable JSON files are those with unique keys, which are therefore valid YAML files.

Due to this; I'm of the opinion that we should treat duplicate fields as errors (or at least, warnings, to begin with) in JSON, too. yaml.v3 errors out if there are duplicate fields already. It can be very non-obvious how parsing is done if duplicate fields don't yield an error, e.g. encoding/json where duplicate fields with object values are merged, whereas all other types are replaced with the latest value (ref: https://github.com/golang/go/issues/24415#issuecomment-373513285)

I coded a sample implementation of duplicate fields error support in upstream json-iterator: https://github.com/luxas/json-iterator/tree/error_for_duplicates

But, I think it should be possible to write this as a json-iter extension as well; that might be the way to go for the moment. If json-iter supported "warnings", it'd also be possible to begin the transition to duplicate and unknown fields erroring by showing the user warnings (in the spirit of https://kubernetes.io/blog/2020/09/03/warnings/)

luxas commented 3 years ago

Two examples of real-world problems this may cause:

image image