jcrist / msgspec

A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML
https://jcristharif.com/msgspec/
BSD 3-Clause "New" or "Revised" License
2.01k stars 59 forks source link

Duplicate key detection #648

Open DavidBuchanan314 opened 4 months ago

DavidBuchanan314 commented 4 months ago

Description

When decoding plain JSON with duplicate keys, all but the last are silently ignored, for example:

>>> msgspec.json.decode(b'{"abc":1,"abc":2}')
{'abc': 2}

This behaviour is "correct" (it's noted by rfc8259 as a possible behaviour), but I would nontheless like to detect this scenario and raise an exception - silently discarding information is undesirable for my use case.

Python's built-in JSON module can be extended to perform this checking like so: https://gist.github.com/DavidBuchanan314/bf57251b2519c5ea55042fc4af62e033, and ideally I'd like similar functionality from msgspec (I don't care about object_pairs_hook specifically, just the ability to check dupes).

p.s. this feature has also been requested (and closed as stale) in orjson ;) https://github.com/ijl/orjson/issues/456