use msgpack for json serialization/deserialization

[rudolfix]

Background We use orjson as our main json serializer/deserializer. We had problems with library being unstable ie.

• 1501

• 1009

  and we are considering to switch orjson to be alternative (but still supported, like simplejson) parser. Here we are looking at msgpack to be a main parser. https://github.com/msgpack/msgpack-python

Tasks

1. • [ ] make sure it can be used at all :) does it work without defining types?
2. • [ ] can we make it to pass our json test suite? note that we use custom serializers, need to be able to serialize in deterministic order of fields (for generating hashes etc.) so the tests must pass
3. • [ ] we should compare speed and memory usage on large, nested json files. ping rudolfix for a test case
4. • [ ] if there's a choice we prefer to work over bytes not strings (like orjson)
5. • [ ] pretty formatting may be ignored...

If all goes well this ticket will have a followup where we'll add msgpack as optional dependency and update our docs on how to switch it on. so we are able to swap json parsers at any moment

[donotpush]

Msgpack and (orjson or simplejson) use distinct serialization formats. Msgpack utilizes a binary format for serialization. It is not directly human-readable and does not support loading or writing JSON files, as all attempts to process JSON data fail.

Although data can be converted between msgpack and JSON formats (msgpack for binary to/from Python objects), msgpack is designed to handle and manipulate msgpack binary files exclusively.

[donotpush]

I have failed to make an implementation that passes all the tests:

My code can be found in the branch exp/1539-msgspec-json

And the following notebooks is an exploration related to custom encoding (failed test test_json_named_tuple):

https://github.com/dlt-hub/dlt/blob/exp/1539-msgspec-json/notebooks/msgspec_exp.ipynb

[rudolfix]

@donotpush I'll take a look today. IMO there's still hope :)

[rudolfix]

@donotpush thanks for the research. it is clear that msgspec is not a drop-in replacement for our current framework and requires a little more work.

my take

1. we can close this ticket
2. we can open another one: use msgspec for json serialization/deserialization and to replace typed-json with msgpack

the idea for the ticket is to:

1. use its json ecnoder/decoder for pure json. our tests are actually passing (I added key sorting and it works). they can also pretty format json
2. use its msgpack encoder/decoder for typed_dump - which will produce msgpack document, not our typed-json document.

pls add this followup ticket and move it to todo. we may implement it at some point