Closed rudolfix closed 1 month ago
Msgpack and (orjson or simplejson) use distinct serialization formats. Msgpack utilizes a binary format for serialization. It is not directly human-readable and does not support loading or writing JSON files, as all attempts to process JSON data fail.
Although data can be converted between msgpack and JSON formats (msgpack for binary to/from Python objects), msgpack is designed to handle and manipulate msgpack binary files exclusively.
I have failed to make an implementation that passes all the tests:
My code can be found in the branch exp/1539-msgspec-json
And the following notebooks is an exploration related to custom encoding (failed test test_json_named_tuple
):
https://github.com/dlt-hub/dlt/blob/exp/1539-msgspec-json/notebooks/msgspec_exp.ipynb
@donotpush I'll take a look today. IMO there's still hope :)
@donotpush thanks for the research. it is clear that msgspec is not a drop-in replacement for our current framework and requires a little more work.
my take
the idea for the ticket is to:
typed_dump
- which will produce msgpack
document, not our typed-json document.pls add this followup ticket and move it to todo. we may implement it at some point
Background We use
orjson
as our main json serializer/deserializer. We had problems with library being unstable ie.1501
1009
and we are considering to switch
orjson
to be alternative (but still supported, likesimplejson
) parser. Here we are looking atmsgpack
to be a main parser. https://github.com/msgpack/msgpack-pythonTasks
orjson
)If all goes well this ticket will have a followup where we'll add
msgpack
as optional dependency and update our docs on how to switch it on. so we are able to swap json parsers at any moment