Open dhirschfeld opened 6 months ago
The crux of the issue seems to be reusing str
as the serialialized representation of multiple different types.
Might it be possible to, instead of serializing multiple types as str
, instead give each their own custom wrapper so they had a unique serialized representation as a tagged object - e.g.
from typing import Any, Type
import msgspec
from msgspec import Struct
from datetime import (
date as Date,
datetime as DateTime,
)
import uuid
class Wrapper(Struct, frozen=True, tag=True):
pass
class DateWrapper(Wrapper):
value: Date
class DateTimeWrapper(Wrapper):
value: DateTime
class UUIDWrapper(Wrapper):
value: uuid.UUID
type_map = {
Date: DateWrapper,
DateTime: DateTimeWrapper,
uuid.UUID: UUIDWrapper,
}
def enc_hook(obj: Any) -> Any:
try:
wrapper = type_map[type(obj)]
except KeyError:
return obj
else:
return wrapper(obj)
def dec_hook(type_: Type, obj: Any) -> Any:
if isinstance(obj, Wrapper):
return obj.value
return obj
☝️ pseudo-code, doesn't actually work, but could something like that be made to work?
☝️ pseudo-code, doesn't actually work, but could something like that be made to work?
IIUC the example code doesn't work because enc_hook
isn't called for "known" types such as datetime
so the conversion to str
is effectively hardcoded and not overridable?
If that's correct, the fix could perhaps be to just allow enc_hook
to override the encoding for "known" types so I can e.g. provide my own object wrapper for datetime
?
This is pretty critical functionality for me and could be a dealbreaker for my using msgspec
to serialize arbitrary kwargs (with a finite set of known types).
If I define an example set of arguments:
from functools import singledispatch
from typing import Any, Type
import msgspec
from msgspec import Struct
from datetime import (
date as Date,
datetime as DateTime,
)
import uuid
kwargs = dict(
seed=42,
method='test',
effective_date=DateTime.now(),
today=DateTime.now().date(),
uid=uuid.uuid4(),
)
>>> kwargs
{'seed': 42,
'method': 'test',
'effective_date': datetime.datetime(2023, 12, 14, 20, 59, 7, 281790),
'today': datetime.date(2023, 12, 14),
'uid': UUID('b4aed260-210b-499a-8035-65dbba88b26b')}
I would like to be able to use msgspec
to serialize these kwargs to json
.
I'm fine defining custom serializers/deserializers, I just want the capability, somehow, in msgspec
.
Am I missing something obvious or is there really no way to serialize the example kwargs
to JSON using msgspec
?
It's possible in the stdlib
:
@singledispatch
def enc_hook(obj: Any) -> Any:
raise NotImplementedError(f"Objects of type {obj!r} are not supported")
@enc_hook.register
def _(obj: DateTime) -> dict:
return dict(type='DateTime', value=obj.isoformat())
@enc_hook.register
def _(obj: Date) -> dict:
return dict(type='Date', value=obj.isoformat())
@enc_hook.register
def _(obj: uuid.UUID) -> dict:
return dict(type='UUID', value=obj.hex)
def dec_hook(obj: Any) -> Any:
match obj:
case {'type': 'DateTime'}:
return pd.to_datetime(obj['value'], format='ISO8601').to_pydatetime()
case {'type': 'Date'}:
return pd.to_datetime(obj['value'], format='ISO8601').date()
case {'type': 'UUID'}:
return uuid.UUID(hex=obj['value'])
case _:
return obj
>>> json.loads(json.dumps(kwargs, default=enc_hook), object_hook=dec_hook)
{'seed': 42,
'method': 'test',
'effective_date': datetime.datetime(2023, 12, 14, 20, 59, 7, 281790),
'today': datetime.date(2023, 12, 14),
'uid': UUID('b4aed260-210b-499a-8035-65dbba88b26b')}
Is there a way with msgspec
to similarly losslessly roundtrip any object by defining custom serializers/deserializers?
>>> json.loads(json.dumps(kwargs, default=enc_hook), object_hook=dec_hook) == kwargs
True
>>> msgspec.json.decode(msgspec.json.encode(kwargs))
{'seed': 42,
'method': 'test',
'effective_date': '2023-12-14T20:59:07.281790',
'today': '2023-12-14',
'uid': 'b4aed260-210b-499a-8035-65dbba88b26b'}
>>> msgspec.json.decode(msgspec.json.encode(kwargs)) == kwargs
False
Being able to disambiguate and hence roundtrip a dict
with both string and date values (amongst others) is critical functionality for me so I'm going with the standard-library json
module for now which gives you the flexibility to define your own serialization formats (hence allowing you to make it work).
I'm just posting now to ask if this is likely to be supported by msgspec
in the near future?
I think what you're suggesting would be a nice addition. While this isn't supported, it's possible to do what you want by defining a model dynamically.
kwargs = dict(
seed=42,
method="test",
effective_date=DateTime.now(),
today=DateTime.now().date(),
uid=uuid.uuid4(),
)
def to_dict(obj: msgspec.Struct) -> dict[str, Any]:
return {field: getattr(obj, field) for field in obj.__struct_fields__}
Model = msgspec.defstruct("Model", [(k, type(v)) for k, v in kwargs.items()])
to_dict(msgspec.json.decode(msgspec.json.encode(kwargs), type=Model)) == kwargs # True
Description
There are a wide variety to objects that get mapped to strings which would seem to preclude being able to properly deserialize them with
msgspec
:cry:Simple example - encode/decode kwargs consisting of ~primitive types:
Round-tripping converts
datetime
tostr
:...and it's impossible to specify the actual types:
It seems there are several similar requests:
Also to allow overriding default representations (which would solve my problem as shown at the bottom):