DAGWorks-Inc / burr

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, persist, and execute on your own infrastructure.
https://burr.dagworks.io
BSD 3-Clause Clear License
561 stars 29 forks source link

Adds modular SERDE approach #175

Closed skrawcz closed 2 days ago

skrawcz commented 3 weeks ago

This PR introduces a modular SERDE approach.

  1. Singledispatch to register type serializers.
  2. Any complex object deserialization, needs to be accompanied with a "type field" that will map to the name of a "string dispatcher function". i.e. like single dispatch, but you register a string value for it. This approach means that most people will get around the need for field level deserialization / having to tell Burr about state types explicitly.

This skips field level serde registration for now. But leaves room for it.

Otherwise this custom serde for:

TODOs:

Changes

How I tested this

Notes

Checklist

skrawcz commented 4 days ago

Update: just need to test more, but I think I have a good design (and update libraries to latest locally...)

core.serde has functions. core.state uses it and imports integrations state.State knows how to serialize & deserialize using it.

People can then register their own for their application -- I haven't tested overriding but I assume it'll work. Added example of pandas serializing somewhere and just showing link back...

Tracker models should be dumb -- so instead I push serialization calling into hooks. Also persisters updated to use serialization.

Once I merge and rebase off of #215 can add persister tests that will also then check for this...