jchappelow
commented
11 months ago JSON does create problems in the following ways:
- the inefficient encoding is not good for compact blocks, and will likely consume more block space than required
- while JSON is well-specified (marshal <-> unmarshal is reliable), there is flexibility in what is considered valid for a given set of keys and values e.g. spaces and field ordering, as @Yaiba noted today with the python sdk variation
JSON is however great in other ways:
- debugging is simple, and the data is human readable
- broad support inc. Go std library and obv. javascript
An efficient encoding is desirable for disk space reasons, but if code outside of kwild must be able to replicate the serialization, it should be simple.
Another idea is separate serialization for different domains. It may not be possible, but it would be ideal to have the payload serialization used for actual on-disk storage be different from what is used by either (a) clients or (b) p2p messaging in cometbft. Regarding block storage differing from wire/p2p serializations, it may be that we simply have no ability to distinguish the two with cometbft.
Yaiba
JSON serialization is not deterministic across systems, and is pretty inefficient / bloated. Kwil should have its own serialization to solve for these two issues.
ENG-192