search

rocicorp / repc

The canonical Replicache client, implemented in Rust.
Other
30 stars 7 forks source link

We should probably move away from JSON for our values. #162

Open aboodman opened 4 years ago

aboodman commented 4 years ago

https://twitter.com/sayrer/status/1301973647267999745

Now that repc is almost done, it's easier for me to see the criticism that has frequently been given that we are unnecessarily coupled to JSON. I think that we can just push the Map<[u8], [u8]> the rest of the way up the stack all the way to the user. It's trivial to JSON.parse() if that's what you want. But pushing it this last step gives people the option to use far better formats, or whatever format they already use.

phritz commented 4 years ago

I can't see that thread but I was going to suggest the same. IIRC the only reason we were doing json is because we had a thing (noms) that enable efficient diffs and fit the data model. Since we decided that json patch was not that important (part of https://github.com/rocicorp/repc/issues/93#issuecomment-675177752) I don't see any reason to have a more constrained thing rather than a more general thing. 👍

phritz commented 4 years ago

I guess the clientview interface https://github.com/rocicorp/diff-server/blob/0fd187fecd7cb4903c99ff97795d2e425f2407c9/serve/types/types.go#L36 will need a tweak as part of this. We need to decide how to delimit keys and values.

sayrer commented 4 years ago

Reading between the lines on other tweets/comments I've seen, I think this plan is mostly good except that I bet your customers assume you will take care of string encoding/decoding. This is getting pretty good in Rust and Swift 5, but it's a total pain everywhere else.

It used to be that you could safely ignore this issue, at least at first, because only uncommon languages would need this to be correct. That's not true anymore, because the newer emoji are outside the BMP.

I think the idea of letting clients deal in bytes is great, but most of them will want you to deal with encoding (either as JSON or something like Flatbuffers/GraphQL). I suggest JSON for basic stuff (like e-commerce stores), Flatbuffers/GraphQL, then binary. I suspect you can make each a superset of the previous one. Sorry if this is a little spammy, but I think about this problem a lot, and might be inclined to type more than most.

sayrer commented 4 years ago

Found this while researching:

https://github.com/WebAssembly/interface-types/issues/38

and, in there, these slides:

https://docs.google.com/presentation/d/1wtAknL-UJWDoIgSbyF5paTBSpVVj-fKU4tiHMxJbSzE/edit#slide=id.p

It looks like WASM will improve at some point. I suspect Rust strings will work really well w/ Swift right now.

Every other system similar to repc pays this UTF-8 -> UTF-16 cost too, it's just usually in browser-supplied code (e.g. the HTML parser).

phritz commented 4 years ago

summary of notes from discussion: this is about ensuring repc doesn't know about json, it's bytes to bytes. not about app/bindings (which will be json). also not in scope is changing client view from json and patch.

motivations:

concretely: