Question: how to work with data types that has a schema?

[pyrocat101]

Hey! First of all, thanks for creating this awesome library!

I have a question about using this library for JSON data types that has to comply with a schema: on the server side, the server can reject changes that may break the schema, but what should the client do?

For example, if the schema imposes that an array can have at most 1 element. Both Alice and Bob now simultaneously insert an element to the array without knowing each other's change. The server may decide to reject Bob's change since Alice's request is handled first. On the client side, how should Bob handle server-sent update containing Alice's operation (when applied, will break the array length limit)? And if server rejected Bob's operation, should it send any operation regarding this rejection to Bob?

[josephg]

Good question. There's certain schema rules which might be valid locally, but will become invalid after operations have merged. The first answer is don't do this - if you have an array which should contain exactly one element, stop using an array. (Or if you must, ignore anything after the first element.) And always prefer structs instead of tuple types (ie {age:10, name:'fred'} instead of [10, 'fred']). The remaining issue is maximum string lengths.

The fallback approach is to have the server reject any operations which make the schema validation fail. Its going to be rare that this happens - it'll only happen on certain concurrent edits into the object. Say you have a string with a maximum length of 8. The length is currently 7 and two users concurrently type a character into the string. The server will accept the first edit (bringing the length to 8). The second edit will be rejected. In this case I'd have the server error back to the client, and have the client locally just undo their operation. It'll look a little weird in the client (you'd type something and then your typed content would disappear), but its probably rare enough not to matter too much.

[pyrocat101]

@josephg

Thanks for the reply! I wonder how real world applications built on top of OT solves this problem. States in those applications will always have schema. Even Google Doc might have constraints such as "max document length". Is the answer here to "build a specialized OT that respects the schema"?

[josephg]

No; it’s the job of the surrounding concurrency control system. This sort of logic belongs in Sharedb.

[pyrocat101]

@josephg How does the surrounding system solve the case of concurrent array insertion I listed in the first post, if it is critical for Bob maintain the max length invariant at all times? Maybe the array max length concern can be solved by chosen a different data type, but does it generalize to more complex data types and schemas?

[josephg]

As I said above, have the server reject any operation which makes schema validation fail. You need to figure out what to do on the client in those cases, but hopefully those events are rare enough that just locally undoing the operation or something should be acceptable.

[pyrocat101]

@josephg Thanks. That makes sense! I will close the issue now.