Object Dependencies to support CRDT Merges

[vyzo]

A perhaps neglected part of the design was how to support object references for CRDT merge semantics in the application layer.

As of v1.0, we only have a single object reference per simple statement body; this sufficient to merge the metadata directly referenced in the statement. However, we also want to support metadata merging in the application layer, which will create internal object references in the body of the object. These references will be unresolvable in the receiver, because there is no mechanism for fetching the associated metadata at merge time.

We can remedy this situation by adding a deps field in the SimpleStatement protobuf message, and extending merge to fetch dependent objects along with the primary object in the statement. By adding the field in the end of the message, we should be backwards compatible in the sense that already published statements will not be invalidated.

The change is somewhat breaking nonetheless, as a v1.0 node who receives statements with dependencies may not be able to unmarshal them and result in an error. So the sooner we introduces this, the less the effect would be (and it should be relatively painless if we do it for v1.1).

[vyzo]

cc @yusefnapora

[yusefnapora]

I think that putting the dependencies in the statement is probably the only way to do it without breaking the "layer boundary" between statement and object. If references between objects are encoded as e.g. ipld links, we can just scan through the object and collect all the links before publishing the statements. That would cause some duplication, but the redundancy would make it easier to resolve "orphaned" objects if you somehow end up with an object that doesn't have a corresponding statement.

[parkan]

+1 for this approach, it also means we can leave the dependency collection up to the application to allow transitives, optionals, etc.