Test deletion of object version (i.e. vclock) corresponding to CRDT version (i.e. context)

[lucafavatella]

Replaces #317. Depends on https://github.com/basho/riak-client-tools/pull/21 (merged).

Please refer to commit message for background and next steps.

[lucafavatella]

Updated this PR following merge of https://github.com/basho/riak-client-tools/pull/21 - tests running now.

[lukebakken]

This is related to basho/riak_kv#1360

As I'm sure you've found, there is no way to resolve a sibling tombstone if a CRDT is updated concurrently with a delete operation.

Are you running into this issue frequently in your environment?

[lucafavatella]

Are you running into this issue frequently in your environment?

@lukebakken Thanks for your note - I was writing more context notes right now and you saved me some typing.

I do not experience this issue in any environment.

[lukebakken]

OK, thanks. I just added some information to basho/riak_kv#1360. Since a resolution to this issue depends on work in Riak I am going to leave this PR open.

[lucafavatella]

@lukebakken Thanks I will follow that ticket then. Feel free to close this PR / change title. I had proposed this (and #317) PR more as a starting point for conversation, and I now understand that that ticket is the correct place to follow.

I became aware of this "not easy point" in the CRDT API while prototyping / designing in a project back in Sep. I now moved on to other items in the same project and may come back to this in Feb 2017 / Mar 2017 (not sure whether and when).

[lucafavatella]

@lukebakken I thought more about this and I understand that basho/riak_kv#1360 is targeting a different issue than the one exposed by this PR.

• basho/riak_kv#1360 aims at merging two already-in-vnode values for the same object - namely a tombstone and a CRDT e.g. after healing from a network partition.

• This PR aims at exposing that Riak KV API does not allow a client to avoid deleting updates it has not seen, hence not creating tombstone in the first place.

  • Do I understand correctly that this is a separate issue (potentially classifiable as wishlist and wontfix)? Or am I missing anything?

---

This PR starts from the assumption that riakc_pb_socket:delete_vclock prevents deleting unseen updates on an object. E.g. for a key:

• Client A reads vclock V1;
• Client B reads vclock V1;
• Client B writes vclock V2 (V2 > V1);
• Then client A calls riakc_pb_socket:delete_vclock with vclock V1;
• Result: the key is not deleted (because V1 < V2) - that is safe.

The tests in this PR aim at preventing deleting unseen updates on a data type e.g. set. E.g. for a key:

• Client A reads context C1;
• Client B reads context C1;
• Client B updates CRDT (at a distance) from context C1, resulting to context C2;
• Context C2 is persisted on all vnodes, no network partition;
• Then client A calls riakc_pb_socket:delete (it cannot call delete_vclock because it has no vclock - only CRDT context C1);
• Result: the key is deleted - that is not safe because client A meant to delete object at CRDT context C1, not any subsequent updates.
  • The tests in this PR show that, in order not to delete unseen updates, client A needs to:
  • Resort to map-reduce for retrieving a consistent snapshot of the couple vclock-context on a vnode (N=1 in map-reduce);
  • Then compare the retrieved context to context C1 (the one based upon which it had taken the decision to delete); and:
    • If equal, issue riakc_pb_socket:delete_vclock with retrieved vclock;
    • If not equal, either:
    • Give up deletion (under likely assumption that there CRDT received update - saying likely as map-reduce could hit vnode with context earlier than C1); or
    • Execute a new read-CRDT & decide & attempt-to-delete-CRDT sequence.

[lukebakken]

Yes, basho/riak_kv#1360 has an eventual goal of dealing with the situation you describe. @russelldb - do you have any comments?