Storages replication doesn't replicate the deletions of key expressions

eclipse-zenoh / zenoh

zenoh unifies data in motion, data in-use, data at rest and computations. It carefully blends traditional pub/sub with geo-distributed storages, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks.

Other

1.5k stars 160 forks source link

Describe the bug

If 2 aligned storages suffer a partition and during this partition some key expressions are deleted, after re-alignment of storages the deleted keys are still present in the replicated storage.

To reproduce

Install zenohd and the FileSystem backend

Download and unzip del_test.zip
Run del_test.sh. This test run 2 zenohd with replicated filesystem storages. It publishes 4 k/v that are stored and replicated, kills the 2nd zenohd, deletes 2k/v, restarts the 2nd k/v and waits for alignment of deleted keys which never happens

While the storage in ~/.zenoh/zenoh_backend_fs/d1 has only 2 remaining files, the storage in ~/.zenoh/zenoh_backend_fs/d2 always has 4 files.

System info

Zenoh v0.10.0-rc

The storages replication relies on Zenoh queries: each participant requesting alignment information and missing data to the others. However, in current version, a Zenoh queryable can't send a reply with a "deleted" Sample. Even if the API accepts a Sample object in Query::reply(Sample) operation and this Sample contains a SampleKind::Delete, the Reply protocol message has no place for this SampleKind.

This issue will be addressed in #673 (checkbox "SampleKind in replies")

Then the align queryable shall be modified to:

Retrieve from its storage not only the existing key values, but also the deleted ones from tombstone file. Currently a Session::get() is used (without SessionLocal destination!!). Would be better to directly query the StorageService object.
Reply also deleted samples here

eclipse-zenoh / zenoh