Document exclusion lists facilitating deletion

[sandreae]

This is an idea that came to mind while working on the replication announcement specification. We are likely to introduce an exclusion list which limits which documents and peers you want to replicate. I'm considering if we could do something similar locally as a way to support full document deletion. The pattern would go like this:

• you receive an operation instructing you to delete a document
• you stop serving the document immediately
• you retain it's entries and operations until you have replicated the delete with x number of peers
• you then add the document id to your local deletion list which is stored in your local db
• you delete all entries from the deleted document, really everything
• you don't accept any entries for this document from now on, either via replication or publish
• there could be a mechanism for flagging peers who try to send something to you

This would mean that you stop replicating the document, and made a best effort to spread that deletion out on the network. Of course this comes with it's own dangers, data could "re-appear" if the deletion doesn't replicate to all peers, but maybe this is something that could be fine tuned to account for most situations. An additional rule for safety would be:

• if you publish a delete message yourself, then you should keep enough entries to attach it to the document in question (ie, don't follow the above suggested pattern) so that you can replicate out the delete again if the document ever re-surfaces.

[adzialocha]

That's a nice description of a deletion protocol!

I understand exclusion lists as a manual configuration by node maintainers so I'd be surprised to suddenly see my configuration automatically being changed without my consent but if we keep track of that internally, on top of these configurations then it's all good, just want to separate the concerns.

[sandreae]

Yeh, this is actually separate thing from the exclusion list (although that triggered the line of thought). A deletion list would only ever be kept locally, never broadcast (it could get massive sometimes).

[sandreae]

I see I confused terms in there, will fix that.