p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Optimization: only issue new dependency tasks for "parent" documents which contain a pinned relation(list) to "child" document #577

Open sandreae opened 11 months ago

sandreae commented 11 months ago

In our dependency task we need to account for cases when a newly reduced document may have skipped over a view which is pinned from another document (this happens when operations are batched). In this case we want to issue new dependency tasks for any of these "parent" documents (the document which contained the pinned relation). We currently do this by issuing dependency tasks for all documents where their schema contains a pinned relation to the schema of the newly reduced document. We could improve this a bit by identifying documents which actually contain the pinned relation we are looking for. This can be done by querying operation values which are of type pinned_relation or pinned_relation_list where the value matches any child_document_view_id for the current document.

It's really similar to what we do when identifying if a blob should be garbage collected (basically, it has no "parent" documents) here:

https://github.com/p2panda/aquadoggo/blob/a24b2d8bd5809836d7a137bf75aaf5a843a862c0/aquadoggo/src/db/stores/blob.rs#L324-L370

sandreae commented 11 months ago

It would be worth discussing whether this is actually an optimization, we would be swapping fewer tasks issued for a more complex (but maybe not that complex?) query.