Open agstephens opened 2 years ago
Thoughts about how to manage multi-level scanning...
Reasons to have a database:
SP suggests that we could use ES queries to tell the item-generator and collection-generator what to scan next.
E.g. get me the latest 1,000 assets that need an item, then work through generating those items.
AS thought that we might update the claim on a record in ES (to avoid another processing claiming it).
Maybe not. An alternative would be:
Previous concerns/questions about pods duplicating work were:
Thoughts about how to manage multi-level scanning...
Do we need a database?
Reasons to have a database:
Could ES be the database?
SP suggests that we could use ES queries to tell the item-generator and collection-generator what to scan next.
E.g. get me the latest 1,000 assets that need an item, then work through generating those items.
How to do claims?
AS thought that we might update the claim on a record in ES (to avoid another processing claiming it).
But do we need to do claims?
Maybe not. An alternative would be: