While writing up Issue #47 it occurs to me that we are going to need to have some model, both for indexers and ingesters, to indicate if a record is new or an update.
The update case is harder, since we may not have the old data, in which case we'd need to fetch it from the database. This suggests that we may need a separate model for "update" versus "initial data insertion." Then again, the idea with bulk loading was that we had a lot of content - the "clean startup" situation. When updating it may be acceptable to just interact directly with the database.
I'll leave this issue so we can think about how this fits into the pipeline. The Indexer->Ingester flow still works with an update model given the way we've structured it (a message queue from the indexer to the ingester) and would be fine for smaller updates.
While writing up Issue #47 it occurs to me that we are going to need to have some model, both for indexers and ingesters, to indicate if a record is new or an update.
The update case is harder, since we may not have the old data, in which case we'd need to fetch it from the database. This suggests that we may need a separate model for "update" versus "initial data insertion." Then again, the idea with bulk loading was that we had a lot of content - the "clean startup" situation. When updating it may be acceptable to just interact directly with the database.
I'll leave this issue so we can think about how this fits into the pipeline. The Indexer->Ingester flow still works with an update model given the way we've structured it (a message queue from the indexer to the ingester) and would be fine for smaller updates.