Activity file with 6 activities, all of them great. They go in data store.
An update is done, one activity now has errors such that the validator marks it not valid. The clean stage will remove it.
Currently
The solrize stage will remove all old activities and put new ones in, so we'll be left with 5 good activities in the datastore.
Feature?
The solrize stage carefully does not remove the last good version of the activity that is now invalid. The datastore has the latest version of 5 good activites in, and an older version of 1 activity from a time when it was not invalid.
Can of worms
I can see this becoming a can of worms; could get very complex and for data removal policy reasons we need to make sure that if a activity is removed from a file it also is removed from the data store!
Scenario
Activity file with 6 activities, all of them great. They go in data store.
An update is done, one activity now has errors such that the validator marks it not valid. The clean stage will remove it.
Currently
The solrize stage will remove all old activities and put new ones in, so we'll be left with 5 good activities in the datastore.
Feature?
The solrize stage carefully does not remove the last good version of the activity that is now invalid. The datastore has the latest version of 5 good activites in, and an older version of 1 activity from a time when it was not invalid.
Can of worms
I can see this becoming a can of worms; could get very complex and for data removal policy reasons we need to make sure that if a activity is removed from a file it also is removed from the data store!