Also improve the story for restarting processes to avoid data loss / duplication.
Hashtag footprints are the most problematic, as the use of stream-stream joins (changes + changesets) results in a lack of clarity as to which replication sequences have actually been processed into MVT changes.
A likely improvement is to create a dummy layer within the MVTs that contains a list of sequences that have been applied (changeset sequences for hashtags, changes for users) in order to prevent re-applying changes.
mojodna
commented
5 years ago
Also improve the story for restarting processes to avoid data loss / duplication.
Hashtag footprints are the most problematic, as the use of stream-stream joins (changes + changesets) results in a lack of clarity as to which replication sequences have actually been processed into MVT changes.
A likely improvement is to create a dummy layer within the MVTs that contains a list of sequences that have been applied (changeset sequences for hashtags, changes for users) in order to prevent re-applying changes.