The incoming record changes are stored as a json object in the database; where they show up in a queue where they can be marked as processed. Once marked as processed, we are at the risk the same records will end up in the queue again, since the ETL process will request all records from a 3-day window, where the same record could appear again and since it carries the same differences it will be inserted in the review queue again.
These are some ideas to overcome that issue:
Keeping the records in the database permanently: If the record already exists in the queue, it will not be inserted. We could keep the records in the database permanently; the only risk is to fill up the database with thousands of unnecessary records.
Removing records after some time: We could leave the records there for some time, more than 3 days. This would block the same changes from showing up in the queue, and also solve the problem of keeping the records forever. The only challenge is to write another ETL process that removes all processed records after X amount of time.
The incoming record changes are stored as a json object in the database; where they show up in a queue where they can be marked as processed. Once marked as processed, we are at the risk the same records will end up in the queue again, since the ETL process will request all records from a 3-day window, where the same record could appear again and since it carries the same differences it will be inserted in the review queue again.
These are some ideas to overcome that issue:
Keeping the records in the database permanently: If the record already exists in the queue, it will not be inserted. We could keep the records in the database permanently; the only risk is to fill up the database with thousands of unnecessary records.
Removing records after some time: We could leave the records there for some time, more than 3 days. This would block the same changes from showing up in the queue, and also solve the problem of keeping the records forever. The only challenge is to write another ETL process that removes all processed records after X amount of time.
Migrated from atd-vz-data #561