Data Integrity

[rivernews]

When working on healthcrowd work, we got unresolvable collision.

After inspection, it is because helpCount can change over time. Because of this, reviews could change overtime.

[rivernews]

Discussion

While we can do more compact data storing, they are quite complicate to maintain. We'll rather keep it simple. But we still need to make trade off on this. Storing an entire object again - indeed is not reasonable.

Some decisions made

• We want to store helpful counts and review data separately.
• For both helpful counts and review data, we want to only write them if they changed. We'll use MD5 here.
• When writing more data to existing dataset, use timestamp in S3 key to avoid overwriting. This should be enough. If we view these changes as a time series, they are no longer collision. It's more like changes observation over time (excluding debug reasons, of course).
• If we use timestamp in filename in any case, we no longer need to check existing data. We may, however, want to generate some sort of report specifying what data is changed.
• Collision strategy will then change to continue and write in any case.

[rivernews]

Approach

• [ ] When writing review data, store helpful counts first, then store review data.
• [ ] For both helpful counts and review data, first pull out latest data, compare their MD5. If different, then write out. If MD5 remains the same, do nothing.
• [ ] When writing to file, always use form of <review id>.<timestamp>.json and <helpfulCount>.<review id>.<timestamp>.json.

[rivernews]

Merged by #27. Store helpfulCount into seperate place from other "stable" fields. AKA, storing varying and stable data separately. so we have better storage efficiency.