search

agrc / forklift

:tractor::package::sparkles: Slinging data all over the place :tractor::package::sparkles:
MIT License
27 stars 3 forks source link

Make duplicate removal an opt-out option #363

Closed stdavis closed 4 months ago

stdavis commented 5 months ago

There are a few exceptions where we want duplicate records to persist through the forklift process. We need to add an option at the crate level to opt out of their removal.

This may create issues with the internal hash-based change detection. Make sure to resolve any issues related to this.

stdavis commented 4 months ago

After digging into the code we realized that crates that use change detection, do not attempt to remove duplicates; just truncate and append. The AGOL crates should all be using change detection. So this feature should not be needed after all.