Sweeper/forklift - Removing duplicate records

agrc / planning-queue

A place to manage issues that do not have a dedicated repository or code... yet.

MIT License

0 stars 0 forks source link

Sweeper/forklift - Removing duplicate records #388

Closed rkelson closed 4 months ago

rkelson commented 9 months ago

We need the ability to exclude various layers from having the duplicate records removed, during the scheduled task, once loaded to SDE. The main issue is with the LIR Parcel layers.

https://agrc.slack.com/archives/C018YQ6J14N/p1703707887603919

stdavis commented 6 months ago

This should probably be looked at in forklift as well. IIRC, forklift removes duplicates which would affect AGOL data.

nathankota commented 6 months ago

Sprint 1

create a meeting (@rkelson) and decide on what the fix should be for LIR (and other possible exclusions)
how to deal with duplicates in Forklift

stdavis commented 5 months ago

@rkelson, @steveoh, and I met today and came up with a plan.

[x] https://github.com/agrc/sweeper/issues/114 ~- [ ] https://github.com/agrc/forklift/issues/363~

These are mid-level effort tasks in my opinion. More than a day or so. I'm also going to move this issue to the PI repo since it has grown to be bigger than just sweeper.

stdavis commented 4 months ago

Additional tasks:

[x] deploy new sweeper version to forklift machine
[x] deploy new forklift version to forklift machine

stdavis commented 4 months ago

I asked Rick to reload the LIR data and we verified that neither sweeper nor forklift messed with the duplicate features. I believe that this is done. ✅