Proposal: Incorporate SGID.META.ChangeDetection for SGID-based crates

[stdavis]

For crates with sources in SGID (internal) we should take advantage of https://github.com/steveoh/cambiador and use the hashes in SGID.META.ChangeDetection rather than hashing the source data.

Proposed implementation steps

• [x] Add a new config value (changeDetectionTables)
  • [x] This value should be array of paths to change detection tables relative to the garage (e.g. SGID.sde\\SGID.META.ChangeDetection). I'm thinking that we may add cambiador to more databases in the future.
  • [x] At the beginning of a lift, these tables should used to build a lookup dictionary where the key is the table name and the value is the hash.

~- [ ] Extend models.Changes to allow for modeling change without specifying adds or deletes.~

• [x] Create new ChangeDetection class for tracking changes
  • [x] Update to write to a hash table that has only one value per dataset.
• [x] Update core._hash():
  • [x] Take into account the change detection lookup and use that if there is a table name match. ~- [ ] Update to use the changes to the models.Changes class.~
• [x] Update core.update() to handle the newer ChangeDetection class.

Additional questions to answer

• [ ] Are there any crates that use feature-level change detection? Maybe we should make it opt-out-able so that we can use it if needed. This may also make sense for very large datasets (e.g. address points or roads) where a full truncate and load may be slower than a truncate and load of changed features only.

[stdavis]

@steveoh Are you open enough to this proposal that it would be worth my time opening a PR?

[steveoh]

I wonder if this is over complicated. Let's chat in slack