Preserve GlobalIds

[stdavis]

Description of Changes

This pull request changes the update process for data that uses change detection so that global id values are preserved. This is done by deleting the destination feature class and then copying the old features. I was not able to find a way to preserve global ids with the append tool when the target is a FGDB.

TBH, I'm not sure that this is a great idea. I'm worried about it messing with relationship classes in the output gdb. I'd love any thoughts that anyone has about it.

This PR also bumps the testing deps (which is related to #340) and modernizes the change detection test suite.

Test results and coverage

================================================================================== test session starts ===================================================================================
platform win32 -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: W:\forklift, configfile: pytest.ini, testpaths: tests
plugins: instafail-0.4.2, requests-mock-1.8.0, anyio-2.2.0, cov-2.12.1, isort-2.0.0, pylint-0.18.0
collected 215 items                                                                                                                                                                       
--------------------------------------------------------------------------------
Linting files
.
--------------------------------------------------------------------------------

tests\__init__.py ss                                                                                                                                                             [  2/215]
tests\PalletWithSyntaxErrors.py ss                                                                                                                                               [  4/215]
tests\SchemaLockPallet.py ss                                                                                                                                                     [  6/215]
tests\benchmark_arcgis.py ss                                                                                                                                                     [  8/215]
tests\conftest.py ss                                                                                                                                                             [ 10/215]
tests\mocks.py ss                                                                                                                                                                [ 12/215]
tests\test_arcgis.py ss.................                                                                                                                                         [ 31/215]
tests\test_change_detection.py ............                                                                                                                                      [ 43/215]
tests\test_changes.py ss......                                                                                                                                                   [ 51/215]
tests\test_config.py ss......                                                                                                                                                    [ 59/215]
tests\test_core.py ss....................................                                                                                                                        [ 97/215]
tests\test_crate.py ss...................                                                                                                                                        [118/215]
tests\test_engine.py ss......................s..........                                                                                                                         [153/215]
tests\test_lift.py ss.........                                                                                                                                                   [164/215]
tests\test_messaging.py ss.....                                                                                                                                                  [171/215]
tests\test_pallet.py ss................................                                                                                                                          [205/215]
tests\test_seat.py ss....                                                                                                                                                        [211/215]
tests\test_slack.py ss..                                                                                                                                                         [215/215]

==================================================================================== warnings summary ====================================================================================
c:\users\agrc-arcgis\appdata\local\esri\conda\envs\forklift\lib\site-packages\_pytest\config\__init__.py:1233
  c:\users\agrc-arcgis\appdata\local\esri\conda\envs\forklift\lib\site-packages\_pytest\config\__init__.py:1233: PytestConfigWarning: Unknown config option: log_print

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/warnings.html

---------- coverage: platform win32, python 3.7.10-final-0 -----------
Name                               Stmts   Miss Branch BrPart     Cover   Missing
---------------------------------------------------------------------------------
src\forklift\arcgis.py               113     29     32      2    70.34%   26, 79-107, 137-138, 143, 146, 163, 178->181, 181, 197->198, 198-200, 222-224
src\forklift\change_detection.py      64      6     16      4    87.50%   32-34, 51->52, 52, 54->55, 55, 81->82, 81->84, 82
src\forklift\config.py                50      1     20      2    95.71%   82->83, 83, 118->117
src\forklift\core.py                 197      6     84      4    95.73%   97->100, 100, 129->130, 130, 153-154, 270, 301->300, 429->430, 430
src\forklift\engine.py               477    171    160     12    61.54%   133->136, 136, 155-156, 212, 224->225, 225, 237->238, 238-318, 322->368, 332->333, 333-335, 358-359, 397-468, 498, 508, 522-538, 543-556, 571->572, 572, 592, 610->611, 611, 624-627, 678->681, 681-690, 718->721, 721-722, 724->727, 727, 741-747, 753, 757->760, 760, 805->806, 806, 911-954
src\forklift\lift.py                 148     52     52      4    61.00%   34, 41->42, 42, 152->153, 153-156, 159->163, 163-164, 187-198, 233-254, 281, 284->285, 285, 292-295, 310-317, 346-357
src\forklift\messaging.py             53     10     16      1    72.46%   58, 78->79, 79, 100-112
src\forklift\models.py               196      4     52      3    96.37%   96, 183->182, 404->407, 407, 451->459, 459-460
src\forklift\slack.py                211     33    102     33    75.72%   24->25, 25-27, 54->57, 57, 71->72, 72-77, 79->80, 80-85, 90->93, 100->101, 101, 102->103, 103, 104->105, 105, 111->114, 116->117, 117-118, 120->87, 134->137, 137, 151->184, 155->158, 160->161, 161-164, 165->168, 168->171, 174->177, 189->192, 195->197, 197->200, 202->205, 236, 259->260, 260, 265, 276->278, 278->279, 279, 284->287, 287->288, 288, 304->305, 305, 331->332, 332, 333->334, 334, 347->348, 348, 351->354, 354->357, 360, 372, 375
---------------------------------------------------------------------------------
TOTAL                               1531    312    540     65    76.10%

3 files skipped due to complete coverage.

================================================================= 180 passed, 35 skipped, 1 warning in 240.03s (0:04:00) =================================================================

Speed test results

W:\forklift (feat/preserve-globalids)
(forklift) λ python -m forklift speedtest
Setting up speed test...
Tests ready starting dry run...
Changing data...
Repeating test...
Dry Run Output

    5 out of 5 pallets ran successfully in 1.91 minutes.
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPallet (54.64 seconds)
                           AddressPoints - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPalletNoReproject (39.39 seconds)
                  AddressPointsNoProject - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:ShapefilePallet (5994 ms)
                   CountiesFromShapefile - Created table successfully.
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:SmallDataPallet (5842 ms)
                                Counties - Created table successfully.
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:TablePallet (6420 ms)
                              SchoolInfo - Created table successfully.

Repeat Run Output

    5 out of 5 pallets ran successfully in 72.9 seconds.
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPallet (8526 ms)
                           AddressPoints - Warning generated during update. Data not modified.
crate message: duplicate features detected!
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPalletNoReproject (52.44 seconds)
                  AddressPointsNoProject - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:ShapefilePallet (3271 ms)
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:SmallDataPallet (4029 ms)
w:\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:TablePallet (3928 ms)

Speed Test Results
Dry Run: 1.91 minutes
Repeat: 72.91 seconds

[stdavis]

Just realized that I need to take into account tables still.

[stdavis]

Maybe I'll wait until tomorrow to merge this since I'm taking the rest of the week off. ;)

[stdavis]

/remind me to merge this next Monday

[github-actions[bot]]

@stdavis set a reminder for 8/16/2021

[steveoh]

you'll still need to install the new bits for the stuff to take hold.

[stdavis]

Hmm. No reminder was posted. :(

I remembered today. I did a fresh git pull and pip install on the server. We'll see how it goes.

[stdavis]

/remind me to check on global ids next week

[github-actions[bot]]

@stdavis set a reminder for 8/24/2021