search

agrc / forklift

:tractor::package::sparkles: Slinging data all over the place :tractor::package::sparkles:
MIT License
28 stars 3 forks source link

Preserve Global ID values #362

Closed stdavis closed 9 months ago

stdavis commented 9 months ago

Description of Changes

This pull requests preserves global id values for crates that do not participate in change detection. It does this by deleting the destination table and recreating it any time there are changes. It should only do this if the dataset has a managed global id field.

Test results and coverage

tests\test_arcgis.py .................                                                                                                 [ 17/182]
tests\test_change_detection.py ..........                                                                                              [ 27/182]
tests\test_changes.py ......                                                                                                           [ 33/182]
tests\test_config.py .......                                                                                                           [ 40/182]
tests\test_core.py ......................................                                                                              [ 78/182]
tests\test_crate.py ...................                                                                                                [ 97/182]
tests\test_engine.py ......................s..........                                                                                 [130/182]
tests\test_lift.py .........                                                                                                           [139/182]
tests\test_messaging.py .....                                                                                                          [144/182]
tests\test_pallet.py ................................                                                                                  [176/182]
tests\test_seat.py ....                                                                                                                [180/182]
tests\test_slack.py ..                                                                                                                 [182/182]

================================================== 181 passed, 1 skipped in 270.35s (0:04:30) ==================================================

Speed test results

(forklift) λ forklift speedtest
Setting up speed test...
Tests ready starting dry run...
Changing data...
Repeating test...
Dry Run Output

    5 out of 5 pallets ran successfully in 2.55 minutes.
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPallet (69.46 seconds)
                           AddressPoints - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPalletNoReproject (35.8 seconds)
                  AddressPointsNoProject - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:ShapefilePallet (10837 ms)
                   CountiesFromShapefile - Created table successfully.
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:SmallDataPallet (20276 ms)
                                Counties - Created table successfully.
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:TablePallet (13838 ms)
                              SchoolInfo - Created table successfully.

Repeat Run Output

    5 out of 5 pallets ran successfully in 2.86 minutes.
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPallet (22274 ms)
                           AddressPoints - Warning generated during update. Data not modified.
crate message: duplicate features detected!
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:LargeDataPalletNoReproject (2.04 minutes)
                  AddressPointsNoProject - Warning generated during update and data updated successfully.
crate message: duplicate features detected!
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:ShapefilePallet (3122 ms)
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:SmallDataPallet (12363 ms)
C:\projects\forklift\src\forklift\..\..\speedtest\SpeedTestPallet.py:TablePallet (9492 ms)

Speed Test Results
Dry Run: 2.55 minutes
Repeat: 2.86 minutes