search

agrc / forklift

:tractor::package::sparkles: Slinging data all over the place :tractor::package::sparkles:
MIT License
27 stars 3 forks source link

build-optimizations #330

Closed stdavis closed 4 years ago

stdavis commented 4 years ago

Description of Changes

  1. Optimize build pallets by caching source describes globally and removing unnecessary calls to arcpy.Exists.
    • Added a new cli command build which builds all (or a single file) pallets without processing any data. This will be useful in testing/profiling builds and testing that the conda environment has all required dependencies.

From my initial tests, this cut the build time on 57 from 1.98 hours down to 1.18 hours. image

image

  1. Upgraded all tests in test_core.py to use the test_gdb fixture. I have to say, it was pretty slick. I wonder if it's time to push test_gdb to its own repo and publish it to pypi.
  2. Adds a check for a Pro license and sends an email before exiting the process. (#312) image image

Other niceties

Test results and coverage

========================================= test session starts ==========================================
platform win32 -- Python 3.6.9, pytest-4.6.5, py-1.8.0, pluggy-0.12.0
rootdir: W:\forklift, inifile: pytest.ini, testpaths: tests
plugins: cov-2.7.1, instafail-0.4.1.post0, isort-0.3.1, pylint-0.14.1
collected 211 items                                                                                     
-----------------------------------------------------------------
Linting files
.
-----------------------------------------------------------------

tests\__init__.py ss                                                                           [  2/211]
tests\PalletWithSyntaxErrors.py ss                                                             [  4/211]
tests\SchemaLockPallet.py ss                                                                   [  6/211]
tests\benchmark_arcgis.py ss                                                                   [  8/211]
tests\conftest.py ss                                                                           [ 10/211]
tests\mocks.py ss                                                                              [ 12/211]
tests\test_arcgis.py ss.................                                                       [ 31/211]
tests\test_change_detection.py ss.......                                                       [ 40/211]
tests\test_changes.py ss......                                                                 [ 48/211]
tests\test_config.py ss......                                                                  [ 56/211]
tests\test_core.py ss...................................                                       [ 93/211]
tests\test_crate.py .....................                                                      [114/211]
tests\test_engine.py ss......................s..........                                       [149/211]
tests\test_lift.py ss.........                                                                 [160/211]
tests\test_messaging.py ss.....                                                                [167/211]
tests\test_pallet.py ss................................                                        [201/211]
tests\test_seat.py ss....                                                                      [207/211]
tests\test_slack.py ss..                                                                       [211/211]

----------- coverage: platform win32, python 3.6.9-final-0 -----------
Name                               Stmts   Miss Branch BrPart     Cover   Missing
---------------------------------------------------------------------------------
src\forklift\arcgis.py               113     29     32      2    70.34%   26, 79-107, 137-138, 143, 146,
163, 178->181, 181, 197->198, 198-200, 222-224
src\forklift\change_detection.py      56      5     12      2    89.71%   31-33, 50->51, 51, 53->54, 54
src\forklift\config.py                50      1     20      2    95.71%   82->83, 83, 118->117
src\forklift\core.py                 197      7     84      4    94.66%   96->97, 97-100, 129->130, 130,
153-154, 301->300, 429->430, 430
src\forklift\engine.py               477    171    160     12    61.54%   133->136, 136, 155-156, 212, 224->225, 225, 237->238, 238-318, 322->368, 332->333, 333-335, 358-359, 397-468, 498, 508, 522-538, 543-556, 571->572, 572, 592, 610->611, 611, 624-627, 678->681, 681-690, 718->721, 721-722, 724->727, 727, 741-747, 753, 757->760, 760, 805->806, 806, 911-954
src\forklift\lift.py                 148     52     52      4    61.00%   34, 41->42, 42, 152->153, 153-156, 159->163, 163-164, 187-198, 233-254, 281, 284->285, 285, 292-295, 310-317, 346-357
src\forklift\messaging.py             53     10     16      1    72.46%   58, 78->79, 79, 100-112
src\forklift\models.py               194      7     54      3    95.16%   93, 180->179, 330-332, 397->400, 400, 442->450, 450-451
src\forklift\slack.py                211     33    102     33    75.72%   24->25, 25-27, 54->57, 57, 71->72, 72-77, 79->80, 80-85, 90->93, 100->101, 101, 102->103, 103, 104->105, 105, 111->114, 116->117, 117-118, 120->87, 134->137, 137, 151->184, 155->158, 160->161, 161-164, 165->168, 168->171, 174->177, 189->192, 195->197, 197->200, 202->205, 236, 259->260, 260, 265, 276->278, 278->279, 279, 284->287, 287->288, 288, 304->305, 305, 331->332, 332, 333->334, 334, 347->348, 348, 351->354, 354->357, 360, 372, 375
---------------------------------------------------------------------------------
TOTAL                               1521    315    538     63    75.81%

3 files skipped due to complete coverage.

=============================== 176 passed, 35 skipped in 320.12 seconds ===============================

Speed test results

image

steveoh commented 4 years ago

This looks awesome 🎉. I'll dive in now for the review.