search

Esri / military-tools-geoprocessing-toolbox

military-tools-geoprocessing-toolbox is a collection of models, scripts, and tools for use in ArcGIS for Desktop and ArcGIS Pro. This toolbox is one component that is a part of Military Tools.
Apache License 2.0
33 stars 14 forks source link

Test framework is not catching errors in the tests #66

Closed mfunk closed 8 years ago

mfunk commented 8 years ago

In testing the PRs for the test building in the repo I'm finding that the unittest lib doesn't seem to be capturing any of the arcpy errors as errors (or failures for that matter). For instance in the Desktop tests, specifically test_lowest_points_desktop we get Error messages, but they are not captured in the test results:

Python 2.7 Tests ===============================
Debug messaging is OFF
Logging results to: MTGT_2016-May-18_13-24-55_seq0.log
adding test: test_convert_coordinates_desktop
adding test: test_table_to_twopointline_desktop
adding test: test_table_to_ellipse_desktop
adding test: test_table_to_lineofbearing_desktop
adding test: test_table_to_point_desktop
adding test: test_table_to_polygon_desktop
adding test: test_table_to_polyline_desktop
adding test: test_find_local_peaks_desktop
adding test: test_lowest_points_desktop
running 9 tests...
ArcPy works
Testing Convert Coordinates (Desktop).
ArcPy works
Testing Table To 2-Point Line (Desktop).
...
...
...
Testing Lowest Points (Desktop).
Running tool (Lowest Points)
ERROR: Executing: LowestPoints C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\AreaofInterest C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\ElevationUTM_Zone10 C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\outputLowestPoints
Start Time: Wed May 18 13:26:12 2016
Executing (Copy Features): CopyFeatures C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\AreaofInterest C:\Users\<user>\AppData\Local\Temp\\copyArea # 0 0 0
Start Time: Wed May 18 13:26:14 2016
ERROR 000210: Cannot create output C:\Users\<user>\AppData\Local\Temp\\copyArea
Failed to execute (Copy Features).
Failed at Wed May 18 13:26:14 2016 (Elapsed Time: 0.27 seconds)
Failed to execute (LowestPoints).
Failed at Wed May 18 13:26:14 2016 (Elapsed Time: 1.96 seconds)
Executing: LowestPoints C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\AreaofInterest C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\ElevationUTM_Zone10 C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\outputLowestPoints
Start Time: Wed May 18 13:26:12 2016
Executing (Copy Features): CopyFeatures C:\Users\<user>\Documents\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\AreaofInterest C:\Users\<user>\AppData\Local\Temp\\copyArea # 0 0 0
Start Time: Wed May 18 13:26:14 2016
ERROR 000210: Cannot create output C:\Users\<user>\AppData\Local\Temp\\copyArea
Failed to execute (Copy Features).
Failed at Wed May 18 13:26:14 2016 (Elapsed Time: 0.27 seconds)
Failed to execute (LowestPoints).
Failed at Wed May 18 13:26:14 2016 (Elapsed Time: 1.96 seconds)
Test success: True
RESULTS =================================================

Number of tests run: 9
Number of errors: 0
Number of failures: 0

END OF TEST =========================================

The issue isn't that the test is failing ... that is another issue. What we have wrong here is that the tests are not reporting these problems. If we only look at the results...

RESULTS =================================================

Number of tests run: 9
Number of errors: 0
Number of failures: 0

END OF TEST =========================================

... we would assume that all is OK. But it is not. I think we are missing something here. Not sure what or where. We might need another set of eyes to look at this one.

mfunk commented 8 years ago

@topowright I think we need a separate set of eyes on this specific issue. I think @JudTown17 and I are too close to it.

mfunk commented 8 years ago

Just thinking online, but perhaps we need to have a check for ArcPy exceptions and raise any errors as unittest asserts.

mfunk commented 8 years ago

Right now in the Pro tests the Radial Line of Sight test (test_radial_line_of_sight) is not trapping the errors in the tool (the tool is having an internal error from issue #83). I'm trying to get the test to fail (using unittest.TestCase.fail()) in the test's exception handler:

            try:
                arcpy.RadialLineOfSight_mt(self.observers, self.inputSurface, self.outputRLOS)
                featureCount = int(arcpy.GetCount_management(self.outputRLOS).getOutput(0))
            except arcpy.ExecuteError:
                self.fail("test_radial_line_of_sight failed.")
                UnitTestUtilities.handleArcPyError()

However whenever I add .fail(), or any variation, then the unittest framework fails:

.....Testing Radial Line Of Sight (Pro).
.....RadialLineOfSightTestCase.test_Radial_line_of_sight
RLOS will use local RADIUS2 values for calculation.
Finding observer's minimum bounding envelope ...
Finding center of observers ...
Recentering Azimuthal Equidistant to centroid ...
Projecting surface ...
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
Traceback (most recent call last):
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\tokenize.py", line 392, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "TestRunner.py", line 150, in <module>
    main()
  File "TestRunner.py", line 67, in main
    result = runTestSuite()
  File "TestRunner.py", line 130, in runTestSuite
    testSuite.run(result)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\suite.py", line 122, in run
    test(result)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\case.py", line 628, in __call__
    return self.run(*args, **kwds)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\case.py", line 588, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\case.py", line 516, in _feedErrorsToResult
    result.addFailure(test, exc_info)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\result.py", line 17, in inner
    return method(self, *args, **kw)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\result.py", line 121, in addFailure
    self.failures.append((test, self._exc_info_to_string(err, test)))
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\unittest\result.py", line 182, in _exc_info_to_string
    msgLines = traceback.format_exception(exctype, value, tb, length)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\traceback.py", line 181, in format_exception
    return list(_format_exception_iter(etype, value, tb, limit, chain))
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\traceback.py", line 153, in _format_exception_iter
    yield from _format_list_iter(_extract_tb_iter(tb, limit=limit))
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\traceback.py", line 18, in _format_list_iter
    for filename, lineno, name, line in extracted_list:
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\traceback.py", line 65, in _extract_tb_or_stack_iter
    line = linecache.getline(filename, lineno, f.f_globals)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\linecache.py", line 15, in getline
    lines = getlines(filename, module_globals)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\linecache.py", line 42, in getlines
    return updatecache(filename, module_globals)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\linecache.py", line 130, in updatecache
    with tokenize.open(fullname) as fp:
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\tokenize.py", line 456, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\tokenize.py", line 433, in detect_encoding
    encoding = find_cookie(first)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\tokenize.py", line 397, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'D:\\GitHub\\military-tools-geoprocessing-toolbox\\toolboxes\\Military_Tools.tbx'

This is actually two problems: the initial error, and an error in handling the first error.... and i'm stuck.

mfunk commented 8 years ago

To back up a bit. If leave the error handling to what it used to be this is what happens in the Pro tests for _test_radial_line_ofsight.

.....Testing Radial Line Of Sight (Pro).
.....RadialLineOfSightTestCase.test_Radial_line_of_sight
RLOS will use local RADIUS2 values for calculation.
Finding observer's minimum bounding envelope ...
Finding center of observers ...
Recentering Azimuthal Equidistant to centroid ...
Projecting surface ...
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
ERROR: Start Time: Thursday, May 26, 2016 8:46:45 AM
Executing (Copy Features): CopyFeatures D:\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\Observers D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\copyObs # 0 0 0
Start Time: Thursday, May 26, 2016 8:46:46 AM
Succeeded at Thursday, May 26, 2016 8:46:47 AM (Elapsed Time: 0.81 seconds)
Executing (sourceRLOSscript): sourceRLOSscript D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\copyObs D:\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\ElevationUTM_Zone10 D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb/OutputVisibility false GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]]
Start Time: Thursday, May 26, 2016 8:46:47 AM
Running script sourceRLOSscript...
RLOS will use local RADIUS2 values for calculation.
Finding observer's minimum bounding envelope ...
Finding center of observers ...
Recentering Azimuthal Equidistant to centroid ...
Projecting surface ...
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
Completed script sourceRLOSscript...
Failed to execute (sourceRLOSscript).
Failed at Thursday, May 26, 2016 8:46:55 AM (Elapsed Time: 8.04 seconds)
Failed to execute (RadialLineOfSight).
Failed at Thursday, May 26, 2016 8:46:55 AM (Elapsed Time: 10.16 seconds)
Start Time: Thursday, May 26, 2016 8:46:45 AM
Executing (Copy Features): CopyFeatures D:\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\Observers D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\copyObs # 0 0 0
Start Time: Thursday, May 26, 2016 8:46:46 AM
Succeeded at Thursday, May 26, 2016 8:46:47 AM (Elapsed Time: 0.81 seconds)
Executing (sourceRLOSscript): sourceRLOSscript D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb\copyObs D:\GitHub\military-tools-geoprocessing-toolbox\testdata\MilitaryToolsTestData.gdb\ElevationUTM_Zone10 D:\GitHub\military-tools-geoprocessing-toolbox\testdata\scratch.gdb/OutputVisibility false GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]]
Start Time: Thursday, May 26, 2016 8:46:47 AM
Running script sourceRLOSscript...
RLOS will use local RADIUS2 values for calculation.
Finding observer's minimum bounding envelope ...
Finding center of observers ...
Recentering Azimuthal Equidistant to centroid ...
Projecting surface ...
ERROR 000622: Failed to execute (Project Raster). Parameters are not valid.
ERROR 000628: Cannot set input into parameter out_coor_system.
Completed script sourceRLOSscript...
Failed to execute (sourceRLOSscript).
Failed at Thursday, May 26, 2016 8:46:55 AM (Elapsed Time: 8.04 seconds)
Failed to execute (RadialLineOfSight).
Failed at Thursday, May 26, 2016 8:46:55 AM (Elapsed Time: 10.16 seconds)
mfunk commented 8 years ago

Think I nailed it. In the test case file we have three methods, one that sets the desktop toolbox, one that sets the pro toolbox, and one that actually does the test:

    def test_radial_line_of_sight_desktop(self):
        arcpy.AddMessage(".....Testing Radial Line Of Sight (Desktop).")
        self.test_radial_line_of_sight(Configuration.military_DesktopToolboxPath)

    def test_radial_line_of_sight_pro(self):
        arcpy.AddMessage(".....Testing Radial Line Of Sight (Pro).")
        self.test_radial_line_of_sight(Configuration.military_ProToolboxPath)

    def test_radial_line_of_sight(self, toolboxPath):
        try:
...
            arcpy.RadialLineOfSight_mt(self.observers, self.inputSurface, self.outputRLOS)
...
        except arcpy.ExecuteError:
            self.fail("test_radial_line_of_sight failed")

So the problem is that we are calling one method from another, and trying to capture the failure from the first method (which is the actual test) in the second. If we move the test into the actual test method, then the failure works:

    def test_radial_line_of_sight_pro(self):
        try:
            arcpy.ImportToolbox(Configuration.military_ProToolboxPath, "mt")
...
            arcpy.RadialLineOfSight_mt(self.observers, self.inputSurface, self.outputRLOS)
...
            except arcpy.ExecuteError:
                self.fail("test_radial_line_of_sight failed")

        except arcpy.ExecuteError:
            self.fail("FAIL: test_radial_line_of_sight_pro:/n" + str(arcpy.GetMessages()))