ngageoint / sarpy

A basic Python library to demonstrate reading, writing, display, and simple processing of complex SAR data using the NGA SICD standard.
MIT License
262 stars 87 forks source link

Bugfix: DTED parsing of null values; accommodate numpy 2.0 #535

Closed pressler-vsc closed 3 months ago

pressler-vsc commented 3 months ago

Description

While investigating a reported issue with SARPy's DTED reading and numpy 2.0, we found that the DTED parsing was incorrect, most notably for null values. From MIL-PRF-89020B:

image

image

The current logic:

example failures running current code with new test ``` SARPY_TEST_PATH=/data/sarpy_test/ nox -- tests/io/DEM/test_dted.py::test_dted_reader nox > Running session test(version='3.6') nox > Creating conda env in .nox/test-version-3-6 with python nox > conda install --yes --prefix /home/vscuser/git/glweb/software/third-party/sarpy/.nox/test-version-3-6 python=3.6 nox > python -m pip install '.[all]' nox > pytest tests/io/DEM/test_dted.py::test_dted_reader =========================================================================== test session starts ============================================================================ platform linux -- Python 3.6.13, pytest-7.0.1, pluggy-1.0.0 rootdir: /home/vscuser/git/glweb/software/third-party/sarpy collected 1 item tests/io/DEM/test_dted.py F [100%] ================================================================================= FAILURES ================================================================================= _____________________________________________________________________________ test_dted_reader _____________________________________________________________________________ @pytest.mark.skipif(not test_data["dted_with_null"], reason="DTED with null data does not exist") def test_dted_reader(): dted_reader = sarpy_dted.DTEDReader(test_data["dted_with_null"][0]) # From entity ID: SRTM3S04W061V1, date updated: 2013-04-17T12:16:47-05 # Acquired from https://earthexplorer.usgs.gov/ on 2024-08-21 known_values = { (1000, 800): -32767, # null (1000, 799): 7, (3, 841): -5, } for index, expected_value in known_values.items(): > assert dted_reader[index] == expected_value E assert array(-1, dtype=int16) == -32767 tests/io/DEM/test_dted.py:37: AssertionError ========================================================================= short test summary info ========================================================================== FAILED tests/io/DEM/test_dted.py::test_dted_reader - assert array(-1, dtype=int16) == -32767 ============================================================================ 1 failed in 0.21s ============================================================================= nox > Command pytest tests/io/DEM/test_dted.py::test_dted_reader failed with exit code 1 nox > Session test(version='3.6') failed. nox > Running session test(version='3.11') nox > Creating conda env in .nox/test-version-3-11 with python nox > conda install --yes --prefix /home/vscuser/git/glweb/software/third-party/sarpy/.nox/test-version-3-11 python=3.11 nox > python -m pip install '.[all]' nox > pytest tests/io/DEM/test_dted.py::test_dted_reader =========================================================================== test session starts ============================================================================ platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 rootdir: /home/vscuser/git/glweb/software/third-party/sarpy collected 1 item tests/io/DEM/test_dted.py F [100%] ================================================================================= FAILURES ================================================================================= _____________________________________________________________________________ test_dted_reader _____________________________________________________________________________ @pytest.mark.skipif(not test_data["dted_with_null"], reason="DTED with null data does not exist") def test_dted_reader(): dted_reader = sarpy_dted.DTEDReader(test_data["dted_with_null"][0]) # From entity ID: SRTM3S04W061V1, date updated: 2013-04-17T12:16:47-05 # Acquired from https://earthexplorer.usgs.gov/ on 2024-08-21 known_values = { (1000, 800): -32767, # null (1000, 799): 7, (3, 841): -5, } for index, expected_value in known_values.items(): > assert dted_reader[index] == expected_value tests/io/DEM/test_dted.py:37: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sarpy/io/DEM/DTED.py:351: in __getitem__ return self._repair_values(data) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ elevations = array(-1, dtype=int16) @staticmethod def _repair_values(elevations): """ This is a helper method for repairing the weird entries in a DTED. The array is modified in place. Parameters ---------- elevations : numpy.ndarray Returns ------- numpy.ndarray """ elevations = numpy.copy(elevations) # BASED ON MIL-PRF-89020B SECTION 3.11.1, 3.11.2 # There are some byte-swapping details that are poorly explained. # The following steps appear to correct for the "complemented" values. # Find negative voids and repair them neg_voids = (elevations < -15000) > elevations[neg_voids] = numpy.abs(elevations[neg_voids]) - 32768 E OverflowError: Python integer 32768 out of bounds for int16 sarpy/io/DEM/DTED.py:374: OverflowError ========================================================================= short test summary info ========================================================================== FAILED tests/io/DEM/test_dted.py::test_dted_reader - OverflowError: Python integer 32768 out of bounds for int16 ============================================================================ 1 failed in 0.20s ============================================================================= nox > Command pytest tests/io/DEM/test_dted.py::test_dted_reader failed with exit code 1 nox > Session test(version='3.11') failed. nox > Ran multiple sessions: nox > * test(version='3.6'): failed nox > * test(version='3.11'): failed ```

Notes

Test Results

tests pass ``` SARPY_TEST_PATH=/data/sarpy_test/ nox nox > Running session test(version='3.6') nox > Creating conda env in .nox/test-version-3-6 with python nox > conda install --yes --prefix /home/vscuser/git/glweb/software/third-party/sarpy/.nox/test-version-3-6 python=3.6 nox > python -m pip install '.[all]' nox > pytest tests =========================================================================== test session starts ============================================================================ platform linux -- Python 3.6.13, pytest-7.0.1, pluggy-1.0.0 rootdir: /home/vscuser/git/glweb/software/third-party/sarpy collected 563 items tests/test_class_string.py . [ 0%] tests/consistency/test_consistency.py ........ [ 1%] tests/consistency/test_cphd_consistency.py .............................................................. [ 12%] tests/consistency/test_sicd_consistency.py ... [ 13%] tests/consistency/test_sidd_consistency.py .... [ 13%] tests/geometry/test_geocoords.py .......... [ 15%] tests/geometry/test_geometry_elements.py .......... [ 17%] tests/geometry/test_latlon.py ... [ 17%] tests/geometry/test_point_projection.py .............. [ 20%] tests/io/test_kml.py . [ 20%] tests/io/DEM/test_dted.py .. [ 20%] tests/io/DEM/test_geoid.py . [ 21%] tests/io/DEM/test_geotiff1deg_list.py .... [ 21%] tests/io/DEM/test_geotiff1deg_reader.py ........ [ 23%] tests/io/complex/test_other_nitf.py ... [ 23%] tests/io/complex/test_reader.py ..sss.ssss. [ 25%] tests/io/complex/test_remote.py s [ 25%] tests/io/complex/test_sicd.py . [ 26%] tests/io/complex/test_utils.py . [ 26%] tests/io/complex/sicd_elements/test_sicd_elements_antenna.py ... [ 26%] tests/io/complex/sicd_elements/test_sicd_elements_base.py ... [ 27%] tests/io/complex/sicd_elements/test_sicd_elements_blocks.py ..................... [ 31%] tests/io/complex/sicd_elements/test_sicd_elements_collectioninfo.py .. [ 31%] tests/io/complex/sicd_elements/test_sicd_elements_errorstatistics.py . [ 31%] tests/io/complex/sicd_elements/test_sicd_elements_geodata.py ... [ 32%] tests/io/complex/sicd_elements/test_sicd_elements_grid.py .. [ 32%] tests/io/complex/sicd_elements/test_sicd_elements_imagecreation.py . [ 32%] tests/io/complex/sicd_elements/test_sicd_elements_imagedata.py . [ 32%] tests/io/complex/sicd_elements/test_sicd_elements_imageformation.py ...... [ 33%] tests/io/complex/sicd_elements/test_sicd_elements_matchinfo.py . [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_pfa.py . [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_position.py ... [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_radarcollection.py .............. [ 37%] tests/io/complex/sicd_elements/test_sicd_elements_radiometric.py . [ 37%] tests/io/complex/sicd_elements/test_sicd_elements_rgazcomp.py . [ 37%] tests/io/complex/sicd_elements/test_sicd_elements_rma.py ... [ 38%] tests/io/complex/sicd_elements/test_sicd_elements_scpcoa.py ................... [ 41%] tests/io/complex/sicd_elements/test_sicd_elements_sicd.py .................................. [ 47%] tests/io/complex/sicd_elements/test_sicd_elements_timeline.py .................. [ 50%] tests/io/complex/sicd_elements/test_sicd_elements_utils.py ... [ 51%] tests/io/general/test_base.py ... [ 51%] tests/io/general/test_data_segment.py .......... [ 53%] tests/io/general/test_format_function.py ........ [ 55%] tests/io/general/test_nitf.py .. [ 55%] tests/io/general/test_nitf_headers.py . [ 55%] tests/io/general/test_nitf_image.py .... [ 56%] tests/io/general/test_tre.py ... [ 56%] tests/io/phase_history/test_cphd.py ....... [ 58%] tests/io/phase_history/test_cphd_versions.py ..... [ 58%] tests/io/phase_history/cphd1_elements/test_cphd.py ...... [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_antenna.py ..... [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_blocks.py .... [ 61%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_channel.py . [ 61%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_data.py . [ 61%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_dwell.py . [ 62%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_geoinfo.py ... [ 62%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_global.py .. [ 63%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_pvp.py . [ 63%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_referencegeometry.py . [ 63%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_supportarray.py .. [ 63%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_txrcv.py . [ 63%] tests/io/phase_history/cphd1_elements/test_utils.py . [ 64%] tests/io/product/test_reader.py ...s [ 64%] tests/io/product/test_sidd.py ... [ 65%] tests/io/product/test_sidd_schema.py ......... [ 66%] tests/io/product/test_sidd_writing.py .. [ 67%] tests/io/product/sidd3_elements/test_exploitationfeatures.py ....................................................................................................... [ 85%] ........................... [ 90%] tests/io/product/sidd3_elements/test_sidd3_elements.py ..... [ 91%] tests/io/received/test_crsd.py . [ 91%] tests/processing/sicd/test_spectral_taper.py ........ss........... [ 95%] tests/utils/test_sicd_sidelobe_control.py .. [ 95%] tests/visualization/test_cphd_kmz_product_creation.py .. [ 95%] tests/visualization/test_crsd_kmz_product_creation.py . [ 96%] tests/visualization/test_kmz_product_creation.py .x [ 96%] tests/visualization/test_remap.py .................... [100%] ============================================================================= warnings summary ============================================================================= tests/geometry/test_point_projection.py::test_image_to_ground_errors tests/geometry/test_point_projection.py::test_image_to_ground_dem /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/geometry/point_projection.py:1882: RuntimeWarning: divide by zero encountered in double_scalars lon_grid_size = min(10, lat_grid_size/abs(numpy.sin(numpy.deg2rad(ref_llh[0])))) tests/visualization/test_kmz_product_creation.py::test_create_kmz[True] /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/visualization/kmz_utils.py:169: RuntimeWarning: invalid value encountered in double_scalars + dir_z**2 * semiaxis_a**2 * semiaxis_b**2 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================================= 551 passed, 11 skipped, 1 xfailed, 3 warnings in 48.31s ========================================================== nox > Session test(version='3.6') was successful. nox > Running session test(version='3.11') nox > Creating conda env in .nox/test-version-3-11 with python nox > conda install --yes --prefix /home/vscuser/git/glweb/software/third-party/sarpy/.nox/test-version-3-11 python=3.11 nox > python -m pip install '.[all]' nox > pytest tests =========================================================================== test session starts ============================================================================ platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0 rootdir: /home/vscuser/git/glweb/software/third-party/sarpy collected 563 items tests/consistency/test_consistency.py ........ [ 1%] tests/consistency/test_cphd_consistency.py .............................................................. [ 12%] tests/consistency/test_sicd_consistency.py ... [ 12%] tests/consistency/test_sidd_consistency.py .... [ 13%] tests/geometry/test_geocoords.py .......... [ 15%] tests/geometry/test_geometry_elements.py .......... [ 17%] tests/geometry/test_latlon.py ... [ 17%] tests/geometry/test_point_projection.py .............. [ 20%] tests/io/DEM/test_dted.py .. [ 20%] tests/io/DEM/test_geoid.py . [ 20%] tests/io/DEM/test_geotiff1deg_list.py .... [ 21%] tests/io/DEM/test_geotiff1deg_reader.py ........ [ 22%] tests/io/complex/sicd_elements/test_sicd_elements_antenna.py ... [ 23%] tests/io/complex/sicd_elements/test_sicd_elements_base.py ... [ 23%] tests/io/complex/sicd_elements/test_sicd_elements_blocks.py ..................... [ 27%] tests/io/complex/sicd_elements/test_sicd_elements_collectioninfo.py .. [ 28%] tests/io/complex/sicd_elements/test_sicd_elements_errorstatistics.py . [ 28%] tests/io/complex/sicd_elements/test_sicd_elements_geodata.py ... [ 28%] tests/io/complex/sicd_elements/test_sicd_elements_grid.py .. [ 29%] tests/io/complex/sicd_elements/test_sicd_elements_imagecreation.py . [ 29%] tests/io/complex/sicd_elements/test_sicd_elements_imagedata.py . [ 29%] tests/io/complex/sicd_elements/test_sicd_elements_imageformation.py ...... [ 30%] tests/io/complex/sicd_elements/test_sicd_elements_matchinfo.py . [ 30%] tests/io/complex/sicd_elements/test_sicd_elements_pfa.py . [ 30%] tests/io/complex/sicd_elements/test_sicd_elements_position.py ... [ 31%] tests/io/complex/sicd_elements/test_sicd_elements_radarcollection.py .............. [ 33%] tests/io/complex/sicd_elements/test_sicd_elements_radiometric.py . [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_rgazcomp.py . [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_rma.py ... [ 34%] tests/io/complex/sicd_elements/test_sicd_elements_scpcoa.py ................... [ 38%] tests/io/complex/sicd_elements/test_sicd_elements_sicd.py .................................. [ 44%] tests/io/complex/sicd_elements/test_sicd_elements_timeline.py .................. [ 47%] tests/io/complex/sicd_elements/test_sicd_elements_utils.py ... [ 47%] tests/io/complex/test_other_nitf.py ... [ 48%] tests/io/complex/test_reader.py ..sss.ssss. [ 50%] tests/io/complex/test_remote.py s [ 50%] tests/io/complex/test_sicd.py . [ 50%] tests/io/complex/test_utils.py . [ 50%] tests/io/general/test_base.py ... [ 51%] tests/io/general/test_data_segment.py .......... [ 53%] tests/io/general/test_format_function.py ........ [ 54%] tests/io/general/test_nitf.py .. [ 55%] tests/io/general/test_nitf_headers.py . [ 55%] tests/io/general/test_nitf_image.py .... [ 55%] tests/io/general/test_tre.py ... [ 56%] tests/io/phase_history/cphd1_elements/test_cphd.py ...... [ 57%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_antenna.py ..... [ 58%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_blocks.py .... [ 59%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_channel.py . [ 59%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_data.py . [ 59%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_dwell.py . [ 59%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_geoinfo.py ... [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_global.py .. [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_pvp.py . [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_referencegeometry.py . [ 60%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_supportarray.py .. [ 61%] tests/io/phase_history/cphd1_elements/test_cphd1_elements_txrcv.py . [ 61%] tests/io/phase_history/cphd1_elements/test_utils.py . [ 61%] tests/io/phase_history/test_cphd.py ....... [ 62%] tests/io/phase_history/test_cphd_versions.py ..... [ 63%] tests/io/product/sidd3_elements/test_exploitationfeatures.py ....................................................................................................... [ 82%] ........................... [ 86%] tests/io/product/sidd3_elements/test_sidd3_elements.py ..... [ 87%] tests/io/product/test_reader.py ...s [ 88%] tests/io/product/test_sidd.py ... [ 88%] tests/io/product/test_sidd_schema.py ......... [ 90%] tests/io/product/test_sidd_writing.py .. [ 90%] tests/io/received/test_crsd.py . [ 91%] tests/io/test_kml.py . [ 91%] tests/processing/sicd/test_spectral_taper.py ..................... [ 95%] tests/test_class_string.py . [ 95%] tests/utils/test_sicd_sidelobe_control.py .. [ 95%] tests/visualization/test_cphd_kmz_product_creation.py .. [ 95%] tests/visualization/test_crsd_kmz_product_creation.py . [ 96%] tests/visualization/test_kmz_product_creation.py .x [ 96%] tests/visualization/test_remap.py .................... [100%] ============================================================================= warnings summary ============================================================================= tests/geometry/test_geometry_elements.py: 77 warnings /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/geometry/geometry_elements.py:131: DeprecationWarning: Arrays of 2-dimensional vectors are deprecated. Use arrays of 3-dimensional vectors instead. (deprecated in NumPy 2.0) dir_cross = float(numpy.cross(R, S)) # the scalar cross product of the direction vectors tests/geometry/test_geometry_elements.py: 43 warnings /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/geometry/geometry_elements.py:137: DeprecationWarning: Arrays of 2-dimensional vectors are deprecated. Use arrays of 3-dimensional vectors instead. (deprecated in NumPy 2.0) end_cross_0 = float(numpy.cross(Q-P, S)) tests/geometry/test_geometry_elements.py: 43 warnings /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/geometry/geometry_elements.py:138: DeprecationWarning: Arrays of 2-dimensional vectors are deprecated. Use arrays of 3-dimensional vectors instead. (deprecated in NumPy 2.0) end_cross_1 = float(numpy.cross(Q-P, R)) tests/geometry/test_point_projection.py::test_image_to_ground_errors tests/geometry/test_point_projection.py::test_image_to_ground_dem /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/geometry/point_projection.py:1882: RuntimeWarning: divide by zero encountered in scalar divide lon_grid_size = min(10, lat_grid_size/abs(numpy.sin(numpy.deg2rad(ref_llh[0])))) tests/test_class_string.py::TestClassString::test_class_str /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/annotation/afrl_rde_schema/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https:// setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources tests/visualization/test_kmz_product_creation.py::test_create_kmz[True] /home/vscuser/git/glweb/software/third-party/sarpy/sarpy/visualization/kmz_utils.py:145: RuntimeWarning: invalid value encountered in scalar divide distance = ( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================================= 553 passed, 9 skipped, 1 xfailed, 167 warnings in 31.10s ========================================================= nox > Session test(version='3.11') was successful. nox > Ran multiple sessions: nox > * test(version='3.6'): success nox > * test(version='3.11'): success ```