GeoscienceAustralia / wagl

Python library for standardising satellite imagery into an Analysis Ready Data (ARD) form
Apache License 2.0
30 stars 7 forks source link

hp5y string deserialization #354

Closed sixy6e closed 3 years ago

sixy6e commented 3 years ago

For h5py > 3.0.0, string behaviour changed (see here causing string arrays to be loaded as byte strings, eg

array([b'period 0', b'period 1', b'period 2', b'period 3', b'period 4',
       b'period 5', b'period 6', b'period 7', b'period 8', b'period 9'],
      dtype=object)

This causes an issue for string comparison such as here during interpolation of the atmospheric coefficients, resulting in a zero-length array of point samples used for driving the interpolation.

The existing test, does pick this up.

ga-ellis commented 3 years ago

@sixy6e Will these updates still work with 2.x?

sixy6e commented 3 years ago

it should as it is being explicit on casting to NumPy's Unicode (U) type, which has been around for a while.

sixy6e commented 3 years ago

When NCI comes back online I can run the test there. Or get @truth-quark to do it.

sixy6e commented 3 years ago

@ga-ellis

Results from a build I just created on NCI:

collected 340 items

tests/test_acquisition.py ......................................................................................................................................................................         [ 48%]
tests/test_acquisition_sentinel.py .......................                                                                                                                                               [ 55%]
tests/test_blrb.py .........                                                                                                                                                                             [ 58%]
tests/test_geobox.py .............................                                                                                                                                                       [ 66%]
tests/test_hdf5.py .........................                                                                                                                                                             [ 74%]
tests/test_interpolation.py ......                                                                                                                                                                       [ 75%]
tests/test_land_sea_masking.py ..                                                                                                                                                                        [ 76%]
tests/test_lon_lat.py ..                                                                                                                                                                                 [ 77%]
tests/test_ls7_gap_mask.py ..................                                                                                                                                                            [ 82%]
tests/test_metadata.py ...                                                                                                                                                                               [ 83%]
tests/test_mtl.py ....................                                                                                                                                                                   [ 89%]
tests/test_random_locations.py ..                                                                                                                                                                        [ 89%]
tests/test_read_subset.py ssss..........s                                                                                                                                                                [ 94%]
tests/test_reformat_tp5.py ..                                                                                                                                                                            [ 94%]
tests/test_tiling.py .....                                                                                                                                                                               [ 96%]
tests/test_tle.py ...                                                                                                                                                                                    [ 97%]
tests/test_tp5.py .........                                                                                                                                                                              [ 99%]
tests/test_vincenty.py .                                                                                                                                                                                 [100%]

=============================================================================================== warnings summary ===============================================================================================
/g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/osgeo/gdal.py:107
  /g/data/v10/public/modules/dea-env/20200612/lib/python3.6/site-packages/osgeo/gdal.py:107: DeprecationWarning:

  gdal.py was placed in a namespace, it is now available as osgeo.gdal

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================ 335 passed, 5 skipped, 1 warning in 130.92s (0:02:10) =============================================================================
uchchwhash commented 3 years ago

Thanks for all the help @sixy6e! You are awesome!