search

pytroll / fogpy

Satellite based fog and low stratus detection and nowcasting
http://fogpy.readthedocs.io/en/latest
GNU General Public License v3.0
20 stars 10 forks source link

Three unit tests fail with AttributeError #82

Closed gerritholl closed 3 years ago

gerritholl commented 3 years ago

With latest fogpy and satpy master, there are three tests that fail with AttributeError:

$ pytest                                                                                                                                                                                                                                                                                                           [382/1994]
========================================= test session starts ==========================================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/gholl/checkouts/fogpy
plugins: cov-2.11.1, socket-0.4.0
collected 95 items

test_algorithms.py .sss...........F..........                                                    [ 27%]
test_composites.py ........                                                                      [ 35%]
test_filters.py .....s...........s........FF...                                                  [ 68%]
test_lowwatercloud.py ............................x.                                             [100%]

=============================================== FAILURES ===============================================
__________________________ Test_PanSharpeningAlgorithm.test_pansharp_resample __________________________

self = <fogpy.test.test_algorithms.Test_PanSharpeningAlgorithm testMethod=test_pansharp_resample>

    def test_pansharp_resample(self):
        """Test resampling to degraded multispectral channel resolution"""
        # Run pansharpening algorithm
        panshalgo = PanSharpeningAlgorithm(**self.input)
>       ret, mask = panshalgo.run()

test_algorithms.py:652:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../algorithms.py:114: in run
    self.procedure()
../algorithms.py:1174: in procedure
    self.apply_hill_sharpening(chn, panrow, pancol, panshrp_chn)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <fogpy.algorithms.PanSharpeningAlgorithm object at 0x7ff01b608b50>
chn = array([[[ 8.2132086 ],
        [ 9.11465832],
        [ 9.81578588],
        ...,
        [ 8.91433616],
        [ 8.9...3.72206802],
        [12.31981289],
        ...,
        [15.22448423],
        [14.72367882],
        [14.12271234]]])
panrow = array([[  0,   0,   0, ...,   0,   0,   0],
       [  0,   0,   0, ...,   0,   0,   0],
       [  0,   0,   0, ...,   ... 140, 140, 140],
       [140, 140, 140, ..., 140, 140, 140],
       [140, 140, 140, ..., 140, 140, 140]], dtype=uint16)
pancol = array([[  0,   0,   0, ..., 297, 297, 297],
       [  0,   0,   0, ..., 297, 297, 297],
       [  0,   0,   0, ..., 29... 297, 297, 297],
       [  0,   0,   0, ..., 297, 297, 297],
       [  0,   0,   0, ..., 297, 297, 297]], dtype=uint16)
output = array([[4.65102510e-310, 6.95006358e-310, 0.00000000e+000, ...,
        0.00000000e+000, 0.00000000e+000, 0.00000000e+...  [0.00000000e+000, 0.00000000e+000, 0.00000000e+000, ...,
        0.00000000e+000, 0.00000000e+000, 0.00000000e+000]])

    def apply_hill_sharpening(self, chn, panrow, pancol, output):
        """ Local regresssion based pansharpening algorithm by HILL et al.
        1999. The nearest neighbor search is based on the scipy KDtree
        implementation, whereas remapping is done with pyresample methods.
        """
        logger.info("Using local regression approach by Hill")
        # Setup KDtree for nearest neighbor search
        indices = np.indices(chn.shape)
        tree = spatial.KDTree(list(zip(indices[0].ravel(),
                                       indices[1].ravel())))
        # Track progress
        todo = chn.size
        ready = 1
        # Array of evaluation criteria
        eval_array = np.empty(chn.shape)

        # Compute global regression
        gm, gc, grsqrt, gmean = self.apply_linear_regression(
                chn.ravel(), self.pan_degrad.ravel())

        # Loop over channel array
        for index, val in np.ndenumerate(chn):
            row = index[0]
            col = index[1]
            # Query tree for neighbors
            queryresult = tree.query(np.array([[row, col]]), k=25)
            neighs = tree.data[queryresult[1][0][1:]]
            # Get channel values for neighbors
>           chn_neigh = chn[tuple(neighs.T)].squeeze()
E           IndexError: arrays used as indices must be of integer (or boolean) type

../algorithms.py:1249: IndexError
----------------------------------------- Captured stderr call -----------------------------------------
2021-05-18 18:03:47,739 - fogpy.algorithms - INFO - Starting pansharpening algorithm for 1 images
2021-05-18 18:03:48,076 - fogpy.algorithms - INFO - Sharpen (141, 298, 1) image to (422, 893)
2021-05-18 18:03:48,076 - fogpy.algorithms - INFO - Using local regression approach by Hill
------------------------------------------ Captured log call -------------------------------------------
INFO     fogpy.algorithms:algorithms.py:1151 Starting pansharpening algorithm for 1 images
INFO     fogpy.algorithms:algorithms.py:1171 Sharpen (141, 298, 1) image to (422, 893)
INFO     fogpy.algorithms:algorithms.py:1226 Using local regression approach by Hill
_______________________ Test_StationFusionFilter.test_fusion_filter_bufr_import ________________________

self = <fogpy.test.test_filters.Test_StationFusionFilter testMethod=test_fusion_filter_bufr_import>

    @unittest.skipUnless(os.getenv("BUFR_TABLES"),
            "BUFR tables not defined, skipping bufr-related tests")
    def test_fusion_filter_bufr_import(self):
        # Create fusion filter
        testfilter = StationFusionFilter(self.ir108,
                                         ir108=self.ir108,
                                         ir039=self.ir039,
                                         lowcloudmask=self.lowcloudmask,
                                         elev=self.elev,
                                         bufrfile=testbufr,
                                         time=self.time,
                                         area=area_def)
>       ret, mask = testfilter.apply()

test_filters.py:6607:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../filters.py:133: in apply
    self.filter_function()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <fogpy.filters.StationFusionFilter object at 0x7ff01b198400>

    def filter_function(self):
        """Station data fusion filter routine

        This filter provide a data fusion approach for satellite derived low
        cloud masked raster data with vector data of visibilities observed by
        sensors at weather stations. This raster/vector data fusion is done in
        several steps.
        First a BUFR file with WMO standardized weather station data is
        imported. Next the visibility measurements are extracted and resampled
        to the given cloud mask raster array shape. Then a fog mask for the
        station data is created and a DEM based interpolation is performed.

        Afterwards the stations are compared to the low cloud and cloud masks
        and case dependant mask corrections are performed.

        Args:
            | ir108 (:obj:`ndarray`): Array for the 10.8 μm channel.
            | ir039 (:obj:`ndarray`): Array for the 3.9 μm channel.
            | lowcloudmask (:obj:`ndarray`): Low cloud mask array.
            | cloudmask (:obj:`ndarray`): General cloud mask array.
            | elev (:obj:`ndarray`): Array of elevation.
            | bufrfile (:obj:`str`): Path to weather station BUFR-file.
            | time (:obj:`Datetime`): Time instance of station data
                                      as *datetime* object.
            | area (:obj:`area_def`): Corresponding area definition for
                                      cloud masks.
            | heightvar (:obj:`float`): Height variance for fog masking in m.
                                        Default is 50 m.
            | fogthres (:obj:`float`): Visibility threshold for fog masking
                                       in m. Default is 1000 m.
            | limit (:obj:`bool`): Boolean to limit the output region to
                                   station data coverage.

        Returns:
            Filter image and filter mask.
        """
        logger.info("Applying Station Data Fusion Filter")

        # 1. Import BUFR file
        stations = read_synop(self.bufrfile, 'visibility')
        currentstations = stations[self.time.strftime("%Y%m%d%H0000")]
        lats = [i[2] for i in currentstations]
        lons = [i[3] for i in currentstations]
        vis = [i[4] for i in currentstations]

        # 2. Port visibility vector data to raster array shape
        self.visarr = np.empty(self.arr.shape[:2])
        self.visarr.fill(np.nan)

        x, y = (self.area.get_xy_from_lonlat(lons, lats))
        xmin, xmax, ymin, ymax = [np.nanmin(x), np.nanmax(x), np.nanmin(y),
                                  np.nanmax(y)]
        vis_ma = np.ma.array(vis, mask=x.mask)
>       self.visarr[y.compressed(), x.compressed()] = vis_ma.compressed()
E       IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (265,) (267,)

../filters.py:1291: IndexError
----------------------------------------- Captured stderr call -----------------------------------------
2021-05-18 18:03:59,303 - fogpy.filters - INFO - Applying Cloud Filter
2021-05-18 18:03:59,308 - fogpy.filters - INFO - Filter results for CloudFilter

                    Cloud filtering for satellite images.
                    Array size:              42018
                    Masking:                 20551
                    Previous masked:         0
                    New filtered:            20551
                    Remaining:               21467
2021-05-18 18:03:59,308 - fogpy.filters - INFO - Applying Station Data Fusion Filter
------------------------------------------ Captured log call -------------------------------------------
INFO     fogpy.filters:filters.py:397 Applying Cloud Filter
INFO     fogpy.filters:filters.py:182 Filter results for CloudFilter

                    Cloud filtering for satellite images.
                    Array size:              42018
                    Masking:                 20551
                    Previous masked:         0
                    New filtered:            20551
                    Remaining:               21467
INFO     fogpy.filters:filters.py:1274 Applying Station Data Fusion Filter
____________________ Test_StationFusionFilter.test_fusion_filter_dem_interpolation _____________________

self = <fogpy.test.test_filters.Test_StationFusionFilter testMethod=test_fusion_filter_dem_interpolation>

    @unittest.skipUnless(os.getenv("BUFR_TABLES"),
            "BUFR tables not defined, skipping bufr-related tests")
    def test_fusion_filter_dem_interpolation(self):
        # Create fusion filter
        testfilter = StationFusionFilter(self.ir108,
                                         ir108=self.ir108,
                                         ir039=self.ir039,
                                         lowcloudmask=self.lowcloudmask,
                                         elev=self.elev,
                                         bufrfile=testbufr,
                                         time=self.time,
                                         area=area_def,
                                         plot=False,
                                         save=True,
                                         resize=5)
>       ret, mask = testfilter.apply()

test_filters.py:6629:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../filters.py:133: in apply
    self.filter_function()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <fogpy.filters.StationFusionFilter object at 0x7ff01b198220>

    def filter_function(self):
        """Station data fusion filter routine

        This filter provide a data fusion approach for satellite derived low
        cloud masked raster data with vector data of visibilities observed by
        sensors at weather stations. This raster/vector data fusion is done in
        several steps.
        First a BUFR file with WMO standardized weather station data is
        imported. Next the visibility measurements are extracted and resampled
        to the given cloud mask raster array shape. Then a fog mask for the
        station data is created and a DEM based interpolation is performed.

        Afterwards the stations are compared to the low cloud and cloud masks
        and case dependant mask corrections are performed.

        Args:
            | ir108 (:obj:`ndarray`): Array for the 10.8 μm channel.
            | ir039 (:obj:`ndarray`): Array for the 3.9 μm channel.
            | lowcloudmask (:obj:`ndarray`): Low cloud mask array.
            | cloudmask (:obj:`ndarray`): General cloud mask array.
            | elev (:obj:`ndarray`): Array of elevation.
            | bufrfile (:obj:`str`): Path to weather station BUFR-file.
            | time (:obj:`Datetime`): Time instance of station data
                                      as *datetime* object.
            | area (:obj:`area_def`): Corresponding area definition for
                                      cloud masks.
            | heightvar (:obj:`float`): Height variance for fog masking in m.
                                        Default is 50 m.
            | fogthres (:obj:`float`): Visibility threshold for fog masking
                                       in m. Default is 1000 m.
            | limit (:obj:`bool`): Boolean to limit the output region to
                                   station data coverage.

        Returns:
            Filter image and filter mask.
        """
        logger.info("Applying Station Data Fusion Filter")

        # 1. Import BUFR file
        stations = read_synop(self.bufrfile, 'visibility')
        currentstations = stations[self.time.strftime("%Y%m%d%H0000")]
        lats = [i[2] for i in currentstations]
        lons = [i[3] for i in currentstations]
        vis = [i[4] for i in currentstations]

        # 2. Port visibility vector data to raster array shape
        self.visarr = np.empty(self.arr.shape[:2])
        self.visarr.fill(np.nan)

        x, y = (self.area.get_xy_from_lonlat(lons, lats))
        xmin, xmax, ymin, ymax = [np.nanmin(x), np.nanmax(x), np.nanmin(y),
                                  np.nanmax(y)]
        vis_ma = np.ma.array(vis, mask=x.mask)
>       self.visarr[y.compressed(), x.compressed()] = vis_ma.compressed()
E       IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (265,) (267,)

../filters.py:1291: IndexError
----------------------------------------- Captured stderr call -----------------------------------------
2021-05-18 18:04:03,852 - fogpy.filters - INFO - Applying Cloud Filter
2021-05-18 18:04:03,857 - fogpy.filters - INFO - Filter results for CloudFilter

                    Cloud filtering for satellite images.
                    Array size:              42018
                    Masking:                 20551
                    Previous masked:         0
                    New filtered:            20551
                    Remaining:               21467
2021-05-18 18:04:03,858 - fogpy.filters - INFO - Applying Station Data Fusion Filter
------------------------------------------ Captured log call -------------------------------------------
INFO     fogpy.filters:filters.py:397 Applying Cloud Filter
INFO     fogpy.filters:filters.py:182 Filter results for CloudFilter

                    Cloud filtering for satellite images.
                    Array size:              42018
                    Masking:                 20551
                    Previous masked:         0
                    New filtered:            20551
                    Remaining:               21467
INFO     fogpy.filters:filters.py:1274 Applying Station Data Fusion Filter
=========================================== warnings summary ===========================================
fogpy/test/test_algorithms.py::Test_BaseSatelliteAlgorithm::test_base_algorithm
  /home/gholl/checkouts/fogpy/fogpy/test/test_algorithms.py:122: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.testarray = np.arange(0, 16, dtype=np.float).reshape((4, 4))

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_complement
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_next
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_linreg_cluster
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_real
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:665: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.dz = np.empty(self.ir108.shape, dtype=np.float)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_complement
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_next
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_linreg_cluster
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_real
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:666: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.cth = np.empty(self.ir108.shape, dtype=np.float)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_complement
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_artificial_next
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_linreg
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_linreg_cluster
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_real
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:826: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
  To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
    m, c = np.linalg.lstsq(A, y)[0]

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_linreg_cluster
fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_real
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:1079: RuntimeWarning: Mean of empty slice
    cth = np.nanmean(cthmargin)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:714: RuntimeWarning: All-NaN slice encountered
    self.minheight = np.nanmin(self.result)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:715: RuntimeWarning: Mean of empty slice
    self.meanheight = np.nanmean(self.result)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:716: RuntimeWarning: All-NaN slice encountered
    self.maxheight = np.nanmax(self.result)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
  /home/gholl/checkouts/fogpy/fogpy/test/test_algorithms.py:448: RuntimeWarning: All-NaN slice encountered
    self.assertEqual(np.isnan(np.nanmax(lcthalgo.dz)), True)

fogpy/test/test_algorithms.py::Test_LowCloudHeightAlgorithm::test_lcth_algorithm_nan_neighbor
  /home/gholl/checkouts/fogpy/fogpy/test/test_algorithms.py:449: RuntimeWarning: All-NaN slice encountered
    self.assertEqual(np.isnan(np.nanmax(lcthalgo.cth)), True)

fogpy/test/test_algorithms.py::Test_PanSharpeningAlgorithm::test_pansharp_resample
fogpy/test/test_algorithms.py::Test_PanSharpeningAlgorithm::test_pansharp_resample
  /home/gholl/checkouts/pyresample/pyresample/image.py:60: FutureWarning: Usage of ImageContainer is deprecated, please use NumpyResamplerBilinear class instead
    warnings.warn(

fogpy/test/test_algorithms.py::Test_PanSharpeningAlgorithm::test_pansharp_resample
  /home/gholl/checkouts/fogpy/fogpy/algorithms.py:1207: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
  To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
    results, resids, rank, s = np.linalg.lstsq(A, y)

fogpy/test/test_algorithms.py: 21 warnings
  <__array_function__ internals>:5: DeprecationWarning: `np.alen` is deprecated, use `len` instead

fogpy/test/test_composites.py::test_get_area_lat_lon
fogpy/test/test_composites.py::test_interim
fogpy/test/test_composites.py::test_fog_comp_night
  /home/gholl/checkouts/satpy/satpy/composites/__init__.py:204: UserWarning: satpy.composites.CompositeBase.check_areas is deprecated, use satpy.composites.CompositeBase.match_data_arrays instead
    warnings.warn('satpy.composites.CompositeBase.check_areas is deprecated, use '

fogpy/test/test_composites.py::test_get_area_lat_lon
fogpy/test/test_composites.py::test_get_area_lat_lon
fogpy/test/test_composites.py::test_interim
fogpy/test/test_composites.py::test_interim
fogpy/test/test_composites.py::test_fog_comp_day
fogpy/test/test_composites.py::test_fog_comp_night
fogpy/test/test_composites.py::test_fog_comp_night
  /data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordinate-reference-systems
    proj_string = self.to_proj4()

fogpy/test/test_composites.py::test_interim
  /data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:1143: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
    return get_provider(package_or_requirement).get_resource_filename(

fogpy/test/test_composites.py::test_save_extras
fogpy/test/test_composites.py::test_save_extras
  /home/gholl/checkouts/satpy/satpy/writers/cf_writer.py:638: UserWarning: Dtype bool not compatible with CF-1.7.
    warnings.warn('Dtype {} not compatible with {}.'.format(str(ds.dtype), CF_VERSION))

fogpy/test/test_filters.py::Test_ArrayFilter::test_array_filter
fogpy/test/test_filters.py::Test_ArrayFilter::test_array_filter_param
fogpy/test/test_filters.py::Test_ArrayFilter::test_marray_filter
  /home/gholl/checkouts/fogpy/fogpy/test/test_filters.py:92: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.testarray = np.arange(0, 16, dtype=np.float).reshape((4, 4))

fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_clusters
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_allfog
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_nofog
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_randomfog
  /home/gholl/checkouts/fogpy/fogpy/filters.py:925: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.cbh = np.empty(self.clusters.shape, dtype=np.float)

fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_clusters
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_allfog
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_nofog
fogpy/test/test_filters.py::Test_LowCloudFilter::test_lowcloud_filter_single_lwp_randomfog
  /home/gholl/checkouts/fogpy/fogpy/filters.py:926: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    self.fbh = np.empty(self.clusters.shape, dtype=np.float)

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_filter_bufr_import
fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_filter_dem_interpolation
  /home/gholl/checkouts/pyresample/pyresample/geometry.py:1920: DeprecationWarning: 'get_xy_from_lonlat' is deprecated, please use 'get_array_indices_from_lonlat' instead.
    warnings.warn("'get_xy_from_lonlat' is deprecated, please use "

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1463: RuntimeWarning: All-NaN axis encountered
    np.nanmin(self.check_zerolist(elev[lowcloudhit])),

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1464: RuntimeWarning: Mean of empty slice
    np.nanmean(self.check_zerolist(

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1466: RuntimeWarning: All-NaN axis encountered
    np.nanmax(self.check_zerolist(elev[lowcloudhit])),

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1482: RuntimeWarning: All-NaN axis encountered
    np.nanmin(self.check_zerolist(

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1484: RuntimeWarning: Mean of empty slice
    np.nanmean(self.check_zerolist(

fogpy/test/test_filters.py::Test_StationFusionFilter::test_fusion_validation
  /home/gholl/checkouts/fogpy/fogpy/filters.py:1486: RuntimeWarning: All-NaN axis encountered
    np.nanmax(self.check_zerolist(

fogpy/test/test_lowwatercloud.py: 265 warnings
  /data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/scipy/optimize/_minimize.py:535: RuntimeWarning: Method BFGS cannot handle constraints nor bounds.
    warn('Method %s cannot handle constraints nor bounds.' % method,

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================= short test summary info ========================================
FAILED test_algorithms.py::Test_PanSharpeningAlgorithm::test_pansharp_resample - IndexError: arrays u...
FAILED test_filters.py::Test_StationFusionFilter::test_fusion_filter_bufr_import - IndexError: shape ...
FAILED test_filters.py::Test_StationFusionFilter::test_fusion_filter_dem_interpolation - IndexError: ...
================== 3 failed, 86 passed, 5 skipped, 1 xfailed, 349 warnings in 37.93s ===================
gerritholl commented 3 years ago

It's actually IndexError, not AttributeError. The BUFR tests are not run in GitHub CI, so those are skipped rather than failed, but one other test does fail there too. As for the IndexError in test_pansharp_resample, this appears because neighs is of dtype float64. Those have been illegal for indexing back since numy 1.11, so that should have failed already much earlier. Maybe something has changed in scipy.kdtree causing it to return floats rather than ints?

gerritholl commented 3 years ago

Indeed: spatial.KDTree([(0, 0)]).data.dtype is int64 in scipy 1.5, but float64 in scipy 1.6. This triggers the IndexError in test_pansharp_resample. See https://github.com/scipy/scipy/issues/14296

gerritholl commented 3 years ago

The other test that fails on GitHub is related to random numbers. Although I seed the random number generator, the result is different on GitHub than it on my local machine. I'm not sure why that happens. Ultimately random numbers are not a good idea in tests, but testing a stochastic algorithm otherwise is difficult.

gerritholl commented 3 years ago

I suppose an alternative would be mocking the random number generators.