Create proof of concept: Iris tests fail if any warnings raised

trexfeathers commented 1 year ago

Steps

[ ] Set PyTest to fail the test run if any warnings are raised.
[x] Write up an account. How may tests fail? How many are from outside Iris? How many from Iris?
[x] Discuss with Iris devs to decide on next steps to take.
[ ] Address the DeprecationWarnings below that do not have their own issues. And the RunTimeWarnings?

acchamber commented 1 year ago

Linking relevant comment https://github.com/SciTools/iris/issues/5463#issuecomment-1708627943

acchamber commented 1 year ago

FutureWarnings can be supressed with a single line in pytest init - same with UserWarnings, although not sure this is the best approach.

By raw numbers of warnings raised the biggest culprit is.

lib/iris/tests/test_mapping.py: 10908 warnings
lib/iris/tests/test_plot.py: 579 warnings
lib/iris/tests/test_quickplot.py: 532 warnings
lib/iris/tests/integration/plot/test_plot_2d_coords.py: 571 warnings
lib/iris/tests/integration/plot/test_vector_plots.py: 6 warnings
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/cartopy/crs.py:827: DeprecationWarning: The 'geom_factory' function is deprecated in Shapely 2.0, and will be removed in a future version
    multi_line_string = cartopy.trace.project_linear(linear_ring,

Which is fixable by updating iris to Cartopy 0.22 - since this uses Shapely 2.0. See this issue in cartopy confirming resolved in 0.22. Running this test in default_next (which has cartopy 0.22) does fix this.

acchamber commented 1 year ago

Types of warning in pytest output

This is a breakdown of the different warnings remaining after you filter out FutureWarnings/Userwarnings, and update your environment with the latest iris lockfiles and cartopy 0.22 (which is not in the latest iris lockfiles but is in default-next and required to fix the above warnings)

Not Our problem/external warnings

While these don't have any path to the Iris team fixing them, they are something we should look at supressing, as it's warnings user receives which they cannot act on.

lib/iris/tests/integration/test_Datums.py: 4 warnings
lib/iris/tests/unit/analysis/cartography/test_project.py: 26 warnings
lib/iris/tests/unit/analysis/regrid/test_RectilinearRegridder.py: 4 warnings
lib/iris/tests/unit/coord_systems/test_LambertAzimuthalEqualArea.py: 8 warnings
lib/iris/tests/unit/plot/test_plot.py: 54 warnings
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pyproj/transformer.py:820: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    return self._transformer._transform_point(

lib/iris/tests/unit/plot/test_plot.py: 36 warnings
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pyproj/geod.py:404: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    return self._inv_point(

Requires Pyproj to fix - having a look at their codebase they seem to have a workaround for future numpy in place but not something handling the warning for older versions.

lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_2d_multi
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_2d_single
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_masked
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_masked_mdtol_0
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_multi_axis
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_multi_axis_multi_percent
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/numpy/lib/function_base.py:4834: DeprecationWarning: Out of bound index found. This was previously ignored when the indexing result contained no elements. In the future the index error will be raised. This error occurs either due to an empty slice, or if an array has zero elements even before indexing.
  (Use `warnings.simplefilter('error')` to turn this DeprecationWarning into an error and get more details on the invalid index.)
    previous = arr[previous_indexes]

lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_2d_multi
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_2d_single
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_masked
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_masked_mdtol_0
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_multi_axis
lib/iris/tests/unit/analysis/test_PERCENTILE.py::Test_lazy_fast_aggregate::test_multi_axis_multi_percent
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/numpy/lib/function_base.py:4835: DeprecationWarning: Out of bound index found. This was previously ignored when the indexing result contained no elements. In the future the index error will be raised. This error occurs either due to an empty slice, or if an array has zero elements even before indexing.
  (Use `warnings.simplefilter('error')` to turn this DeprecationWarning into an error and get more details on the invalid index.)
    next = arr[next_indexes]

Problem is the _quantile function in numpy causing these warnings, so no code we wrote,. Can't see any issues in numpy github about this,

Things we can fix

unit/experimental/ugrid/mesh/test_Connectivity.py:17
  /net/home/h05/achamber/git/iris/lib/iris/tests/unit/experimental/ugrid/mesh/test_Connectivity.py:17: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import parse_version

../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
../../../../../../../../../home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

Fix by migrating to new API. Issue already raised at #5491 for the sake of documentation

lib/iris/tests/test_cdm.py: 1 warning
lib/iris/tests/test_cube_to_pp.py: 58 warnings
lib/iris/tests/test_file_save.py: 8 warnings
lib/iris/tests/test_pp_cf.py: 6 warnings
lib/iris/tests/integration/test_pp.py: 2 warnings
lib/iris/tests/integration/fast_load/test_fast_load.py: 188 warnings
lib/iris/tests/unit/fileformats/um/fast_load/test_FieldCollation.py: 16 warnings
  /net/home/h05/achamber/git/iris/lib/iris/fileformats/pp.py:1199: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -99 to uint32 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    lb[index] = header_elem

Probably a one-line fix, but because I don't understand the intended casting behaviour written up a separate issue at #5492

lib/iris/tests/experimental/regrid/test_regrid_area_weighted_rectilinear_src_and_grid.py::TestAreaWeightedRegrid::test_lazy_nop
lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_lazy
lib/iris/tests/integration/analysis/test_area_weighted.py::AreaWeightedTests::test_regrid_area_w_lazy_chunked
lib/iris/tests/unit/analysis/area_weighted/test_AreaWeightedRegridder.py::TestLazy::test_output_lazy
lib/iris/tests/unit/analysis/area_weighted/test_AreaWeightedRegridder.py::TestLazy::test_src_stays_lazy
  /net/home/h05/achamber/git/iris/lib/iris/analysis/_area_weighted.py:606: DeprecationWarning: Out of bound index found. This was previously ignored when the indexing result contained no elements. In the future the index error will be raised. This error occurs either due to an empty slice, or if an array has zero elements even before indexing.
  (Use `warnings.simplefilter('error')` to turn this DeprecationWarning into an error and get more details on the invalid index.)
    src_area_datas_square = src_data[
        ..., square_data_indices_y, square_data_indices_x
    ]

Issue here feeding large arrays as indexes for slicing, which numpy currently supports but won't soon. Spent an hour or two trying to fix and got nowhere, so leaving be

lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_derived_points
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_derived_points_with_bounds
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_derived_points_with_bounds
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_no_depth
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_no_depth_c
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_no_eta
lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py::Test_make_coord::test_no_sigma
  /net/home/h05/achamber/git/iris/lib/iris/tests/unit/aux_factory/test_OceanSigmaZFactory.py:185: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    nsigma_slice = slice(0, nsigma)

This is a single line in some tests causing this warning, rather than the user-facing codebase. I'm not entirely sure what's triggering it but couldn't fix it with a simple int(nsigma) so moved onto other problems.

  /net/home/h05/achamber/git/iris/lib/iris/tests/unit/cube/test_Cube.py:78: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
    data = np.matrix([[1, 2, 3], [4, 5, 6]])

lib/iris/tests/unit/data_manager/test_DataManager.py::Test_data__setter::test_coerce_to_ndarray
  /net/home/h05/achamber/git/iris/lib/iris/tests/unit/data_manager/test_DataManager.py:463: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpy-for-matlab-users.html). Please adjust your code to use regular ndarray.
    matrix = np.matrix(real_array)

So, the intent on these tests was to check that cubes data was being properly casted to ndarray from other numpy ndarray subtypes, but looking at the numpy documentation they are moving away from allowing those entirely? These tests might not even be relevant anymore, and even if not when they are deprecated so is the thing they intend to test, so left be.

Runtime warnings

I haven't looked into these like the DeprecatationWarnings, so leaving them without comments. Not many of them, though.

lib/iris/tests/test_analysis.py: 5 warnings
lib/iris/tests/unit/analysis/test_MAX_RUN.py: 38 warnings
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/numpy/core/numeric.py:407: RuntimeWarning: invalid value encountered in cast
    multiarray.copyto(res, fill_value, casting='unsafe')

lib/iris/tests/unit/analysis/regrid/test_RectilinearRegridder.py::Test__regrid__extrapolation_modes::test_method_result_types
  /net/home/h05/achamber/git/iris/lib/iris/analysis/_regrid.py:836: RuntimeWarning: invalid value encountered in cast
    data[tuple(index)] = interpolate(src_subset)

lib/iris/tests/unit/analysis/maths/test_divide.py::TestMasking::test_unmasked_div_zero
  /net/home/h05/achamber/git/iris/lib/iris/tests/unit/analysis/maths/test_divide.py:54: RuntimeWarning: divide by zero encountered in divide
    com = self.data_op(dat_b, dat_a)

lib/iris/tests/unit/analysis/maths/test_divide.py::TestMasking::test_unmasked_div_zero
  /net/home/h05/achamber/git/iris/lib/iris/analysis/maths.py:918: RuntimeWarning: divide by zero encountered in divide
    data = operation_function(lhs, rhs)

lib/iris/tests/test_plot.py::Test2dPoints::test_circular_changes
  /home/h05/achamber/.conda/envs/iris-dev/lib/python3.11/site-packages/matplotlib/collections.py:963: RuntimeWarning: invalid value encountered in sqrt
    scale = np.sqrt(self._sizes) * dpi / 72.0 * self._factor

acchamber commented 1 year ago

Also there's a np.product -> np.prod find/replace PR at #5493 as part of this

ESadek-MO commented 1 year ago

All warnings brought up in this issue now have their own issues, I consider this issue resolved.

SciTools / iris