search

metoppv / improver

IMPROVER is a library of algorithms for meteorological post-processing.
http://improver.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
101 stars 84 forks source link

Masked errors no longer retained in `CalculateForecastBias` Plugin #1968

Closed benowen-bom closed 2 months ago

benowen-bom commented 8 months ago

Following an upgrade from IMPROVER version 1.4.0 to 1.7.0, an issue has arisen with the bias (forecast_error) cubes used in applying simple bias correction (which is used within the BoM for statistical downscaling of global ensemble models). It appears that the masked areas of the analysis (truth) dataset are no longer being retained and instead filled with values of -9.96921e+36. Consequently, these bad values are being applied during the bias-correction step.

Tracking down the source, it appears that between IMPROVER version 1.4.0 and 1.5.0, the behaviour of the following operation has changed: https://github.com/metoppv/improver/blob/a1b744b51576c2f2fd3e5cba3afb53af6e30643e/improver/calibration/simple_bias_correction.py#L77 where forecasts and truths are iris.cube.Cube objects. Previously the mask was the logical_or of forecasts.data.mask and truths.data.mask, however following 1.5.0 this has changed to copying the mask from the first argument. Thus the mask is no longer retained and the -9.96921e+36 values are coming through via the default fill value for truth.data in the the masked areas.

Reviewing the commit history, there is no obvious change that would explain what is happening here.

A simple fix is to explicit perform the logical_or operation on the masks to ensure that the mask is carried over to the forecast_error cube. However, without understanding the reason behind this change, it is difficult to say whether this is an isolated issue or could have arisen in other parts of the codebase.

benowen-bom commented 2 months ago

This issue was resolved in #1969, however what gave rise to the change in default behaviour in Line 77 remains unclear.