andrewphilipsmith commented 7 months ago

The scope of this issue is just to add indicative automated tests. It does not require any changes to main pipeline code until we have determined the desired behaviour.

To contribute to resolving #100 and #106, it would be helpful to have tests that determine the current behaviour of the calendar mapping functions. The tests should include (at least) the following scenarios:

Test the behaviour of the bits of code referenced in this comment.

eg what is the actual difference between data and data_360 from this line of code:

data_360 = data.convert_calendar(dim="time", calendar="360_day", align_on="year")

What is in behaviour does the enforce_date_dropping function make? What is the difference between data_dates_dropped and data_360 in this snipit?

if data.time.dt.is_leap_year.any():
    data_dates_dropped = enforce_date_dropping(data, data_360)

Test for a leap year and non-year
When mapping from 365/366 days ==> 360 days,
- Do we just drop the extra days?
- How should we handle 30th Feb and (for non-leap years) 29th Feb?
When mapping from 360 days ==> 365/366 days, how should se add extra days?
See this spreadsheet for an example of how the different calendars map on different dates: https://thealanturininstitute-my.sharepoint.com/:x:/g/personal/a_smith_turing_ac_uk/EXPWO4JyXGxEpb-9cbY2KCQBTKkwst5aI68Ynt45eI7FIg?e=lWOR3V

Sub tasks

[x] Create test data files (eg one-pixel for a leap year and a non leap-year)
[x] Write unittest for the resample_hadukgrid function within python/resampling/resampling_hads.py, asserting on the dimensions of the input and output data.
[x] Write unittest for the enforce_date_dropping function within python/resampling/resampling_hads.py, asserting on the dimensions of the input and output data.

~Write unittest for preprocess_data in python/debiasing/preprocess_data.py, asserting on the dimensions of the input and output data.~ the python/debiasing/pre_process_data.py process may now be superseded. Will open a new ticket separately if needed.

griff-rees commented 7 months ago

Thanks @andrewphilipsmith. @aranas FYI: refactoring your work on #32 to check and automate testing is my focus, specifically:

python/resampling/check_calendar.py
python/resampling/resampling_hads.py
python/resampling/check_calendar_log.txt

griff-rees commented 7 months ago

The doctests of enforce_date_dropping in python/resample_hads.py in the day-sampling branch demonstrate that by default that function produces 1437 days over 4 years, which looks like 3 days less than the expected 360 days per year ($360 * 4 = 1440$).

    >>> ts_4_years: xr.DataArray = enforce_date_dropping(
    ...     test_4_years_xarray, test_4_years_xarray)
    >>> ts_4_years
    <xarray.DataArray (time: 1437, space: 3)>
    array([[0.5488135 , 0.71518937, 0.60276338],
           [0.43758721, 0.891773  , 0.96366276],
           [0.38344152, 0.79172504, 0.52889492],
           ...,
           [0.0916689 , 0.62816966, 0.52649637],
           [0.50034874, 0.93687921, 0.88042738],
           [0.71393397, 0.57754071, 0.25236931]])
    Coordinates:
      * time     (time) object 1980-11-30 1980-12-02 ... 1984-11-28 1984-11-29
      * space    (space) <U10 'Glasgow' 'Manchester' 'London'
    >>> len(ts_4_years) == 365*4 + 1  # Would keep all days
    False
    >>> len(ts_4_years) == 360*4      # Would enforce all years at 360 days
    False
    >>> len(ts_4_years)               # 3 days fewer than 360 per year
    1437

griff-rees commented 3 months ago

This issue is addressed in this branch and should be closable when merged: https://github.com/alan-turing-institute/clim-recal/pull/145

alan-turing-institute / clim-recal

Add automated tests to determine the actual behaviour of the calendar mapping functions #132

Sub tasks