bayliffe
commented
1 year ago Test directory sizes at outset
The list below shows the initial state of the acceptance test data in terms of data volume. The list is provided in descending order of size so that work can start on the largest sets of test files first.
Acceptance test directory sizes
535M ./wind_downscaling
236M ./generate-realizations
153M ./weighted_blending
102M ./apply-night-mask
98M ./nbhood
97M ./wind_direction
83M ./threshold
70M ./time-lagged-ens
56M ./extract
53M ./wet-bulb-temperature
52M ./recursive-filter
52M ./orographic_enhancement
40M ./wet-bulb-temperature-integral
38M ./wxcode
31M ./combine
26M ./wind-gust-diagnostic
26M ./resolve-wind-components
25M ./phase-change-level
24M ./apply-lapse-rate
23M ./temporal-interpolate
21M ./nbhood-land-and-sea
21M ./generate-percentiles
20M ./nowcast-optical-flow-from-winds
19M ./regrid
19M ./apply-emos-coefficients
17M ./generate-topography-bands-weights
15M ./spot-extract
15M ./between-thresholds
14M ./nbhood-iterate-with-mask
14M ./cloud-top-temperature
12M ./feels_like_temp
11M ./nowcast-optical-flow
11M ./generate-orographic-smoothing-coefficients
11M ./estimate-emos-coefficients
9.2M ./wxcode-modal
8.9M ./neighbour-finding
8.4M ./estimate-emos-coefficients-from-table
7.8M ./nowcast-extrapolate
6.3M ./standardise
5.9M ./cloud-condensation-level
5.8M ./nowcast-accumulate
5.8M ./generate-topography-bands-mask
5.6M ./interpolate-using-difference
5.6M ./apply-rainforests-calibration
5.2M ./blend-adjacent-points
4.7M ./lightning-from-cape-and-precip
4.2M ./hail-fraction
3.6M ./apply-bias-correction
3.1M ./uv-index
3.1M ./phase-probability
2.6M ./freezing-rain
2.4M ./vertical-updraught
2.4M ./field-texture
2.1M ./relabel_to_period
1.8M ./merge
1.7M ./apply-reliability-calibration
1.4M ./hail-size
1.3M ./temp-lapse-rate
1.2M ./generate-metadata-cube
1.2M ./extend-radar-mask
1.1M ./vicinity
1.1M ./fill-radar-holes
1.1M ./calculate-forecast-bias
908K ./create-grid-with-halo
904K ./construct-reliability-tables
780K ./manipulate-reliability-table
656K ./generate-clearsky-solar-radiation
644K ./shower-condition-probability
644K ./expected-value
528K ./phase-mask
520K ./aggregate-reliability-tables
516K ./remake-as-shower-condition
516K ./max-in-time-window
396K ./generate-solar-time
392K ./snow-fraction
392K ./sleet_probability
392K ./generate-landmask
392K ./convection-ratio
392K ./blend-cycles-and-realizations
260K ./interpret_metadata
20K ./shower-condition
Approach
Where possible we will coarsen the resolution of data. This will maintain the geographic context, and an impression of the synoptic situation, in the test inputs and known good outputs. In cases where the grid scale is important, for example in neighbourhooding, recursive filter, or topographically aware processes, we will use a smaller domain to preserve the grid resolution whilst reducing the data volume.
During an in-person discussion, it was agreed that reducing the size of the acceptance test data is a useful target, as this will make maintenance of the dataset easier and make the acceptance tests quicker to run.
The intention is to reduce spatial dimensions to somewhere around 50x50 to 150x150, depending on the diagnostic and the plugin - smaller is better if possible, but some will need a larger spatial area to include realistic weather features in the data. Some data files contain probability thresholds, percentiles or ensemble members and these can also be thinned out where practical to reduce the data size.
There are also currently a few tests which are disabled as they take a very long time to run on the current dataset and these could be re-enabled as smaller input datasets will almost certainly take less time to run.
These changes are best done over a series of PRs, rather than one large PR. This series-of-PRs approach will allow incremental reductions in the data size, reduce the chance of merge conflicts and also assist with keeping the review size manageable. Most of the review effort will be looking at the data files, the code changes are likely to be small.
Acceptance criteria:
The synthetic data approach as described in #1218, #1274, #1275 and #1276 is still valuable, but will require significantly more resources to implement.