Open JackKelly opened 2 years ago
I'll try to do this later this afternoon using means = ds_from_zarr['stacked_eumetsat_data'].mean(dim=['time', 'x', 'y']).compute()
OK, I've written a little script to get the means and mins...
Just ignoring the eumetsat_zarr_2020_02.zarr
(and not ignoring the "rectangle of zeros") the new stats are:
Coordinates:
* variable (variable) <U6 'IR_016' 'IR_039' 'IR_087' ... 'WV_062' 'WV_073'
Data variables:
mean (variable) float64 141.9 800.7 701.6 733.8 ... 116.1 601.8 517.8
std (variable) float64 157.3 200.6 181.3 183.3 ... 135.5 177.6 159.4,)
which, reassuringly, look identical to the existing means and standard deviations!
I'll also compute the means and standard deviations ignoring the "rectangle of zeros".
I'll update this when my script runs again and prints all the values out!
The "rectangle of zeros" starts at 1,037,047 meters (OSGB "northing") and extends north. So need to ignore everything north of this
Detailed Description
Issue #30 documents an issue in non-HRV Zarr data whereby there's a 'rectangle of zeros'. We should re-compute the means and standard deviations, ignoring northern latitudes which often have a 'rectangle of zeros'.
https://github.com/openclimatefix/nowcasting_dataset/issues/484 documents an issue whereby
v2/eumetsat_zarr_2020_02.zarr
is full of noise from -30,000 to 30,000. We should ignorev2/eumetsat_zarr_2020_02.zarr
when computing the means and stds.Possible Implementation
Maybe create a simple script in Satip which computes the means and standard deviations (using dask?)