Re-compute means and standard deviation of satellite data, ignoring 'rectangle of zeros' and ignoring `v2/eumetsat_zarr_2020_02.zarr`

[JackKelly]

Detailed Description

Issue #30 documents an issue in non-HRV Zarr data whereby there's a 'rectangle of zeros'. We should re-compute the means and standard deviations, ignoring northern latitudes which often have a 'rectangle of zeros'.

https://github.com/openclimatefix/nowcasting_dataset/issues/484 documents an issue whereby v2/eumetsat_zarr_2020_02.zarr is full of noise from -30,000 to 30,000. We should ignore v2/eumetsat_zarr_2020_02.zarr when computing the means and stds.

Possible Implementation

Maybe create a simple script in Satip which computes the means and standard deviations (using dask?)

[JackKelly]

I'll try to do this later this afternoon using means = ds_from_zarr['stacked_eumetsat_data'].mean(dim=['time', 'x', 'y']).compute()

[JackKelly]

OK, I've written a little script to get the means and mins...

Just ignoring the eumetsat_zarr_2020_02.zarr (and not ignoring the "rectangle of zeros") the new stats are:

Coordinates:
  * variable  (variable) <U6 'IR_016' 'IR_039' 'IR_087' ... 'WV_062' 'WV_073'
Data variables:
    mean      (variable) float64 141.9 800.7 701.6 733.8 ... 116.1 601.8 517.8
    std       (variable) float64 157.3 200.6 181.3 183.3 ... 135.5 177.6 159.4,)

which, reassuringly, look identical to the existing means and standard deviations!

I'll also compute the means and standard deviations ignoring the "rectangle of zeros".

I'll update this when my script runs again and prints all the values out!

[JackKelly]

The "rectangle of zeros" starts at 1,037,047 meters (OSGB "northing") and extends north. So need to ignore everything north of this