openclimatefix / metoffice_ec2

Subset Met Office MOGREPS-UK and UKV on AWS EC2
MIT License
11 stars 2 forks source link

Add some files from the MetOffice feed for testing #18

Closed tomwhite closed 4 years ago

tomwhite commented 4 years ago

It would be useful to have a few representative files checked into the repo for testing purposes (i.e. to write unit tests against). Also if we want to change the way we store things (e.g. #17) so we can test changes locally.

JackKelly commented 4 years ago

Definitely agree!

UPDATE: Ignore the rest of this comment! Please see the next comment!...

The Met Office NetCDF files are about 100 MBytes.

100 MB maybe feels a little large to put into our git repo. What do you think?

We could put a handful of testing NetCDF files in an OCF public S3 bucket (which would also help us test the code that reads from S3. But that would slow the tests down (when running locally, at least)). Or we could reduce the filesize by throwing away most of the data, and then saving the NetCDF in git. Or we could do both?!? Or something else? What would you prefer?

JackKelly commented 4 years ago

Scrap what I said above! Some of the NetCDF files are less than a MB (so we can probably just stick them into git).

For reference, here's the mean size (in MB) of MOGREPS-UK NetCDF files from the Met Office:

air_pressure                                                                             229.484640
air_pressure_at_sea_level                                                                  3.359660
air_temperature                                                                           21.432869
atmosphere_boundary_layer_thickness                                                        3.463518
cloud_area_fraction                                                                        2.139006
cloud_area_fraction_assuming_only_consider_surface_to_1000_feet_asl                        1.191044
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_2p5_oktas      1.198854
cloud_base_altitude_assuming_only_consider_cloud_area_fraction_greater_than_4p5_oktas      1.142050
cloud_volume_fraction_in_atmosphere_layer                                                 31.550393
dew_point_temperature                                                                      4.072501
fog_area_fraction                                                                          0.287845
freezing_level_altitude                                                                    4.102886
high_type_cloud_area_fraction                                                              1.457913
land_binary_mask                                                                           0.209811
low_type_cloud_area_fraction                                                               2.073978
lwe_graupel_and_hail_fall_rate                                                             0.145724
lwe_snowfall_rate                                                                          0.142409
lwe_thickness_of_graupel_and_hail_fall_amount                                              0.145957
lwe_thickness_of_snowfall_amount                                                           0.143660
lwe_thickness_of_surface_snow_amount                                                       0.368421
medium_type_cloud_area_fraction                                                            1.509519
number_of_lightning_flashes_per_unit_area                                                  0.139348
rainfall_rate                                                                              0.545779
relative_humidity                                                                         70.426761
soil_temperature                                                                           5.251025
surface_air_pressure                                                                       7.087338
surface_altitude                                                                           2.405621
surface_diffusive_downwelling_shortwave_flux_in_air                                        4.592893
surface_direct_downwelling_shortwave_flux_in_air                                           4.460242
surface_downwelling_longwave_flux_in_air                                                   6.106000
surface_downwelling_shortwave_flux_in_air                                                  5.235519
surface_downwelling_ultraviolet_flux_in_air                                                1.054016
surface_temperature                                                                        2.256696
surface_upward_sensible_heat_flux                                                          4.923685
surface_upwelling_ultraviolet_flux_in_air                                                  0.400496
thickness_of_rainfall_amount                                                               0.620976
total_radar_reflectivity                                                                  24.705189
total_radar_reflectivity_max_in_column                                                     3.865009
upward_air_velocity                                                                      112.851796
visibility_in_air                                                                          7.156086
wet_bulb_freezing_level_altitude                                                           4.419323
wet_bulb_potential_temperature                                                             4.963114
wind_from_direction                                                                       81.070012
wind_speed                                                                                33.663551
wind_speed_of_gust                                                                         3.214056