openclimatefix / ocf-data-sampler

A repo for sampling from weather data for renewable energy prediction
MIT License
4 stars 4 forks source link

refactor nwp handling in pvnet.py #29

Open AUdaltsova opened 2 months ago

AUdaltsova commented 2 months ago

Detailed Description

Several functions in pvnet.py (find_valid_t0_times, slice_datasets_by_time, process_and_combine_datasets) have to unpack nested nwp sources when iterating through datasets and include preparation of specific parameters needed for nwp-specific functions. It might be better to take all of this out into separate functions and keep nwp datasets untested for this bit of code, to allow the functions to cleanly and homogeneously iterate over the sources

Context

Currently these three functions are a bit bulky and feel like they are doing a lot of stuff, this will compartmentalize them more.

Possible Implementation

something vaguely like this:

def is_nwp(key):
  split_key = key.split()
  return split_key[0] == "nwp"

def return_spacial_slice(da, config, key):
  if is_nwp(key):
    return spacial_slice_nwp(da, config, **get_additional_nwp_kwargs(da, config))
  else:
    return spacial_slice(da, config)

for key, dataset in datasets_dict.items():
  config = config.key
  slice = return_spacial_slice(dataset, config, key)
  slices.append(slice)