search

openclimatefix / nowcasting_dataset

Prepare batches of data for training machine learning solar electricity nowcasting data
https://nowcasting-dataset.readthedocs.io/en/stable/
MIT License
25 stars 6 forks source link

Missing NWP data #494

Closed peterdudfield closed 2 years ago

peterdudfield commented 2 years ago

Detailed Description

In ':/mnt/storage_ssd_4tb/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/prepared_ML_training_data/v14/train/nwp' there are only 769 batches. Should be 4000 non in the test folder

Context

need this to run ML models

Possible Implementation

run preapre_ml_script.py - but might be after https://github.com/openclimatefix/nwp/issues/6

JackKelly commented 2 years ago

yeah, please wait for nwp#6 before re-creating nwp batches

creating nwp batches should be a lot faster after my latest pr is merged

JackKelly commented 2 years ago

hopefully we just need to re-run prepare_ml_data.py --data_source nwp after the new NWP Zarr is created (see https://github.com/openclimatefix/nwp/issues/6) (and we'll need to update the config to use the new NWP zarr)

JackKelly commented 2 years ago

We now have a full set of NWP batches