Change batches into dataframe

[peterdudfield]

• filter area depending on #221
• load all batchs and coarsen to 4 by 4 grid (xr.coarsen)
• just take 1 hour wind energy data
• use train and validation folders

Time	Target_time	Step	u10_0	...	u10_15	v10_0	wind energy
2023-01-01	2023-01-01	0	0	0	0	0	0	0	0.5
2023-01-01	2023-01-01 01:00	1	0	0	0	0	0	1000
2023-01-01	2023-01-01 02:00	2	0	0	0	0	0	2000

[Sukh-P]

@peterdudfield just to check which ECMWF variables should I take for this? The batches have 16 in them but not sure if you want them all or just the ones listed here https://huggingface.co/openclimatefix/windnet_india/blob/main/data_config.yaml?

[peterdudfield]

yea, i'd start with those 7

[Sukh-P]

Outputs from this have been put here the training data is from 20,000 samples used in training for the Windnet model that have been converted into a dataframe, the csv is around a million rows and 100 columns, the validation data is roughly 15% the size of the training data (made up of around 3,000 samples). Will let others review the outputs to check is nothing is missing.

Some thoughts:

• Could be interesting to modify this dataset to include additional columns like windspeed (u**2 + v**2)**0.5 at the different lat long locations (just u and v 100) alongside the existing NWP channels
• Would be interesting to have the dataset with just windspeed at the lat long locations without the NWP channels to see how much impact the channels by themselves have