search

openclimatefix / nowcasting_dataset

Prepare batches of data for training machine learning solar electricity nowcasting data
https://nowcasting-dataset.readthedocs.io/en/stable/
MIT License
25 stars 6 forks source link

`PVDataSource` is ignoring 52 PV systems from the PassivSystems data... let's manually check if those 52 PV systems really should be removed :slightly_smiling_face: #433

Open JackKelly opened 2 years ago

JackKelly commented 2 years ago

Not urgent

2021-11-17 17:55:06,391 DEBUG processID=1555225 Creating pv DataSource object. | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/manager.py#L103
2021-11-17 17:55:06,393 DEBUG processID=1555225 Loading PV metadata from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/system_metadata_OCF_ONLY.csv | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/pv/pv_data_source.py#L72
DEBUG:nowcasting_dataset.data_sources.pv.pv_data_source:Loading PV metadata from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/system_metadata_OCF_ONLY.csv
2021-11-17 17:55:06,514 DEBUG processID=1555225 Loading PV Power data from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/passiv.netcdf | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/pv/pv_data_source.py#L98
DEBUG:nowcasting_dataset.data_sources.pv.pv_data_source:Loading PV Power data from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/passiv.netcdf
2021-11-17 17:55:06,514 DEBUG processID=1555225 Loading Solar PV Data from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/passiv.netcdf from None to None. | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/pv/pv_data_source.py#L355
DEBUG:nowcasting_dataset.data_sources.pv.pv_data_source:Loading Solar PV Data from /mnt/storage_b/data/ocf/solar_pv_nowcasting/nowcasting_dataset_pipeline/PV/Passiv/ocf_formatted/v0/passiv.netcdf from None to None.
2021-11-17 17:55:27,848 DEBUG processID=1555225 Loading Solar PV Data: done | /home/jack/dev/ocf/nowcasting_dataset/nowcasting_dataset/data_sources/pv/pv_data_source.py#L383
DEBUG:nowcasting_dataset.data_sources.pv.pv_data_source:Loading Solar PV Data: done
52 bad PV systems found and removed!
peterdudfield commented 2 years ago

yea, would be good to expand on what 'bad' means. Perhaps its nans in the data? Or not over a certain threshold?