Get more data - Githubissues

openclimatefix / nowcasting_dataset

Prepare batches of data for training machine learning solar electricity nowcasting data

https://nowcasting-dataset.readthedocs.io/en/stable/

MIT License

24 stars 6 forks source link

Get more data #9

Closed JackKelly closed 3 years ago

JackKelly commented 3 years ago

[ ] NWP (more UKV from CEDA but need to talk to Met Office to get access again. Maybe MOGREPs. Maybe ECMWF).
[ ] EUMETSAT SEVIRI RSS (extend Future Energy Associate's Airflow pipeline code for ingesting data from EUMETSAT's API)
[ ] PV (sub-tasks: get more data from PVOutput.org for UK, using OCF's PVOutput Python code. Get data from PassivSystems (UK only). Get PV data from European PV systems)
[ ] CM-SAF irradiance?
[ ] Precipitation (UK rainfall radar. Jacob has already loaded this into his code, I think)
[ ] EUMETSAT cloud mask? (Jacob has cloud mask data, I think)

JackKelly commented 3 years ago

On the topic of UK precipitation data, Jacob says:

Haven’t gotten around to actually getting it to the same projection as the other stuff, focused more on models, etc.
CEDA’s software for reading it (All Python2): https://data.ceda.ac.uk/badc/ukmo-nimrod/software/python
The best looking script I made compatible with Python 3 here: https://github.com/openclimatefix/satflow/blob/main/satflow/data/utils/nimrod.py

JackKelly commented 3 years ago

To load the UK Rainfall data, Karel says we should use the "NIMROD reader from the IRIS library, which is by itself basically a standalone file: https://github.com/SciTools/iris/blob/main/lib/iris/fileformats/nimrod.py and it specifies and parses all the fields from the NIMROD header. The resulting NimrodField(...).data is a numpy array."

JackKelly commented 3 years ago

Closing this issue in favour of finer-grained issues: