Integrating Windnet

[confusedmatrix]

Pull Request

Description

Uses Windnet for producing forecasts when asset_type == "wind"

TODO

• [x] Succesfully run inference on Windnet model
• [x] Re-enable and fix skipped tests once integration complete
• [x] Pull in and use live generation data as wind inputs
• [x] Use dummy NWP data rather than pass in a file while NWP consumer is not ready
• [x] Handle case when there are missing values in generation data

Fixes #

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• [x] Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

• [ ] Yes

Checklist:

• [x] My code follows OCF's coding style guidelines
• [x] I have performed a self-review of my own code
• [x] I have made corresponding changes to the documentation
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have checked my code and corrected any misspellings

[peterdudfield]

Sorry @confusedmatrix Ive done a few updates, hopefully they are not too bad to merge. Let me know if you need any help

[confusedmatrix]

Mostly ready to review - I think the Dockerfile just needs some updating to get it to run the app.

Currently getting this error when trying to run the container:

Rioxarray is not installed, so not importing OpenTopography
Traceback (most recent call last):
  File "/venv/bin/app", line 5, in <module>
    from india_forecast_app.app import app
  File "/venv/lib/python3.11/site-packages/india_forecast_app/app.py", line 18, in <module>
    from .models import DummyModel, PVNetModel
  File "/venv/lib/python3.11/site-packages/india_forecast_app/models/__init__.py", line 4, in <module>
    from .pvnet.model import PVNetModel
  File "/venv/lib/python3.11/site-packages/india_forecast_app/models/pvnet/model.py", line 16, in <module>
    from ocf_datapipes.training.pvnet import construct_sliced_data_pipeline as pv_base_pipeline
  File "/venv/lib/python3.11/site-packages/ocf_datapipes/training/pvnet.py", line 10, in <module>
    from ocf_datapipes.training.common import (
  File "/venv/lib/python3.11/site-packages/ocf_datapipes/training/common.py", line 13, in <module>
    from ocf_datapipes.load import (
ImportError: cannot import name 'OpenTopography' from 'ocf_datapipes.load' (/venv/lib/python3.11/site-packages/ocf_datapipes/load/__init__.py)

Also there is a bug in the tests if they are run between the start of the hour and quarter past - something to do with the init_times in the NWP data:

 KeyError: "not all values found in index 'step'\nThis exception is thrown by __iter__ of SelectTimeSliceNWPIterDataPipe(dropout_frac=0, dropout_timedeltas=None, forecast_duration=datetime.timedelta(days=2), history_duration=datetime.timedelta(seconds=3600), sample_period_duration=datetime.timedelta(seconds=3600), source_datapipe=AddT0IdxAndSamplePeriodDurationIterDataPipe, t0_datapipe=_ChildDataPipe)"

[peterdudfield]

Mostly ready to review - I think the Dockerfile just needs some updating to get it to run the app.

Currently getting this error when trying to run the container:

Rioxarray is not installed, so not importing OpenTopography
Traceback (most recent call last):
  File "/venv/bin/app", line 5, in <module>
    from india_forecast_app.app import app
  File "/venv/lib/python3.11/site-packages/india_forecast_app/app.py", line 18, in <module>
    from .models import DummyModel, PVNetModel
  File "/venv/lib/python3.11/site-packages/india_forecast_app/models/__init__.py", line 4, in <module>
    from .pvnet.model import PVNetModel
  File "/venv/lib/python3.11/site-packages/india_forecast_app/models/pvnet/model.py", line 16, in <module>
    from ocf_datapipes.training.pvnet import construct_sliced_data_pipeline as pv_base_pipeline
  File "/venv/lib/python3.11/site-packages/ocf_datapipes/training/pvnet.py", line 10, in <module>
    from ocf_datapipes.training.common import (
  File "/venv/lib/python3.11/site-packages/ocf_datapipes/training/common.py", line 13, in <module>
    from ocf_datapipes.load import (
ImportError: cannot import name 'OpenTopography' from 'ocf_datapipes.load' (/venv/lib/python3.11/site-packages/ocf_datapipes/load/__init__.py)

Also there is a bug in the tests if they are run between the start of the hour and quarter past - something to do with the init_times in the NWP data:

 KeyError: "not all values found in index 'step'\nThis exception is thrown by __iter__ of SelectTimeSliceNWPIterDataPipe(dropout_frac=0, dropout_timedeltas=None, forecast_duration=datetime.timedelta(days=2), history_duration=datetime.timedelta(seconds=3600), sample_period_duration=datetime.timedelta(seconds=3600), source_datapipe=AddT0IdxAndSamplePeriodDurationIterDataPipe, t0_datapipe=_ChildDataPipe)"

Might need to add https://pypi.org/project/rioxarray/ to requirements to get bug working with docker file. Maybe we should run the tests inside docker, in order to catch these erros in the CI. https://github.com/openclimatefix/ocf_datapipes/blob/main/ocf_datapipes/load/__init__.py - this gave me a clue

Ill have a look at the 0-15 test bug, a bit later

[peterdudfield]

Also could use @freeze_time("2021-01-01 12:00:01") in order to help debugin that first 15 mins bug, although with fixtures im not sure this works

[peterdudfield]

For the first 15 minute bug, lets conside these two examples

IN PROGRESS		9.07	9.25	11.55	12.01	12.17
app.py - timestamp	9.00	9.15	11.45	12.00	12.15
pvnet/utils.py t0_datetime_utc	09.00	09.00	09.00	12.00	12.00
select_time_slice_nwp.py init_time			09.00	12.00	12.00
select_time_slice_nwp.py target_times			11.00 onwards	11.00 onwards	12.00 onwards

Ive added a minus 1 hour to the t0_datetime_utc in the pvnet/utils, which i think will sort this.

[peterdudfield]

15 minute bug is sovled

[peterdudfield]

The docker issue should now be sovled, but im struggling to test it, Ill keep trying.

Would you be able to add the command you use into the readme.md of how to run things locally in the docker file

[confusedmatrix]

The docker issue should now be sovled, but im struggling to test it, Ill keep trying.

Would you be able to add the command you use into the readme.md of how to run things locally in the docker file

Done

[peterdudfield]

Update: Weather data is now there, im tempted to merge this, get this up and running. And solve any bugs. And then @confusedmatrix if you have time get the it working with weather data, I might have to give you AWS access, or we think of another way

[peterdudfield]

I thought I would merge this @confusedmatrix so I can try testing it. Then any bug fixes, or NWP data, can be done on a new branch