[Meta] Get Power Perceiver model into production

[peterdudfield]

Here's the doc which describes what still needs to be done

TODO:

• [x] https://github.com/openclimatefix/power_perceiver/issues/149
• [x] Import power_perceiver
• [x] make new config
• [x] #118
• [x] add test for fake data (create new tests folder for power_perceiver, similar to the existing structure for cnn)
• [x] https://github.com/openclimatefix/power_perceiver/issues/152
• [x] run with real data from s3, see if values are sensible
• [ ] update forecaster for infrastructure or run multiple forecast - https://github.com/openclimatefix/nowcasting_infrastructure/issues/126
• [ ] Check all the normalisation
• [ ] Make sure the config is loading Passiv PV data
• [ ] Add tag to Power_perceiver, so if the code changes, the forecasts will continue to work for the moment
• [ ] Check the PV system IDs (do we need a look-up-table for the "pv system row number"?)
• [x] https://github.com/openclimatefix/nowcasting_forecast/issues/161
• [ ] do we need to retrain for NWP channels?
• [ ] do we need to retrain for satellite data delay?

[JackKelly]

tbh, given the time pressure, and how fiddly the data is, I'm going to prioritise getting the model working on "real" data first, if that's OK?

[peterdudfield]

tbh, given the time pressure, and how fiddly the data is, I'm going to prioritise getting the model working on "real" data first, if that's OK?

As in, get the batch from s3? and run it on there?

[peterdudfield]

Hey @jacobbieker

Im just looking at the PP code - https://github.com/openclimatefix/power_perceiver/blob/main/power_perceiver/experiments/exp_027_longer_forecasts.py

I'm using this blog as useful guidance

Im a bit worried about the code going into production.

I think we've got two options

1. Copy the code out to this forecast
2. keep the code in that repo, but move it to a new folder, add github actions to repo

Either way, we need to

• refactor code, would be good to divide it up to dataloader, SatellitePredictor, SatelliteTransformer, PVRNN and FullModel
• add tests to the model, this could be as much in testing https://github.com/openclimatefix/power_perceiver/issues/177
• Add method to change Batch object to dict going into model (would be good to make this a Pydantic object)

What do you think?

[jacobbieker]

I'd prefer keeping it in power perceiver, so we get more used to being able to more easily swap out models by importing them rather than copying them in every time. I agree on refactoring, but think we can keep the different parts of the full model together in a new folder, and just break it into fullmodel+component parts, the dataloader and training.

I agree on unit tests, I'd just go with one end to end test of loading the batch at the moment just so we can get the model into production a bit quicker, and then add more unit tests for the individual components.

[peterdudfield]

I'd prefer keeping it in power perceiver, so we get more used to being able to more easily swap out models by importing them rather than copying them in every time. I agree on refactoring, but think we can keep the different parts of the full model together in a new folder, and just break it into fullmodel+component parts, the dataloader and training.

I agree on unit tests, I'd just go with one end to end test of loading the batch at the moment just so we can get the model into production a bit quicker, and then add more unit tests for the individual components.

Thats sounds good, so roughly itll be this

• move into new folder - https://github.com/openclimatefix/power_perceiver/issues/183
• refactor into fullmodel+component parts - https://github.com/openclimatefix/power_perceiver/issues/183
• https://github.com/openclimatefix/power_perceiver/issues/182
• Convert 'Batch' object into dict needed for model - https://github.com/openclimatefix/power_perceiver/issues/184
• add e2e test - https://github.com/openclimatefix/power_perceiver/issues/185

Optional later

• add tests form component parts

What do you think about a pydantic model for the data / dict that goes into the model? I'm happy to tackle this if you want - https://github.com/openclimatefix/power_perceiver/issues/188

[jacobbieker]

Yeah, a pydantic model for the data would be great, if you could tackle that as you have more experience with that, that would be awesome! I've already started the refactor locally for the production model, I'll push it and add the GH actions/basic test tomorrow

[peterdudfield]

Im doing the github actions at the moment, so hopefully tomorrow you just push the test

[peterdudfield]

Current Status is

• ocf data pipes works, can get test data out and it looks ok i..e the correct shapes
• model weight son HF, and can load things well

memory issue

• Currently cant run batch_size=1 on my laptop or on Github actions. This makes it a bit hard to run things.

Potental solution

• Set up a EC2 instance with a GPU on it.
• See if test passes, which runs batch_size=1.
• Run forecast from ec2 manually, see if results are sensible.
• Set up Cron job to run forecast every 5mins, and see what happens
• later: set up infrastructure so forecast can be run using terraform code.

Would be interested in your thoughts @JackKelly (and @jacobbieker but I know you are on holiday at the moment)

[peterdudfield]

For batch size 2, running on a CPU

• Tried a 't3.2xlarge' which has 32 GB of memory. - Did not work
• Tried a 'm4.4xlarge' which has 64 GB of memory. - Did not work
• Tried a 'm5a.8xlarge' which has 128 GB of memory. - Did not work
• Tried a 'm5a.16xlarge' which has 256 GB of memory. Did not work

          RuntimeError: [enforce fail at alloc_cpu.cpp:73] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 50324972800 bytes. Error code 12 (Cannot allocate memory)

50324972800 bytes I think is 50GB

[peterdudfield]

Idea: Make sure floats are float32 not float64

[peterdudfield]

When reducing to float32,

• Tried a 'm4.2xlarge' which has 32 GB of memory. - Did not work RuntimeError: [enforce fail at alloc_cpu.cpp:73] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 50324972800 bytes. Error code 12 (Cannot allocate memory) Exactly the same error

• Tried 'm6a.24xlarge' which has 384 GB of RAM. - same error

Also ran for batch size 4, and got this error

RuntimeError: [enforce fail at alloc_cpu.cpp:73] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 100649945600 bytes. Error code 12 (Cannot allocate memory)

Which is double the precious amount of memory - so atleast batch sizing does work

[peterdudfield]

I managed to run it on a GPU, and batchsize 4 ran into a memory issue on a GPU with 16 GB.

[peterdudfield]

Current status,

• Can now run it on laptop. Its takes about 1 min to run a batch size of 4. This could be speed up if use multiple CPU cores for loading data, and could use GPU for inference. The fix was to only predict forward 8 PV systems, not 2056. Probably need to do a cost benefit of what this is worth doing
• There also seems to be a problem with some nans in the satellite predictions.
• Some GSPs which have no PV systems dont run, but these are that important, can also expand PV looking window
• Currently Satellite is 1 hour behind, which can be fixed with a hot fix, but the model shold really be re trained for this.
• Model should also been retrained to only use 4 channels.