search

openclimatefix / power_perceiver

Machine learning experiments using the Perceiver IO model to forecast the electricity system (starting with solar)
MIT License
7 stars 1 forks source link

Maybe we shouldn't train the unet when training the rest of the model #125

Closed JackKelly closed 2 years ago

JackKelly commented 2 years ago

Maybe train the unet once. Then don't train it at all.

This should speed up training of the "PV" parts of the model, and might actually result in better performance on the PV stuff because the PV loss won't be "diluted" by the satellite loss.

And the code to use different DataSource combos will still be useful so we can train on images centered over PV systems

JackKelly commented 2 years ago

Maybe move all the satellite loss calculations into SatellitePredictor, and call satellite_predictor.validation_step from the FullModel?

JackKelly commented 2 years ago

Huh! This doesn't actually speed up model training that much, because it appears that data loading is now a bottleneck! More workers would almost certainly help.

But, very early indications are that this will help improve forecast skill for GSP and PV :slightly_smiling_face:

Maybe try, like, 5 workers but 1 GPU.

Even better would be to use uint8 for satellite (#129) (to save RAM) and then use 5 workers per GPU, and 2 GPUs.

JackKelly commented 2 years ago

now using uint8 for satellite, and 4 data loader workers per GPU. Works well!