Strange results for CNN model

[peterdudfield]

When first running the model we get some odd results. Often the predictions are negative

select * from forecast
join forecast_value on forecast.id = forecast_value.forecast_id
join location on location_id = location.id
join model on model_id = model.id
where model.name  = 'cnn'
and gsp_id = 1

• [x] Save one batch to s3, so we can see what is going into the model
• [x] check same values are normalized in training and in inference, for sat and nwp
• [x] Notice PV data has very little data in it
• [ ] Investigate effect of little PV data
• [x] Check no difference between examples with PV data and examples without
• [x] Sun is None in inference batch, is it in training? yes
• [ ] Run inference using one of the pre preapred batches using the model and check results are ok, v15
• [ ] Chekc Satellite compression is the same in training and inference
• [ ] Check for any Ints where they should be floats
• [ ] check tensor floats 32 / 64 is the same in training and inference
• [ ] plot satellite image in prediction and training batch, see if there is anything different
• [x] check model is loaded - https://github.com/openclimatefix/nowcasting_forecast/pull/82

• [ ] should we run the model in 'eval' mode
• [ ] does the model post processing mucking up the results? https://github.com/openclimatefix/nowcasting_forecast/pull/82 might help with this - live logs
• [x] run model locally with batchml from s3. Check model results are reasonable. i.e non negative. If so post-processing could be mucking it up All model results are postive
• [ ] ...

[jacobbieker]

• [x] Sun is None in inference batch, is it in training?
• [ ] Is there a difference between v15 satellite and now? -> POssible difference with the new compression

[peterdudfield]

We think we've sovled it

https://github.com/openclimatefix/nowcasting_forecast/blob/main/nowcasting_forecast/models/utils.py#L244

model.load_model(use_hf=use_hf).eval()

should be

model = model.load_model(use_hf=use_hf).eval()