Recently we had a spike in the predictions in our production system. This was due to the satellite going down for 30 minutes. The problem was that the model, which uses 90 minutes of history filled in the missing 30 minutes with NaNs.
We should update the check_model_inputs_available() function to check not just if the latest satellite time stamp is acceptable, but the whole period. This change will need to reflect that sometimes we use 15-minutely satellite data and infill it. It will also need to reflect that datapipes is currently configured to infill a couple of satellite frames if for example the RSS went down for 5-10 minutes. So won't be entirely trivial
peterdudfield
commented
2 months ago
Recently we had a spike in the predictions in our production system. This was due to the satellite going down for 30 minutes. The problem was that the model, which uses 90 minutes of history filled in the missing 30 minutes with NaNs.
A more thorough report is here
We should update the
check_model_inputs_available()function to check not just if the latest satellite time stamp is acceptable, but the whole period. This change will need to reflect that sometimes we use 15-minutely satellite data and infill it. It will also need to reflect that datapipes is currently configured to infill a couple of satellite frames if for example the RSS went down for 5-10 minutes. So won't be entirely trivial