Zig-zag forecast: Satellite variable delay

[peterdudfield]

We find that the forecasts over predict on the hour, and under predict on the half hour

This might becasue satellite image are available on the hour

Detailed Description

The forecasts at 00:15 have 15 minute delay of satellite data, but the forecast at 00:45 have 45 minutes of delay of data. This leads to some zig-sag effects in the 0 hour forecast

Context

• good to have smooth forecasts that look relastic

Possible Implementation

• [x] recursive CNN (done)
• [x] train on 30 mins and 60 mins gap in data (done)
• [x] only use satellite images from 60 minutes ago - unfortunately this did not help
• [ ] Check NWP shape
• [x] Check when Satellite data appears. It should be around 12.15.
• [ ] Save one batch to s3, so we can investigate

[peterdudfield]

Satellite data, at 12.07 UTC we have thte satellite data up to 12.00 UTC ready to use in AWS s3

[peterdudfield]

By looking at the batches, I can see that the satellite data has lots of zeros in it. batch for predicting at

• 16:00 has 15% zeros, these were the first 2 timesteps of data
• 15:30 has 62% zeros, these were the first 7 timesteps of data

This looks to be becasue the satellite data is limited to 2 hours. But actually we need 2.5 hours of satellite in s3. This is becasue if the prediction is made at 13.29, we need data from 11 to 12. 12 to 13 is assumed blank. So 11 to 13.29 is 2.5 hours. Will adjust this in Satip repo

[peterdudfield]

I've deployed a fix to the satellite consumer to keep more data, hopefully tomorrow we will solve the zig zag problem

[peterdudfield]

Think this is now sovled, bu waiting for a sunny day

[peterdudfield]

zig zag has now gone away