bug: PVnet suddenly drops

[peterdudfield]

Detailed Description

PVnet seems to drop all froecasts. After running it again in 30 mins it seems fine

Context

• see a blip in the 4 hour forecasts sometimes
• NWP consumer on dev was only fixed about for the first 9.45 run

[peterdudfield]

[peterdudfield]

Idea: could be missing data in satellite - that then gets filled in with zeros.

If we se again

• check satellite data latest, check raw
• check logs of pvnet2

[peterdudfield]

[peterdudfield]

But I cant seem to find this is PVnet - might have to look at logs in blend service

[peterdudfield]

[peterdudfield]

It was the 8.15 UTC run of PVnet 2

[peterdudfield]

Nothing unsual int he logs - https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252Fforecast_pvnet$252F/log-events/streaming$252Fforecast$252F2505f5d10f7b45509891cb2b804ea329$3Fstart$3D-43200000

[peterdudfield]

We see this in Braintree too, but not in other GSPs around it

[peterdudfield]

We do see this in GSPs 30 to 39 (inclusive) which is one batch of the PVnet run

[peterdudfield]

The batch was made and run at 8.17.51 to 8.17.57 UTC No satellite data was uploaded then NWP was uploaded but with no changes, at 8.17.36

[peterdudfield]

Another few today on dev

[dfulu]

We do see this in GSPs 30 to 39 (inclusive) which is one batch of the PVnet run

I think it would actually be batches of GSP IDs 1-10, 11-20, ... 31-40

[peterdudfield]

I worry its the NWP data upload that is interferrring in a small way. I could change the NWP data to run every 10,25,40,55 rather than 15,30,45,0, then its less likely to interferr

[peterdudfield]

We've seen more spikes in early gsp numbers that later ones, so this could correspond with NWP changing

[dfulu]

I think this was solved by the NWP download and the inversion fix. Closing for now