Upstream generation data is incorrect

[Samreay]

Describe the issue

Hi team, I think this will be another one to send upstream to get fixed.

Steps to Reproduce

Steps to reproduce the behavior:

1. Go to https://developer.data.elexon.co.uk/api-details#api=prod-insol-insights-api&operation=get-datasets-fuelhh-stream
2. Click on request CCGT generation for 2023-01-24 (full request below in the relevant section).
3. Inspect JSON output
4. See error

Expected behaviour

CCGTs have long ramping times and should be fairly consistent one SP to the next.

Observed behaviour

Some settlement periods have erroneous data. In the example below, we have smooth generation at around 23GW for many settlement periods, then suddenly almost 20GW of generation vanishes at 11am, but the very next settlement period we're back to 22GW. This simply isn't possible with CCGTs.

[{
    "dataset": "FUELHH",
    "publishTime": "2023-01-24T12:30:00Z",
    "startTime": "2023-01-24T12:00:00Z",
    "settlementDate": "2023-01-24",
    "settlementPeriod": 25,
    "fuelType": "CCGT",
    "generation": 23185
}, {
    "dataset": "FUELHH",
    "publishTime": "2023-01-24T12:00:00Z",
    "startTime": "2023-01-24T11:30:00Z",
    "settlementDate": "2023-01-24",
    "settlementPeriod": 24,
    "fuelType": "CCGT",
    "generation": 22841
}, {
    "dataset": "FUELHH",
    "publishTime": "2023-01-24T11:30:00Z",
    "startTime": "2023-01-24T11:00:00Z",
    "settlementDate": "2023-01-24",
    "settlementPeriod": 23,
    "fuelType": "CCGT",
    "generation": 3896
}, {
    "dataset": "FUELHH",
    "publishTime": "2023-01-24T11:00:00Z",
    "startTime": "2023-01-24T10:30:00Z",
    "settlementDate": "2023-01-24",
    "settlementPeriod": 22,
    "fuelType": "CCGT",
    "generation": 22018
}, {
    "dataset": "FUELHH",
    "publishTime": "2023-01-24T10:30:00Z",
    "startTime": "2023-01-24T10:00:00Z",
    "settlementDate": "2023-01-24",
    "settlementPeriod": 21,
    "fuelType": "CCGT",
    "generation": 22637
}]

HTTP requests/responses

GET https://data.elexon.co.uk/bmrs/api/v1/datasets/FUELHH/stream?settlementDateFrom=2023-01-24&settlementDateTo=2023-01-25&fuelType=CCGT HTTP/1.1

Cache-Control: no-cache

Additional context

Add any other context about the problem here, e.g. the browser you are using (if applicable).

Checks

Please answer the following questions for yourself before submitting an issue.

• [x] I checked the documentation and found no answer
• [x] I checked to make sure that this issue has not already been filed
• [x] I'm reporting the issue to the correct repository
• [x] I'm reporting a problem and not a feature request
• [x] I have added the correct labels @zghanty I believe the template issue markdown needs this line removed still.
• [x] I have added screenshots or other relevant evidence (if applicable)

[trader-short]

Just to add some more info, something similar was observed in the case of NUCLEAR and WIND fuel types. There was a reported drop of around 4GW and then the very next SP, it was recovered.

Taking a look at the five-minute iterations, noticed that for SP23, only the first interval reported a non-zero value and after 11:05 there was some data outage. FUELHH reported values were evaluated with just the initial five minutes of data, 23377MW divided by 6 giving the erratic 3896 MW for CCGT.

[zghanty]

@Samreay @trader-short : Sorry for the delay, NGESO colleague confirmed " For issue 15 related to missing generation data on 24 January 2023 we have identified this was probably caused by a metering fault, and these files are rarely updated post event"

We need to think about alerting/sign-posting when we are aware of such issues.

[Samreay]

That is frustrating to hear from ESO. Publishing data comes with a responsibility to make sure its correct and fix mistakes, surely. Its hard to automate checks on these numbers in many cases (apart from simple things like "nuclear shouldnt drop to zero randomly", so I'm worried that continuing to use this data source in the future represents a risk if there aren't any checks and monitoring, nor correcting, done on ESOs side. I'll think this over and see if any easy wins come to mind. Thanks for the update, Zaahir :)