MILS/MELS overlapping data between 08/06 and 14/06

Describe the issue

For MILS/MELS data with a notification time of between around 15:00 UTC on 08/06/2023 and around 14:45 UTC on 14/06/2023, the Insights API is returning responses with overlapping declarations.

Since the Insights frontend uses this API, the following frontend pages are also affected:

Steps to Reproduce

Steps to reproduce the behavior:

Send a GET request to https://data.elexon.co.uk/bmrs/api/v1/datasets/MILS?from=2023-06-14T06%3A30Z&to=2023-06-14T07%3A00Z&bmUnit=2__BFLEX001
Observe that there are MILS declarations for the BMU 2__BFLX01 covering the following overlapping time periods: a. 06:30 - 07:00 b. 06:32 - 07:00 c. 06:33 - 06:35 d. 06:35 - 07:00 e. 06:37 - 07:00 f. 06:39 - 07:00

Expected behaviour

No overlaps are observed, as for each time period the most recent declaration is taken over all of the previous declaration.

Observed behaviour

Overlaps are observed as described in "Steps to Reproduce" above, without clarity as to which was the correct final MIL/MEL.

HTTP requests/responses

For the https://data.elexon.co.uk/bmrs/api/v1/datasets/MELS?from=2023-06-14T02%3A00Z&to=2023-06-14T02%3A30Z&bmUnit=T_DRAXX-3 example, this is the response body:

{
   "data":[
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":8,
         "timeFrom":"2023-06-14T02:30:00Z",
         "timeTo":"2023-06-14T03:00:00Z",
         "levelFrom":645,
         "levelTo":645,
         "notificationTime":"2023-06-14T02:08:07Z",
         "notificationSequence":104326557,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:13:00Z",
         "timeTo":"2023-06-14T02:30:00Z",
         "levelFrom":645,
         "levelTo":645,
         "notificationTime":"2023-06-14T02:08:07Z",
         "notificationSequence":104326557,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:11:00Z",
         "timeTo":"2023-06-14T02:13:00Z",
         "levelFrom":620,
         "levelTo":645,
         "notificationTime":"2023-06-14T02:08:07Z",
         "notificationSequence":104326556,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:10:00Z",
         "timeTo":"2023-06-14T02:30:00Z",
         "levelFrom":620,
         "levelTo":620,
         "notificationTime":"2023-06-14T02:07:07Z",
         "notificationSequence":104326505,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:08:00Z",
         "timeTo":"2023-06-14T02:10:00Z",
         "levelFrom":645,
         "levelTo":620,
         "notificationTime":"2023-06-14T02:07:07Z",
         "notificationSequence":104326503,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:05:00Z",
         "timeTo":"2023-06-14T02:30:00Z",
         "levelFrom":645,
         "levelTo":645,
         "notificationTime":"2023-06-13T21:58:41Z",
         "notificationSequence":104318494,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      },
      {
         "dataset":"MELS",
         "settlementDate":"2023-06-14",
         "settlementPeriod":7,
         "timeFrom":"2023-06-14T02:00:00Z",
         "timeTo":"2023-06-14T02:05:00Z",
         "levelFrom":590,
         "levelTo":645,
         "notificationTime":"2023-06-13T21:58:41Z",
         "notificationSequence":104319612,
         "nationalGridBmUnit":"DRAXX-3",
         "bmUnit":"T_DRAXX-3"
      }
   ]
}

Checks

Please answer the following questions for yourself before submitting an issue.

[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed
[x] I'm reporting the issue to the correct repository
[x] I'm reporting a problem and not a feature request
[x] I have added the correct labels
[x] I have added screenshots or other relevant evidence (if applicable)

elexon-data / insights-issues