search

elexon-data / iris-clients

Clients for connecting to the Insights Real-time Information Service
MIT License
17 stars 7 forks source link

Reproducing MELS data from message stream #21

Closed MarcinKonowalczyk closed 6 months ago

MarcinKonowalczyk commented 8 months ago

I'm having issues reproducing some of the data which available through your website. There are a couple of cases but the simplest one i'd like to ask about is MELS for E_LYNE1 between the times 14/03/24 01:00 -> 02:00. The data from your website can be seen here (also see screenshot below) and the relevant records I've received are attached below.

How does the data on the website follow from these records? If there is any documentation you could point me to which explains the process of "stitching" together MELS I would be very grateful.

screenshots

MELS on elexon website:

Screenshot 2024-03-14 at 14 18 50

MELS data as put together from the messages

plot_8

animation

animate

records

E_LYNE1_MELS.json

rit-bit commented 7 months ago

Hi @MarcinKonowalczyk, thanks for raising this issue. We are currently looking into it and will update this issue when we have more information

rit-bit commented 6 months ago

Latest update:

Thank you @MarcinKonowalczyk for identifying an edge case which occurs when redeclarations are submitted where both of the following are true:

On the second point, our logic assumes this doesn't happen because the extrapolation between the two levels would be ambiguous. The extra spike in the example above is caused by splitting at the settlement period boundaries (which is necessary for the sequencing calculations) since we assume levelFrom and levelTo have the same value.

The data submitted has not matched our assumptions, and Elexon is looking into why this is the case. In the meantime we will not change the way we process MILS/MELS data until we receive clarification, so I'm closing this issue.