Missing data

[MarcinKonowalczyk]

I'm comparing the data streams between bmreports api and iris and I found that there are some messages missing from the iris data stream. As an example here is the plot of Physical Notification from T_LLWSD-1 on 28/02/24 (aka today):

You can see that there is a discrepancy on SP 24 (11:30-12:00 BST). When consulting the bmreports' "Search by BM Unit" website I find that that data is indeed correct. When, in turn I consult the elexon's "Balancing Mechanism data by BMU" I find that the relevant SP is missing:

---

This is of course highly troubling since BMRS is scheduled to be switched off Mid May 2024 (NOTE: <- elexon.co.uk does not work in Safari)

[MarcinKonowalczyk]

At some point between 14:30 and 16:00 (aka now) the data on [elexon's website(https://bmrs.elexon.co.uk/balancing-mechanism-bmu-view?bmuId=T_LLWSD-1&activeTab=Physical&startDate=2024-02-28T00%3A00%3A00.000Z&endDate=2024-02-28T14%3A30%3A00.000Z) got updated:

This is still a serious issue since no realtime data was available through iris for at least 2.5 hours, while it was clearly available through the old data service.

[beccar97]

Hi Marcin,

Thanks for getting in touch 🙂

There was a planned data outage across both the BM Reports site and the Insights solution today from 10am until 1pm, which was extended until 2pm. This was due to some work behind the scenes relating to the upstream data connections that feed both systems. Between 2pm and 3pm both systems were recovering from this outage, and importing data from that period. You seem to have been incredibly lucky (or unlucky) in when you looked for this, as the file containing the PN data for SP 24 was ingested into the Insights solution, and sent out via IRIS, at 14:43 today, and BM Reports would most likely only have received the same file shortly before this.

The data outage impacted the majority, but not all datasets for the Insights solution, and IRIS continued to make data available throughout the outage period. The IRIS archive can be used to confirm all the data sent out via IRIS, if you ever wish to check on the time that data was made available through this data stream.

[MarcinKonowalczyk]

@beccar97 hi :)) thanks for confirming and a very quick response! much appreciated!

just a quick question: i have not seen any information about the outage a priori (in fact, i still don't) on system events. is that a good place to get information about the outages in the future? ideally i'd love an api endpoint to integrate information about the outages into our system. maybe the SYSWARN messages?