Delays in PN, BOD, MEL, FUELINST stream endpoints

Describe the issue

The endpoint streams for Physical Notification(PN), Bid Offer Data(BOD), and Maximum Export Limit(MEL) are having significant delays while trying to get the latest data. Suppose the time is 9:00 AM (UTC), with the settlement period SP21 starting to deliver. Now this point in time is the Gate Closure for SP23, so the physical data should be available. But on using Insights API, this data arrives at over 15 minutes of delay(after 9:15 AM). The same data can be fetched using old BMRS and is also displayed on the site within a couple of minutes of lag. A similar delay issue is noticed in the FUELINST stream where the latest iteration isn't obtained until over 15 minutes are lapsed from real-time.

At first, this wasn't the case as the Insights API had a nominal delay of a couple of minutes(no more than 5 minutes). These delays have been observed over the last couple of weeks.

Steps to Reproduce

Steps to reproduce the behavior:

Go to https://bmrs.elexon.co.uk/api-documentation/endpoint/datasets/PN/stream
Fetch the latest data i.e for settlement period +2 than the present one(if SP21 is delivering, then SP23 data should be available) for any selected BM units.
We get an empty response.

Expected behaviour

After the gate closure, the physical data of the settlement period should be made available.

Observed behaviour

Even after 15 minutes of gate closure, there is no data.

Checks

Please answer the following questions for yourself before submitting an issue.

[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed
[x] I'm reporting the issue to the correct repository
[x] I'm reporting a problem and not a feature request
[x] I have added the correct labels
[x] I have added screenshots or other relevant evidence (if applicable)

elexon-data / insights-issues