Missing/delayed priceinfo from entso-e

joe-ave commented 1 year ago

Hello Olli,

there seems to be missing/delayed priceinfo from the entso-e platform, probably something to do with this: https://transparency.entsoe.eu/news/widget?id=63ee08ae5092e77e4d3d7bf8

Could there be an idéa to have the possibility to manually enter the prices when this happen, since they exists in Nordpool?

//Jonas

stikstone commented 1 year ago

Is there now problem with updating prices at the ensoe or my side. Prices for tomorrow have not been updated. Tried restarting arska node. Did not help.

joe-ave commented 1 year ago

Is there now problem with updating prices at the ensoe or my side. Prices for tomorrow have not been updated. Tried restarting arska node. Did not help.

It looks like problem at the entso-e platform.

//Jonas

Olli69 commented 1 year ago

Hi Jonas,

Status update while writing this: Now Entso-E seems to have update prices for tomorrow.

Oh no - but thanks for letting me know about the delay and idea! That is something which is would be technically possible, but I think other, easier.., workarounds should considered first. A few points:

First, we should be able trust that Entso-E, the association for the cooperation of the European transmission system operators (TSOs), is able to provide up-to-date data on it's API. Last problems - if I remember right - have been fixed anyway same day. But it has been annoying to wait it...
Making a backup service from where Arska could fetch (manually entered) price info would be against Entso-E terms & conditions (for free use). So every user should enter prices manually - and know when there is a problem. We try to keep Arska simple to users, so entering prices manually cannot be normal procedure.
There could errors also in internet connection, hardware etc, so in critical applications rules and wirings should be error-prone.
So, let's hope that Entso-E get the problems solved, but keep your idea of manual update also in the mind. Olli

joe-ave commented 1 year ago

Hello Olli,

Yes we should be able to trust Entso-e as reliable provider, and it`s probably not very often they have this type of issues that they anyway solve the same day.

The stuff that I use Arska for works allso without Arska, only less cost efficiant. So in that meaning it’s failsafe.

So manually entering priceinfo is not needed very often, but more of a nice to have feature sometime in the future.

//Jonas

joe-ave commented 1 year ago

Hello Olli,

since the 28/5 when the pirceinfo from entso-e was totaly missing for that day the Arska-nodes does not seem to handle the new priceinfo for 29/5 in a correct way. For the older .93 version it pasted in the values in graph from 29/5 at the place för 28/5 but the variable readout this morning looked correct. For the newer .99 alfa 3 version ( latest from download ) it did not seemed to work at all, tried restart, power on/off and reset this morning ( 29/5 when there was ok pricedata) but variables still seemed wrong and nothing on the graph. I´ll reflash the .99 version later today.

//Jonas

stikstone commented 1 year ago

Yep, At least .99beta1 and beta2 have this same problem. No price info since of yesterday, and flashing did not seem to help.

Olli69 commented 1 year ago

Hi Jonas, It seems that ENTSO-E is totally missing prices for Sunday 28.5.2023 and thats why Arska price query crashes on it. It is not currently build for situations that whole day is missing from Entso-E time-series (That should not happen from Entso-E's side...) and making quick fixes could break the logic. I have a look anyway.

Olli69 commented 1 year ago

Hi Jonas, In addition to Entso-E time series problem there was a typo in the latest development version alfa3. Now this is fixed in alfa4 and getting SE3 prices seem to work - very quick testing. It still gives error messages and no diagram, because of missing data on Entso-E side but current price should be available in the variables.

Olli

joe-ave commented 1 year ago

Hello Olli,

thanks!, I´ll try it out later today.

//Jonas

stikstone commented 1 year ago

.99 alpha4 worked for me, and i got the price info back, everything working now.

joe-ave commented 1 year ago

Now it works again!

//Jonas

joe-ave commented 1 year ago

Hello,

there seems to be something wrong with the pricefile today from Entso-e, the price seems to be there when I look at he serial monitor but the time is probably wrong ( dump from some of the lines ):

Combined line:22 Combined line:18.03</price.amount> Invalid timeSeries index -565974 Combined line:23 Combined line:17.73</price.amount> Invalid timeSeries index -565974 Combined line:24 Combined line:17.14</price.amount> ***Invalid timeSeries index -565974 Combined line:

//Jonas

joe-ave commented 1 year ago

Hello Olli,

just want to document that priceinfo from entso-e for the Nordic countries is missing since yesterday for today 2023-09-28, it´s missing at their webpage also so not just a query thing.

//Jonas

Olli69 commented 1 year ago

Hej Jonas, Thanks for recording! Yes, now Entso-E publish announcement about that. Elering seemed to publish prices in time - unfortunately Scandinavian price areas are not included,.

Olli

joe-ave commented 1 year ago

Hello Olli,

just documenting that hour 35 tomorrow ( 11:00 ) has a pricevalue of 0 for Sweden and Norway when I look at the serial printout. It should be in the area of about 11.7 cent/kw. Denmark and Finland looks ok. Maybee they will correct tomorrow.

//Jonas

Olli69 commented 1 year ago

Hi Jonas @joe-ave , Thanks for the note! It corrected the value later on automatically but I catched the error and fixed it. If you have your own build and try to test it, you could replace functions getElementValue and read_http11_linewith a new fixed versions below. These need to be tested more before wider release.

Olli

Edited 16.11.23 20:07 : function read_http11_line tuned

String getElementValue(String outerXML)
{
  int s,e;
  s = outerXML.indexOf(">", 0)+1;
  e = outerXML.indexOf("</");
  if (e ==-1) // end tag missing, try to read to the end
    e = outerXML.length();
  if (s>e) // no start tag found in the beginning (but the end), get from the start of the string
    s = 0;
     return outerXML.substring(s, e);
}


String read_http11_line(WiFiClientSecure *client_https)
{
  String line;
  String line2;
  if (client_https->available())
  {
    line = client_https->readStringUntil('\n');
    if (line.charAt(line.length() - 1) == 13)
    {
      line = line.substring(0, line.length() - 1); // remove cr 
      if (client_https->available())
      {
        line2 = client_https->readStringUntil('\n');
        if (is_chunksize_line(line2))
        { // skip this line and read  next one
          Serial.print("Skipped line:");
          Serial.println(line2);
          if (client_https->available())
          {
            line2 = client_https->readStringUntil('\n');
          }
          else
            line2 = "";
        }
        return line  + line2;
      }
      else
        return line;
    }
    else
      return line;
  }
  return "";
}```

joe-ave commented 12 months ago

Hello Olli,

just documenting that priceinfo for tomorrow (2023-12-05) for the nordic countries is still missing at 22:00, probably it will be corrected tomorrow. No info at Entso-e site.

//Jonas

joe-ave commented 1 month ago

Hello Olli,

for info entso-e seem to be having some problem again, both today 2024-10-05 and tomorrow 2024-10-06 prices seem so be missing for all countries. No info at entso-e site.

//Jonas

joe-ave commented 1 month ago

Hello Olli,

Documenting that some priceinfo are missing for tomorrow 2024-10-26 in SE1 to SE4, it looks like 14:00 to 23:00, other countries seem to work.

request sent headers received Waiting the document Debug before get_price_data_entsoe 1729980000, 4848 period_start: 1729720800 record_start: 1729807200 - period_end: 1729980000 ENTSO-E price data missing future prices, end_reached 0, price_rows 38 Price query failed

//Jonas

Olli69 commented 1 month ago

Hi @joe-ave , Thanks for reporting! I had a look and could repeat that but did not find a fix yet. It look like a buffer overflow (exists when prices of 3 days are queried) or something else with communication, but strange if that only occurs with SE areas. Anyway this should have automatically fixed before the day with missing prices starts (Arska retries later and then result set from Entso-E contains only prices of 2 days)

EDIT: Now it should be fixed in the code (not in the binary yet) dev-1.1/src/main.cpp where I added extra wait, if read buffer is empty:

    // If read buffer is empty, wait for a while if new data is coming
    if (!client_https.available()) {
      Serial.println("Waiting new stuff to the buffer");
      delay(1000);
    }

EDIT 2 (Oct 27 2024): The fix is now included in binary 1.1.0-beta10 available on Arska installation page.

Netgalleria / arska-node

Missing/delayed priceinfo from entso-e #29