Shows wrong prices, brobably bug in currency conversion EUR / NOK

[tronde-ams]

Aidon 3p IT / generic ESP32 / NO1 / own api key See screenshots. Prices are ex VAT. Json data shows same values as GUI

I noticed the same error yesterday.

Re-boot does not help, but it does help to delete the api key and restart and then re-install the key and re-start again. The wrong prices are 11.76 x to low, which seems to be pretty close to EUR / NOK exchange rate, so guess some gremlins are hiding there?

.

.

[atle-dahlman]

I have the same thing on both my readers, where one is v.2.2.21 and the other is v.2.2.23. Looks like something happened on October 30'th at 1.56.

[atle-dahlman]

[ArnieO]

Entso-E:

[ArnieO]

NO-1, multiplier 1.00. Seems to be correct?

[atle-dahlman]

In the web interface on the reader, everything is now correct. Seemed like ENTSO-E might hav fixed something. Are there anyway to refresh the values in Home Assistant? The values received by MQTT for the current day are still the same values as received earlier.

[atle-dahlman]

Spoke to soon. Mine is still not correct.

[atle-dahlman]

Hmmm. Removed the API-key from the settings, and the prices corrected themselves, but then....

Inserted the key again, and everything seems correct. For now...

[ArnieO]

Hmmm. Removed the API-key from the settings, and the prices corrected themselves, but then.... !

When you use the key, data is fetched directly from Entso-E. Without the key you will read from our intermediate server, which fetches from Entso-E only once per day. So if Entso-E makes changes, it is not reflected on the data from our intermediate server.

IIRC, you force a reread from Entso-E by rebooting the device.

[atle-dahlman]

Ok, I got that. What's with the error message when I removed the key? If data is fetched from your intermediate server, is there supposed to be an error message?

[ArnieO]

If data is fetched from your intermediate server, is there supposed to be an error message?

No, it should not, so that is strange. I tied to reboot my device now, and do not see that. NO-3, multiplier 1.25

Does it persist if you reboot?

[atle-dahlman]

I haven't tried. I re-entered my API-key, and now everything is fine.

[tronde-ams]

I'm quite certain my problem is related to fetching or applying the correct exchange rate. The graphics is correct, but the numeric values are wrong, so prices was fetched from entso-e. My multiplier is 1.00.

The correct prices should have been 11.76 times higher, and at that time google said EUR / NOK was 11.82. We have some rounding errors, and maybe different time involved here.

[ArnieO]

I'm quite certain my problem is related to fetching or applying the correct exchange rate. The graphics is correct, but the numeric values are wrong, so prices was fetched from entso-e. My multiplier is 1.00.

The correct prices should have been 11.76 times higher, and at that time google said EUR / NOK was 11.82. We have some rounding errors, and maybe different time involved here.

Huh, strange. Let us hear what @gskjold says when he gets around to it.

[tronde-ams]

Maybe a test version showing which conversion rate is used and what time that rate was fetched can help us?

[gskjold]

If you enable debugger with level "DEBUG", it will output exchange rate in debugger. Maybe there is something about DNBs currency API that has changed, not sure. But 1.0 exchange rate is default is nothing is found. I don't currently have any hardware in front of me to test, but I will have a look at it when I do.

[tronde-ams]

Telnet was not friendly tonight - disconnected just before new hour. Regardless of this, the error was there at least between 01:00 and 02:00. New hour at 02:00 did not change anything, but deleteing an re-installing the api key got the prices right again. It does still seem to be related to missing exchange rate NOK / EUR, so I think that part of the code should get a closer look.

[tronde-ams]

If you enable debugger with level "DEBUG", it will output exchange rate in debugger. Maybe there is something about DNBs currency API that has changed, not sure. But 1.0 exchange rate is default is nothing is found. I don't currently have any hardware in front of me to test, but I will have a look at it when I do.

I tried to get data via telnet at 15:00, but I can't see anything about exchange rate, I think? Everything updated as it should, but I need to figure out how to catch the relevant information.

(D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Received valid DLMS at 18 (D) Serving /tariff.json over http... (D) Serving /dayplot.json over http... (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Its time to update data storage (D) Received valid DLMS at 18 (D) Its time to update data storage (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Its time to update data storage (D) Received valid DLMS at 18 (D) Its time to update data storage (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Its time to update data storage (D) (AmsDataStorage) Last day update: 1698843611 (D) (AmsDataStorage) Last month update: 1698793212 (I) (AmsDataStorage) Usage for hour 14: 3016 - 0 (I) Saving data (I) (EnergyAccounting) New local hour 15 (D) Received valid DLMS at 18 (D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18 (D) Serving /data.json over http... (D) Received valid DLMS at 18

My setting: .

[ArnieO]

Try "Verbose". It will output all there is (while "Debug" only outputs a subset of debug messages).

[gskjold]

Exchange rate is only fetched when prices are fetched. You will have to disable price in config, reboot, connect to telnet and then enable price fetching. It should then log URL for both entso-e and dnb as well as the exchange rate

[tronde-ams]

Exchange rate is only fetched when prices are fetched. You will have to disable price in config, reboot, connect to telnet and then enable price fetching. It should then log URL for both entso-e and dnb as well as the exchange rate

If I do so, the prices has always been corrected, so it will not help figuring out why they was wrong.

We need some information about the exchange rate used when we actually see that the prices are wrong. I have not managed to figure out when the error occurs, I have only seen that "now is something wrong".

It is not possible for me at the moment to have long-time logging.

[gskjold]

Looking through the code, the only way this can happen is if there is a problem fetching the exchange rate. There is an issue here that it will keep the 1.0 multiplier for a while, but I'll add a fix for that. Minor adjustments in attached firmware: esp32s2.zip esp32.zip esp8266.zip esp32c3.zip esp32solo.zip

[mortenm22]

I had the same error this morning. With Enable price fetch from remote server checked and token from Entso-e. Version d82e705.

[gskjold]

Assumption is that it is not fixed. I would like to see both screenshot and energyprice.json when this happens

[mortenm22]

I have only the screenshot from my phone 3 days ago. I can try to catch the json next time it happens.

[gskjold]

Thanks, I'd really like to compare with the json. Crossing fingers someone are able to catch it

[tronde-ams]

This is FW d82e705 on generic Esp32.

Don't know whether it's related to the original topic, but I don't see a need for a new issue just now.

Why does it loose prices if there is an issue with the price api in the period between 13:00 - 13:00? They should have been stored in some way, I think?

/energyprice.json says null all over, except currency NOK

The usual tricks with reboot or deleting / re-installation of api key changes nothing. Uptime is 2 days, so no reboot since new prices at about 13:00. The prices was OK before midnight and disappered some time after that, but before 02:30.

[mortenm22]

It looks like it is after boot of AMS reader. This night AMS reader booted 02:02. I rebooted this morning and prices are correct again.

[13.11.2023] [02:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.9632,"1":0.9466,"2":0.9077,"3":0.9487,"4":0.9909,"5":1.1326,"6":1.4156,"7":1.4514,"8":1.3702,"9":1.1598,"10":1.0694,"11":1.0663,"12":0.9963,"13":0.9513,"14":0.9340,"15":0.9532,"16":0.9498,"17":0.8710,"18":0.8314,"19":0.7636,"20":0.7101,"21":0.6436,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.6436,"max":1.4514,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [03:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.0797,"1":0.0764,"2":0.0799,"3":0.0835,"4":0.0954,"5":0.1192,"6":0.1223,"7":0.1154,"8":0.0977,"9":0.0901,"10":0.0898,"11":0.0839,"12":0.0801,"13":0.0787,"14":0.0803,"15":0.0800,"16":0.0734,"17":0.0700,"18":0.0643,"19":0.0598,"20":0.0542,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.0542,"max":0.1223,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [04:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.0764,"1":0.0799,"2":0.0835,"3":0.0954,"4":0.1192,"5":0.1223,"6":0.1154,"7":0.0977,"8":0.0901,"9":0.0898,"10":0.0839,"11":0.0801,"12":0.0787,"13":0.0803,"14":0.0800,"15":0.0734,"16":0.0700,"17":0.0643,"18":0.0598,"19":0.0542,"20":-127.0000,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.0542,"max":0.1223,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [05:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.0799,"1":0.0835,"2":0.0954,"3":0.1192,"4":0.1223,"5":0.1154,"6":0.0977,"7":0.0901,"8":0.0898,"9":0.0839,"10":0.0801,"11":0.0787,"12":0.0803,"13":0.0800,"14":0.0734,"15":0.0700,"16":0.0643,"17":0.0598,"18":0.0542,"19":-127.0000,"20":-127.0000,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.0542,"max":0.1223,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [06:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.0835,"1":0.0954,"2":0.1192,"3":0.1223,"4":0.1154,"5":0.0977,"6":0.0901,"7":0.0898,"8":0.0839,"9":0.0801,"10":0.0787,"11":0.0803,"12":0.0800,"13":0.0734,"14":0.0700,"15":0.0643,"16":0.0598,"17":0.0542,"18":-127.0000,"19":-127.0000,"20":-127.0000,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.0542,"max":0.1223,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [07:00:00] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":0.0954,"1":0.1192,"2":0.1223,"3":0.1154,"4":0.0977,"5":0.0901,"6":0.0898,"7":0.0839,"8":0.0801,"9":0.0787,"10":0.0803,"11":0.0800,"12":0.0734,"13":0.0700,"14":0.0643,"15":0.0598,"16":0.0542,"17":-127.0000,"18":-127.0000,"19":-127.0000,"20":-127.0000,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.0542,"max":0.1223,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}} [13.11.2023] [07:31:40] {"id":"XX:XX:XX:XX:XX:XX","prices":{"0":1.1326,"1":1.4156,"2":1.4514,"3":1.3702,"4":1.1598,"5":1.0694,"6":1.0663,"7":0.9963,"8":0.9513,"9":0.9340,"10":0.9532,"11":0.9498,"12":0.8710,"13":0.8314,"14":0.7636,"15":0.7101,"16":0.6436,"17":-127.0000,"18":-127.0000,"19":-127.0000,"20":-127.0000,"21":-127.0000,"22":-127.0000,"23":-127.0000,"24":-127.0000,"25":-127.0000,"26":-127.0000,"27":-127.0000,"28":-127.0000,"29":-127.0000,"30":-127.0000,"31":-127.0000,"32":-127.0000,"33":-127.0000,"34":-127.0000,"35":-127.0000,"36":-127.0000,"37":-127.0000,"min":0.6436,"max":1.4514,"cheapest1hr":"2023-11-13T22:00:00Z","cheapest3hr":"2023-11-13T20:00:00Z","cheapest6hr":"2023-11-13T17:00:00Z"}}

[gskjold]

Thanks @mortenm22 , I'm assuming you are still having a ENTSO-E API key configured?

[mortenm22]

Yes, I am using my entso-e token.

[gskjold]

Testing some adjustments: esp32solo.zip esp32.zip esp8266.zip esp32c3.zip esp32s2.zip

[tronde-ams]

Buggy again. Was OK before midnight. Don't remember if I checked just after, but now at 03:30 it is like this. FW d4feed3 and generic ESP32 with own api key.

All prices says null

[gskjold]

I had the same happen to my test unit during the night. The logs show that Norges Bank API was unavailable when it was trying to fetch the exchange rate. I will make adjustments to keep existing exchange rate when this happens

[tronde-ams]

Is there any good reason why it skips all prices? I would have expected it to keep the existing prices fetched at about 13:00, even if it search for new prices long time before 13:00?

[mortenm22]

I did not get json string with prices between 03:00 and 05:00

[gskjold]

The current implementation updates the exchange rate every day, discarding the old one. And when it fails, it has no exchange rate. Prices are stored with original currency (EUR)

[ArnieO]

The current implementation updates the exchange rate every day, discarding the old one. And when it fails, it has no exchange rate. Prices are stored with original currency (EUR)

So it stores prices in EUR, and exchange rate is not stored but fetched each time it is needed for displaying the graph?

[gskjold]

No, the exchange rate is kept for the whole day, but invalidated after midnight so that it will fetch new exchange rate when the date change. Also, this only applies for when you are using ENTSO-E API key

[ArnieO]

No, the exchange rate is kept for the whole day, but invalidated after midnight so that it will fetch new exchange rate when the date change. Also, this only applies for when you are using ENTSO-E API key

💡 Thank you for clarifying that.

[gskjold]

New test firmware, preserving previous exchange rate if unable to update: esp32solo.zip esp32.zip esp32s2.zip esp8266.zip esp32c3.zip

[atle-dahlman]

I thought everything was ok, but it wasn't. I'm now trying without my ENTSO-E key. Hope that helps.

[Mithrox]

I have to restart every day still, on latest firmware. How does it work if you do not use ENTSO-E key?

I am in Norway and use price in NOK.

Is the problem that it cannot pickup the correct EUR/NOK conversion rate at midnight? Would it be an option to have it pick this up one hour later or something? It would give a slight deviation, but it's not the end of the world, better than having wrong price whole time.

[ArnieO]

I thought everything was ok, but it wasn't. I'm now trying without my ENTSO-E key. Hope that helps.

It would be helpful if you could try the test code here above, with Entso-E key - to help us verify if the fix did what was intended.

[ArnieO]

I have to restart every day still, on latest firmware. How does it work if you do not use ENTSO-E key?

I am in Norway and use price in NOK.

Is the problem that it cannot pickup the correct EUR/NOK conversion rate at midnight? Would it be an option to have it pick this up one hour later or something? It would give a slight deviation, but it's not the end of the world, better than having wrong price whole time.

It seems that the API at Norges Bank no longer responds during the night. This is relatively new. The test code in https://github.com/UtilitechAS/amsreader-firmware/issues/665#issuecomment-1819153526 should fix that by using previous exchange rate if API does not answer. Please try it!

If you do not enter Entso-E key, data will be read from our intermediate server instead.

[mortenm22]

I use ENTSO-E key and firmware 4f68258. The prices have been correct with this version.

[ArnieO]

I use ENTSO-E key and firmware 4f68258. The prices have been correct with this version.

Good to hear, thank you for feedback!

[tronde-ams]

My generic ESP32 with own API-key and 4f68258 appears to behave itself too as far as I can tell. Uptime 10 days, and not spotted any discrepancies.