search

UtilitechAS / amsreader-firmware

ESP8266 and ESP32 compatible firmware to read, interpret and publish data to MQTT from smart electrical meters, both DLMS and DSMR is supported
Other
382 stars 73 forks source link

Nice to have #680

Closed Farjestad closed 7 months ago

Farjestad commented 11 months ago

Describe the solution you'd like Hi in Sweden we need to pay for (Energiavgift) is based on the highest average hourly consumption in a month. So if you could see for each hour an expected result depending on the consumption, and save the highest of the month. We also have a High-load charge based on the same principle, but only between November and March on weekdays between 06:00 - 18:00

ArnieO commented 11 months ago

Interesting, please elaborate on how this is calculated!

We have something similar in Norway, already implemented in the code. Please see sections 5.7 and 7.2.6 of Firmware user manual.

I believe this feature is currently only activated if a Norwegian price region is selected (@gskjold ?). If it is useful also in Sweden, it can be activated also for Swedish price regions.

Farjestad commented 11 months ago

HiEffect fee. Fee the customer pays for their maximum load of the power grid's capacity and expressed in SEK/kW, month. The fee is based on the highest average power taken per hour during the month, i.e. the hour you used the most electricity. If the consumption of electricity during one hour is 5 kWh, the average power is 5 kWh/1 h = 5 kW.//HansSkickat från min Galaxy -------- Originalmeddelande --------Från: ArnieO @.> Datum: 2023-11-23 09:17 (GMT+01:00) Till: UtilitechAS/amsreader-firmware @.> Kopia: Farjestad @.>, Author @.> Ämne: Re: [UtilitechAS/amsreader-firmware] Nice to have (Issue #680) Interesting, please elaborate on how this is calculated! We have something similar in Norway, already implemented in the code. Please see sections 5.7 and 7.2.6 of Firmware user manual. I believe this feature is currently only activated if a Norwegian price region is selected @.*** ?). If it is useful also in Sweden, it can be activated also for Swedish price regions.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

ArnieO commented 11 months ago

OK, so it seems that the functionality we developed in Norway can be used. Here the value is not only from one single hour, but a bit more complex: The average of the highest consumption hour in 3 different days. And there are steps defined (which is in principle different for each grid company...). Each time that average surpasses a step, your bill for next month is increased by a fixed amount.

But this is parametrized in our functionality, so if you enter "1" in the "Average of" field and just defines the threshold that is relevant for you, I think you will find it to be what you seek? image

If you agree that this is what is needed, I am sure @gskjold will activate that functionality also for Sweden in next firmware update.

Farjestad commented 11 months ago

Ok, sounds good!And I will be happy to try.And another nice to have, if we also can have a meter that calculate the expected value for the hour based on the consumption you use juts that moment, then you can turn off some electric so the average get lower if the value is to hi.//HansSkickat från min Galaxy -------- Originalmeddelande --------Från: ArnieO @.> Datum: 2023-11-23 18:38 (GMT+01:00) Till: UtilitechAS/amsreader-firmware @.> Kopia: Farjestad @.>, Author @.> Ämne: Re: [UtilitechAS/amsreader-firmware] Nice to have (Issue #680) OK, so it seems that the functionality we developed in Norway can be used. Here the value is not only from one single hour, but a bit more complex: The average of the highest consumption hour in 3 different days. And there are steps defined (which is in principle different for each grid company...). Each time that average surpasses a step, your bill for next month is increased by a fixed amount. But this is parametrized in our functionality, so if you enter "1" in the "Average of" field and just defines the threshold that is relevant for you, I think you will find it to be what you seek?

If you agree that this is what is needed, I am sure @gskjold will activate that functionality also for Sweden in next firmware update.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

ArnieO commented 11 months ago

And another nice to have, if we also can have a meter that calculate the expected value for the hour based on the consumption you use juts that moment, then you can turn off some electric so the average get lower if the value is to hi.

Can you be a bit more specific? Please suggest a calculation method - as this is not an obvious thing to implement.

gskjold commented 11 months ago

Here is a firmware where tariff thresholds have been enabled for sweden esp32solo.zip esp8266.zip esp32c3.zip esp32s2.zip esp32.zip

Farjestad commented 11 months ago

Thanks Gunnar!I did start to test now!//HansSkickat från min Galaxy -------- Originalmeddelande --------Från: Gunnar Skjold @.> Datum: 2023-11-25 10:18 (GMT+01:00) Till: UtilitechAS/amsreader-firmware @.> Kopia: Farjestad @.>, Author @.> Ämne: Re: [UtilitechAS/amsreader-firmware] Nice to have (Issue #680) Here is a firmware where tariff thresholds have been enabled for sweden esp32solo.zip esp8266.zip esp32c3.zip esp32s2.zip esp32.zip

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

Farjestad commented 10 months ago

Hi! Nice this seems to work in Sweden, but can we get one more meter for the peak load charge, its calculated in the same way as the power charge, but only non-holiday weekdays 06:00-18:00 during the months of November-March.

ArnieO commented 10 months ago

Hi! Nice this seems to work in Sweden, but can we get one more meter for the peak load charge, its calculated in the same way as the power charge, but only non-holiday weekdays 06:00-18:00 during the months of November-March.

Do you have a link to a precise definition?

Farjestad commented 10 months ago

https://karlstadsnat.se/elnat/priser/elnatsavgift-2023

Farjestad commented 10 months ago

And a meter that shows expected consumption for the current hour, so we save the consumption peak like every minute starting from every full hour and from this we show an expected consumption peak. then you can see if you should turn off any electricity. so the top doesn't get so high.

ArnieO commented 10 months ago

Thank you for additional input and suggestions, we are very interested in adding functionality that can be generally useful!

Grid fees are in general challenging to model, as grid companies du not use identical cost models. This is at least the case in Norway, and we suspect it is the same challenge in other countries. Not only the parameters are different - but the calculation model itself.

So for the time being we will only add features that can be parametrized in the same way for an entire country.

As an example: The peak power calculation (explained here for Norway: https://github.com/UtilitechAS/amsreader-firmware/issues/680#issuecomment-1824767290) is the same for all grid companies in Norway - but the steps vary between grid companies. Which is why we chose the current model (that you have now tested and confirm is useful also for Sweden).

We will discuss your additional propositions on the same basis.

Farjestad commented 10 months ago

Is ther a api to get data from the Ams box

ArnieO commented 10 months ago

You can read JSON strings from it, please see the bottom of this page in the Wiki: https://github.com/UtilitechAS/amsreader-firmware/wiki/Message-formats You find this information also in the Firmware User Manual: https://amsleser.no/index.php?controller=attachment&id_attachment=53

SVH-Powel commented 9 months ago

Even if this functionality is implemented in amsleser, it does not work very well.

On my system, it happens 2-3 times each month that an hour reading is missing and amsleser is adding the missing energy usage to the next hour which then gets around twice as much as it should be. And that leads to a peak consumption for that month to be 18kWh instead of 9kWh. This error will then last until it is reset when a new month starts. For me, this is a functionality in amsleser which is completely unusable. Luckily, I get the correct information from Tibber instead.

ArnieO commented 9 months ago

Even if this functionality is implemented in amsleser, it does not work very well.

On my system, it happens 2-3 times each month that an hour reading is missing and amsleser is adding the missing energy usage to the next hour which then gets around twice as much as it should be. And that leads to a peak consumption for that month to be 18kWh instead of 9kWh. This error will then last until it is reset when a new month starts. For me, this is a functionality in amsleser which is completely unusable. Luckily, I get the correct information from Tibber instead.

That is of course very unfortunate - and it reveals that you are either in Norway or Sweden, where the meters send datagrams according to NVE standard (accumulated power only once every whole hour). Danish users, plus all users with the newer P1 interface, receive accumulated energy in all datagrams and do not see this issue, even if one data package is lost.

Your issue is that some whole-hour transmissions from the meter to the reader are lost or rejected for some reason, and some kind of hardware issue. Potentially a power consumption issue - which could be linked to how the unit (a Pow-U?) is physically located. If it struggles transmitting on Wi-Fi, it could potentially turn up the transmission power and cause reboot issues.

SVH-Powel commented 9 months ago
  • Which hardware are you running?

An esp32 node

  • How is your device located?

approx. 5m from the meter. There is a twisted pair cable between meter and the esp32 node.

  • What is the RSSI level?

-38dBm

  • What is indicated as Reboot cause on your Info-page?

It doesn't unexpectedly reboot. Last reboot is when I do firmware updates and that could be several months between.

  • Are you using a Tibber Pulse in parallell on the same HAN port, with an RJ45 splitter? (Could be the cause of your issue with lost payloads?)

No

SVH-Powel commented 9 months ago

Could it be that there is no lost data, but the meter instead reporting zero or the same consumption as last hour?

ArnieO commented 9 months ago

An esp32 node

Please elaborate, and also provide information on how your device is powered.

Also:

Could it be that there is no lost data, but the meter instead reporting zero or the same consumption as last hour?

Sorry, but I do not understand what you mean. If device receives updated accumulated energy (kWh) at whole hour (assuming you are in Norway or Sweden, on a meter that transmits accumulated energy only once per hour) I do not see why it should work differently on your device compared to all other devices.

SVH-Powel commented 9 months ago

It is a long time since this was installed so I don't remember all the details. But it is a standard esp32 node and I am using a standard level converter between the meter and the esp32. It is powered by a 2A/5V usb supply on mains. To clarify: It is delivering data to Home Assistant via MQTT apparantly fine. There is no indication of lost data, except for these 2 or 3 missing hour reports each month where consumption is zero in the graph and double height the next hour.

No, I don't see any red error messages. How long are those messages shown? And are they available on MQTT so that I can log them? Are there any checksum on the data that the meter send to amsleser? And are there any checksum failure information on MQTT?

ArnieO commented 9 months ago

I think the error messages that are shown in case the device received invalid data is shown until the device has received 2-3 valid messages. Depending on which meter you are on (the period between transmissions is not the same on all meters) that could mean 10 to 30 seconds.

Yes, there is a CRC checksum on the data coming from the meter. If checksum fails, the message is rejected and an error message is shown. Information on rejected messages is not forwarded to MQTT.

There are a variety of ESP32 modules, with varying quality - so difficult to way what could be the cause of your issues. But as a general statement, power stability during Wi-Fi activity is the largest cause of issues on such modules. They work just fine for most projects, but this application exploits Wi-Fi quite heavily. The whole-hour message is timed so that it is sent in-between two "normal" messages, so the Wi-Fi activity load is then at its peak. If decoupling on the ESP module is insufficient this can lead to issues, even if the power supply is beefy (and yours at 2A is more than sufficient).

The Pow-U has a huge supercapacitor on the 3,3V in order to ensure voltage to the ESP is stable.

SVH-Powel commented 9 months ago

So, if I understand you correctly, there is no way to find out why data is missing? Unless I sit down and look at the web page for a month.......

ArnieO commented 9 months ago

So, if I understand you correctly, there is no way to find out why data is missing? Unless I sit down and look at the web page for a month.......

We have a way of finding out more, the difficulty in your case is that the problems seem to appear quite seldom. And by experience (you will find several threads here if you search - where people have issues with using their own hardware) the kind of issues you see are linked to power issues as described with their custom hardware.

I propose you start a Telnet debug just before passing of whole hour, and see if you detect an anomaly around the passing of the hour. You'll probably have to repeat this several times, if the problem appears seldom.

As an alternative you could try to set Power saving to "Maximum", like this: image

The GUI could become a bit laggy, but might remedy your issue.

stigvi commented 8 months ago

Happened again on mine

image

I have moved the PowU to the outside after this happened. It was on the inside on a 5m long cable, on USB power and -25dbm signal strength. It was not booting when this happened.

It is now on the outside on a 0.2m long cable, no USB power and not so good signal strength. I wan't to see if the shorter cable have something to say.

image image

ArnieO commented 8 months ago

Happened again on mine

Interesting; it seems like it is not averaging as it should. Thank you for reporting, we'll have a look at it.

And it could be very helpful if anyone is able to catch with Telnet debug such an incident while passing whole hour.

SVH-Powel commented 8 months ago

Happened again on mine

Interesting; it seems like it is not averaging as it should. Thank you for reporting, we'll have a look at it.

And it could be very helpful if anyone is able to catch with Telnet debug such an incident while passing whole hour.

I think a more viable solution is if you could send all errors to mqtt. It would be A LOT easier to catch.

ArnieO commented 8 months ago

I think a more viable solution is if you could send all errors to mqtt. It would be A LOT easier to catch.

That is a good proposition, which we will certainly consider.

SVH-Powel commented 8 months ago

I suspect that data from my water meter is sent to the electricity meter and then from the electricity meter to the grid company. I would guess that does not influence data transmitted on the HAN interface, but I know nothing about that.

But I think that any info like the boot reason, stuff that fail any checksums, messages on HAN that is not recognized or any other faults detected by amsleser, should be sent to mqtt.

ArnieO commented 8 months ago

We have difficulties reproducing this in our testing. If a whole-hour message is missed, it stays at zero until next whole hour message is register, then the average will be shown for the preceding hours. But that does not happen in your case.

Do you have a sensor that logs the Accumulated energy messages? They should arrive once every whole hour. It would be interesting to know whether there is a missing measurement, or whether the same value is received twice.

SVH-Powel commented 8 months ago

When I hoover over the line in this time frame, I see there is no new data registered by HA. But HA is logging state changes so it is impossible to answer your question.

image

ArnieO commented 8 months ago

If I understand correctly: If the meter sent two identical messages, there would be no point there.

Is it possible for you to set up a sensor in Node-Red that reads it? I might have something to help you if you need, I've already done some MQTT payload parsing in Node-Red.

SVH-Powel commented 8 months ago

Is it possible for you to set up a sensor in Node-Red that reads it?

I am using Home Assistant and don't have node-red

ArnieO commented 8 months ago

HELP NEEDED Is there any Home Assistant expert reading this, that can help figure out if there is a way to log all payloads arriving from the amsreader?

SVH-Powel commented 8 months ago

HELP NEEDED Is there any Home Assistant expert reading this, that can help figure out if there is a way to log all payloads arriving from the amsreader?

That is possible in HA, but I have not enabled it. I will set it to True and see what happens.

https://www.home-assistant.io/integrations/sensor.mqtt/#force_update

ArnieO commented 8 months ago

I will set it to True and see what happens.

Did it work as expected?

SVH-Powel commented 8 months ago

I will set it to True and see what happens.

Did it work as expected?

Don't know, yet. Have to wait for the next time an hour reading is skipped. That could be weeks until it happens.

stigvi commented 7 months ago

And it could be very helpful if anyone is able to catch with Telnet debug such an incident while passing whole hour.

I tried the telnet, but the connection is closed in less than an hour so catching something that happens 1 or 2 times each month is next to impossible with telnet.

ArnieO commented 7 months ago

I tried the telnet, but the connection is closed in less than an hour so catching something that happens 1 or 2 times each month is next to impossible with telnet.

Yes we know - this is tricky indeed.

SVH-Powel commented 7 months ago

I tried the telnet, but the connection is closed in less than an hour so catching something that happens 1 or 2 times each month is next to impossible with telnet.

Yes we know - this is tricky indeed.

You could send relevant errors to mqtt, as menioned before. It would make catching errors a lot easier.

Farjestad commented 6 months ago

MOVED TO NEW THREAD: https://github.com/UtilitechAS/amsreader-firmware/issues/779

Skickat från min GalaxyHej!Efter senaste uppdateringen så fungerar inte min Pow -P1, den blinkar grönt men webbsidan fungerar inte. Provat att dra ur sladden  -------- Originalmeddelande --------Från: Gunnar Skjold @.> Datum: 2024-03-21 20:35 (GMT+01:00) Till: UtilitechAS/amsreader-firmware @.> Kopia: Farjestad @.>, Author @.> Ämne: Re: [UtilitechAS/amsreader-firmware] Nice to have (Issue #680) Closed #680 as completed.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>