search

UtilitechAS / amsreader-firmware

ESP8266 and ESP32 compatible firmware to read, interpret and publish data to MQTT from smart electrical meters, both DLMS and DSMR is supported
Other
364 stars 69 forks source link

Only getting hourly accumulated data #741

Closed pvprodk closed 4 months ago

pvprodk commented 4 months ago

I'm only getting hourly updated values on "Accumulated active import" and Export sensors, however I can see the number (circled in red below) updating every few seconds on the AMS Reader dashboard, so the data is avaliable. Is there any way of getting it to MQTT / Home Assistant?

image

Hardware information:

Relevant firmware information:

tfriberg commented 4 months ago

My mqtt-settings looks like this in my Pow-P1 reader:

image

This gives me a bunch of sensors in HA (cropped for space):

image

In MQTT Explorer in HA it looks like this:

image
pvprodk commented 4 months ago

I have similiar MQTT settings and the same sensors in HA. My problem is, that HA only recieves hourly updates on the Accumulated sensors, but as I describe in the initial post, the AMS-reader gets much more frequent update from the Kamstrup meter, but it is only passed on to HA every hour. My Active Import and Active Export updates every 5-10 seconds as expected.

Here is an example of Accumulated Import, which shows hourly data update in HA:

image

pvprodk commented 4 months ago

The Energy Sensors are not published very often it looks like image

ArnieO commented 4 months ago

This might be an actual issue. HA payload code was initially written with meters in mind that follow the HAN-NVE standard, where accumulated energy is only reported by the meter once per hour.

Danish meters send accumulated energy in every payload (each 10 second on Kamstrup Omnipower), and accumulated energy should then be sent in every MQTT message to HA.

-> @gskjold

gskjold commented 4 months ago

If meter really sends accumulated on every payload, it should also be sent to MQTT/HA. The counter on the dashboard is no indication, as it is calculated if the value is not updated from the meter. A telnet debug and MQTT payload debug is required to determine where the issue is

pvprodk commented 4 months ago

A telnet debug and MQTT payload debug is required to determine where the issue is

I would need some instructions on how to do that ;)

I think I am being fooled by the mentioned calculation, because every hour I get the Energy data: image

Problem is that HA timestamps the energy consumption in the wrong hour this way.

ArnieO commented 4 months ago

I would need some instructions on how to do that ;) https://www.amsleser.no/module/ets_blog/blog?id_post=24

Danish payloads will normally send accumulated energy in all payloads, so this is a bit weird. We'll see when we receive the telnet debug output and decode it.

Problem is that HA timestamps the energy consumption in the wrong hour this way.

This is a separate issue already discussed in several threads here. The problem is that the whole-hour accumulated energy is sent from the meter a few seconds after whole hour. HA then uses the arrival timestamp and assigns the value to the wrong hour. We have searched in vain on how to override that functionality.

pvprodk commented 4 months ago

We'll see when we receive the telnet debug output and decode it.

Thank you for the instructions

Here is the output of a short telnet session, setting the output to Verbose

*** Remote debug - over telnet - for ESP32 - version 3.0.5
* Host name: amskamstrup IP:192.168.176.160 Mac address:D4:F9:8D:6E:85:48
* Free Heap RAM: 110336
* ESP SDK version: 4.4.5.230722
******************************************************
* Commands:
    ? or help -> display these help of commands
    q -> quit (close this connection)
    m -> display memory available
    v -> set debug level to verbose
    d -> set debug level to debug
    i -> set debug level to info
    w -> set debug level to warning
    e -> set debug level to errors
    s -> set debug silence on/off
    l -> show debug level
    t -> show time (millis)
    profiler:
      p      -> show time between actual and last message (in millis)
      p min  -> show only if time is this minimal
      P time -> set debug level to profiler
    c -> show colors
    filter:
          filter <string> -> show only debugs with this
          nofilter        -> disable the filter

* Please type the command and press enter to execute.(? or h for this help)
***
(E) Ended up in default case while unwrapping...(tag D5)
(W) Unknown data received
(E) Ended up in default case while unwrapping...(tag 79)
(W) Unknown data received
v
* Debug: Command received: v
* Debug level set to Verbose
(D) Serving /data.json over http...
(D) Serving /data.json over http...
(V) HDLC frame:
(V) 7E A0 E2 2B  21 13 23 9A  E6 E7 00 0F  00 00 00 00
(V) 0C 07 E8 03  01 05 0C 25  14 FF 80 00  00 02 19 0A
(V) 0E 4B 61 6D  73 74 72 75  70 5F 56 30  30 30 31 09
(V) 06 01 01 00  00 05 FF 0A  10 35 37 30  36 35 36 37
(V) 32 38 32 32  39 39 34 37  35 09 06 01  01 60 01 01
(V) FF 0A 12 36  38 34 31 31  33 38 42 4E  31 34 33 31
(V) 30 31 30 31  30 09 06 01  01 01 07 00  FF 06 00 00
(V) 00 15 09 06  01 01 02 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 03  07 00 FF 06  00 00 00 00  09 06 01 01
(V) 04 07 00 FF  06 00 00 00  BF 09 06 01  01 1F 07 00
(V) FF 06 00 00  00 4F 09 06  01 01 33 07  00 FF 06 00
(V) 00 00 30 09  06 01 01 47  07 00 FF 06  00 00 00 7B
(V) 09 06 01 01  20 07 00 FF  12 00 E2 09  06 01 01 34
(V) 07 00 FF 12  00 DD 09 06  01 01 48 07  00 FF 12 00
(V) E2 BD 2B 7E
(V) LLC frame:
(V) E6 E7 00 0F  00 00 00 00  0C 07 E8 03  01 05 0C 25
(V) 14 FF 80 00  00 02 19 0A  0E 4B 61 6D  73 74 72 75
(V) 70 5F 56 30  30 30 31 09  06 01 01 00  00 05 FF 0A
(V) 10 35 37 30  36 35 36 37  32 38 32 32  39 39 34 37
(V) 35 09 06 01  01 60 01 01  FF 0A 12 36  38 34 31 31
(V) 33 38 42 4E  31 34 33 31  30 31 30 31  30 09 06 01
(V) 01 01 07 00  FF 06 00 00  00 15 09 06  01 01 02 07
(V) 00 FF 06 00  00 00 00 09  06 01 01 03  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  04 07 00 FF  06 00 00 00
(V) BF 09 06 01  01 1F 07 00  FF 06 00 00  00 4F 09 06
(V) 01 01 33 07  00 FF 06 00  00 00 30 09  06 01 01 47
(V) 07 00 FF 06  00 00 00 7B  09 06 01 01  20 07 00 FF
(V) 12 00 E2 09  06 01 01 34  07 00 FF 12  00 DD 09 06
(V) 01 01 48 07  00 FF 12 00  E2
(V) DLMS frame:
(V) 0F 00 00 00  00 0C 07 E8  03 01 05 0C  25 14 FF 80
(V) 00 00 02 19  0A 0E 4B 61  6D 73 74 72  75 70 5F 56
(V) 30 30 30 31  09 06 01 01  00 00 05 FF  0A 10 35 37
(V) 30 36 35 36  37 32 38 32  32 39 39 34  37 35 09 06
(V) 01 01 60 01  01 FF 0A 12  36 38 34 31  31 33 38 42
(V) 4E 31 34 33  31 30 31 30  31 30 09 06  01 01 01 07
(V) 00 FF 06 00  00 00 15 09  06 01 01 02  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  03 07 00 FF  06 00 00 00
(V) 00 09 06 01  01 04 07 00  FF 06 00 00  00 BF 09 06
(V) 01 01 1F 07  00 FF 06 00  00 00 4F 09  06 01 01 33
(V) 07 00 FF 06  00 00 00 30  09 06 01 01  47 07 00 FF
(V) 06 00 00 00  7B 09 06 01  01 20 07 00  FF 12 00 E2
(V) 09 06 01 01  34 07 00 FF  12 00 DD 09  06 01 01 48
(V) 07 00 FF 12  00 E2
(D) Received valid DLMS at 29
(V) Using application data:
(V) 02 19 0A 0E  4B 61 6D 73  74 72 75 70  5F 56 30 30
(V) 30 31 09 06  01 01 00 00  05 FF 0A 10  35 37 30 36
(V) 35 36 37 32  38 32 32 39  39 34 37 35  09 06 01 01
(V) 60 01 01 FF  0A 12 36 38  34 31 31 33  38 42 4E 31
(V) 34 33 31 30  31 30 31 30  09 06 01 01  01 07 00 FF
(V) 06 00 00 00  15 09 06 01  01 02 07 00  FF 06 00 00
(V) 00 00 09 06  01 01 03 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 04  07 00 FF 06  00 00 00 BF  09 06 01 01
(V) 1F 07 00 FF  06 00 00 00  4F 09 06 01  01 33 07 00
(V) FF 06 00 00  00 30 09 06  01 01 47 07  00 FF 06 00
(V) 00 00 7B 09  06 01 01 20  07 00 FF 12  00 E2 09 06
(V) 01 01 34 07  00 FF 12 00  DD 09 06 01  01 48 07 00
(V) FF 12 00 E2
(V) DLMS
(V) (EnergyAccounting) Adding 0.0001 kWh import
(V) (EnergyAccounting)  calculating threshold, currently at 3
(V) (EnergyAccounting)  new threshold 3
(D) Serving /data.json over http...
(V) HDLC frame:
(V) 7E A0 E2 2B  21 13 23 9A  E6 E7 00 0F  00 00 00 00
(V) 0C 07 E8 03  01 05 0C 25  1E FF 80 00  00 02 19 0A
(V) 0E 4B 61 6D  73 74 72 75  70 5F 56 30  30 30 31 09
(V) 06 01 01 00  00 05 FF 0A  10 35 37 30  36 35 36 37
(V) 32 38 32 32  39 39 34 37  35 09 06 01  01 60 01 01
(V) FF 0A 12 36  38 34 31 31  33 38 42 4E  31 34 33 31
(V) 30 31 30 31  30 09 06 01  01 01 07 00  FF 06 00 00
(V) 00 10 09 06  01 01 02 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 03  07 00 FF 06  00 00 00 00  09 06 01 01
(V) 04 07 00 FF  06 00 00 00  BE 09 06 01  01 1F 07 00
(V) FF 06 00 00  00 4E 09 06  01 01 33 07  00 FF 06 00
(V) 00 00 30 09  06 01 01 47  07 00 FF 06  00 00 00 7B
(V) 09 06 01 01  20 07 00 FF  12 00 E2 09  06 01 01 34
(V) 07 00 FF 12  00 DD 09 06  01 01 48 07  00 FF 12 00
(V) E2 15 16 7E
(V) LLC frame:
(V) E6 E7 00 0F  00 00 00 00  0C 07 E8 03  01 05 0C 25
(V) 1E FF 80 00  00 02 19 0A  0E 4B 61 6D  73 74 72 75
(V) 70 5F 56 30  30 30 31 09  06 01 01 00  00 05 FF 0A
(V) 10 35 37 30  36 35 36 37  32 38 32 32  39 39 34 37
(V) 35 09 06 01  01 60 01 01  FF 0A 12 36  38 34 31 31
(V) 33 38 42 4E  31 34 33 31  30 31 30 31  30 09 06 01
(V) 01 01 07 00  FF 06 00 00  00 10 09 06  01 01 02 07
(V) 00 FF 06 00  00 00 00 09  06 01 01 03  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  04 07 00 FF  06 00 00 00
(V) BE 09 06 01  01 1F 07 00  FF 06 00 00  00 4E 09 06
(V) 01 01 33 07  00 FF 06 00  00 00 30 09  06 01 01 47
(V) 07 00 FF 06  00 00 00 7B  09 06 01 01  20 07 00 FF
(V) 12 00 E2 09  06 01 01 34  07 00 FF 12  00 DD 09 06
(V) 01 01 48 07  00 FF 12 00  E2
(V) DLMS frame:
(V) 0F 00 00 00  00 0C 07 E8  03 01 05 0C  25 1E FF 80
(V) 00 00 02 19  0A 0E 4B 61  6D 73 74 72  75 70 5F 56
(V) 30 30 30 31  09 06 01 01  00 00 05 FF  0A 10 35 37
(V) 30 36 35 36  37 32 38 32  32 39 39 34  37 35 09 06
(V) 01 01 60 01  01 FF 0A 12  36 38 34 31  31 33 38 42
(V) 4E 31 34 33  31 30 31 30  31 30 09 06  01 01 01 07
(V) 00 FF 06 00  00 00 10 09  06 01 01 02  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  03 07 00 FF  06 00 00 00
(V) 00 09 06 01  01 04 07 00  FF 06 00 00  00 BE 09 06
(V) 01 01 1F 07  00 FF 06 00  00 00 4E 09  06 01 01 33
(V) 07 00 FF 06  00 00 00 30  09 06 01 01  47 07 00 FF
(V) 06 00 00 00  7B 09 06 01  01 20 07 00  FF 12 00 E2
(V) 09 06 01 01  34 07 00 FF  12 00 DD 09  06 01 01 48
(V) 07 00 FF 12  00 E2
(D) Received valid DLMS at 29
(V) Using application data:
(V) 02 19 0A 0E  4B 61 6D 73  74 72 75 70  5F 56 30 30
(V) 30 31 09 06  01 01 00 00  05 FF 0A 10  35 37 30 36
(V) 35 36 37 32  38 32 32 39  39 34 37 35  09 06 01 01
(V) 60 01 01 FF  0A 12 36 38  34 31 31 33  38 42 4E 31
(V) 34 33 31 30  31 30 31 30  09 06 01 01  01 07 00 FF
(V) 06 00 00 00  10 09 06 01  01 02 07 00  FF 06 00 00
(V) 00 00 09 06  01 01 03 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 04  07 00 FF 06  00 00 00 BE  09 06 01 01
(V) 1F 07 00 FF  06 00 00 00  4E 09 06 01  01 33 07 00
(V) FF 06 00 00  00 30 09 06  01 01 47 07  00 FF 06 00
(V) 00 00 7B 09  06 01 01 20  07 00 FF 12  00 E2 09 06
(V) 01 01 34 07  00 FF 12 00  DD 09 06 01  01 48 07 00
(V) FF 12 00 E2
(V) DLMS
(V) (EnergyAccounting) Adding 0.0000 kWh import
(V) (EnergyAccounting)  calculating threshold, currently at 3
(V) (EnergyAccounting)  new threshold 3
(D) Serving /data.json over http...
(D) Serving /data.json over http...
(V) HDLC frame:
(V) 7E A0 E2 2B  21 13 23 9A  E6 E7 00 0F  00 00 00 00
(V) 0C 07 E8 03  01 05 0C 25  28 FF 80 00  00 02 19 0A
(V) 0E 4B 61 6D  73 74 72 75  70 5F 56 30  30 30 31 09
(V) 06 01 01 00  00 05 FF 0A  10 35 37 30  36 35 36 37
(V) 32 38 32 32  39 39 34 37  35 09 06 01  01 60 01 01
(V) FF 0A 12 36  38 34 31 31  33 38 42 4E  31 34 33 31
(V) 30 31 30 31  30 09 06 01  01 01 07 00  FF 06 00 00
(V) 00 17 09 06  01 01 02 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 03  07 00 FF 06  00 00 00 00  09 06 01 01
(V) 04 07 00 FF  06 00 00 00  BF 09 06 01  01 1F 07 00
(V) FF 06 00 00  00 4E 09 06  01 01 33 07  00 FF 06 00
(V) 00 00 30 09  06 01 01 47  07 00 FF 06  00 00 00 7B
(V) 09 06 01 01  20 07 00 FF  12 00 E2 09  06 01 01 34
(V) 07 00 FF 12  00 DD 09 06  01 01 48 07  00 FF 12 00
(V) E2 A1 62 7E
(V) LLC frame:
(V) E6 E7 00 0F  00 00 00 00  0C 07 E8 03  01 05 0C 25
(V) 28 FF 80 00  00 02 19 0A  0E 4B 61 6D  73 74 72 75
(V) 70 5F 56 30  30 30 31 09  06 01 01 00  00 05 FF 0A
(V) 10 35 37 30  36 35 36 37  32 38 32 32  39 39 34 37
(V) 35 09 06 01  01 60 01 01  FF 0A 12 36  38 34 31 31
(V) 33 38 42 4E  31 34 33 31  30 31 30 31  30 09 06 01
(V) 01 01 07 00  FF 06 00 00  00 17 09 06  01 01 02 07
(V) 00 FF 06 00  00 00 00 09  06 01 01 03  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  04 07 00 FF  06 00 00 00
(V) BF 09 06 01  01 1F 07 00  FF 06 00 00  00 4E 09 06
(V) 01 01 33 07  00 FF 06 00  00 00 30 09  06 01 01 47
(V) 07 00 FF 06  00 00 00 7B  09 06 01 01  20 07 00 FF
(V) 12 00 E2 09  06 01 01 34  07 00 FF 12  00 DD 09 06
(V) 01 01 48 07  00 FF 12 00  E2
(V) DLMS frame:
(V) 0F 00 00 00  00 0C 07 E8  03 01 05 0C  25 28 FF 80
(V) 00 00 02 19  0A 0E 4B 61  6D 73 74 72  75 70 5F 56
(V) 30 30 30 31  09 06 01 01  00 00 05 FF  0A 10 35 37
(V) 30 36 35 36  37 32 38 32  32 39 39 34  37 35 09 06
(V) 01 01 60 01  01 FF 0A 12  36 38 34 31  31 33 38 42
(V) 4E 31 34 33  31 30 31 30  31 30 09 06  01 01 01 07
(V) 00 FF 06 00  00 00 17 09  06 01 01 02  07 00 FF 06
(V) 00 00 00 00  09 06 01 01  03 07 00 FF  06 00 00 00
(V) 00 09 06 01  01 04 07 00  FF 06 00 00  00 BF 09 06
(V) 01 01 1F 07  00 FF 06 00  00 00 4E 09  06 01 01 33
(V) 07 00 FF 06  00 00 00 30  09 06 01 01  47 07 00 FF
(V) 06 00 00 00  7B 09 06 01  01 20 07 00  FF 12 00 E2
(V) 09 06 01 01  34 07 00 FF  12 00 DD 09  06 01 01 48
(V) 07 00 FF 12  00 E2
(D) Received valid DLMS at 29
(V) Using application data:
(V) 02 19 0A 0E  4B 61 6D 73  74 72 75 70  5F 56 30 30
(V) 30 31 09 06  01 01 00 00  05 FF 0A 10  35 37 30 36
(V) 35 36 37 32  38 32 32 39  39 34 37 35  09 06 01 01
(V) 60 01 01 FF  0A 12 36 38  34 31 31 33  38 42 4E 31
(V) 34 33 31 30  31 30 31 30  09 06 01 01  01 07 00 FF
(V) 06 00 00 00  17 09 06 01  01 02 07 00  FF 06 00 00
(V) 00 00 09 06  01 01 03 07  00 FF 06 00  00 00 00 09
(V) 06 01 01 04  07 00 FF 06  00 00 00 BF  09 06 01 01
(V) 1F 07 00 FF  06 00 00 00  4E 09 06 01  01 33 07 00
(V) FF 06 00 00  00 30 09 06  01 01 47 07  00 FF 06 00
(V) 00 00 7B 09  06 01 01 20  07 00 FF 12  00 E2 09 06
(V) 01 01 34 07  00 FF 12 00  DD 09 06 01  01 48 07 00
(V) FF 12 00 E2
(V) DLMS
(V) (EnergyAccounting) Adding 0.0001 kWh import
(V) (EnergyAccounting)  calculating threshold, currently at 3
(V) (EnergyAccounting)  new threshold 3
pvprodk commented 4 months ago

This is a separate issue already discussed in several threads here. The problem is that the whole-hour accumulated energy is sent from the meter a few seconds after whole hour. HA then uses the arrival timestamp and assigns the value to the wrong hour. We have searched in vain on how to override that functionality.

Yeah, I know thats not easy to solve when the meter only puts out data every hour - hence my desire to have live energy-data. I have a friend living in the same area as me, and his Kamstrup reports energy every update. Same utility company, he has slightly newer meter, and maybe different firmware on the meters.

ArnieO commented 4 months ago

Here is the output of a short telnet session, setting the output to Verbose

Great. @gskjold will check the content and respond. But that payload looks very short...

gskjold commented 4 months ago

Thanks, this is HAN-NVE format, not encrypted, so you will only receive accumulated once per hour. I have seen this once before in DK, but i believe that was a company. Your grid provider might be able to adjust the output. @ArnieO do you remember the name of the default DK payload in their configuration?

pvprodk commented 4 months ago

I have recieved GPK60 + GPK61 decryption keys from the utility company, which I have entered into the configuration, so when you say it is not encrypted is that because this data is taken after decryption? Or is my utility company just drunk? 🤣

ArnieO commented 4 months ago

Yeah, I know thats not easy to solve when the meter only puts out data every hour - hence my desire to have live energy-data. I have a friend living in the same area as me, and his Kamstrup reports energy every update. Same utility company, he has slightly newer meter, and maybe different firmware on the meters.

I am starting to suspect that the setting of your meter is incorrect. Please see the Omnipower HAN Interface Specification. You will see that it describes two types of payload:

  1. HAN-NVE, where accumulated energy is sent in List 2, once per hour
  2. DLMS push 1, where accumulated energy is sent in all payloads

On page 16 you'll see a screendump from Omnisoft Vision Air, which is the software the grid company uses when they activate your HAN port. They might have selected the wrong HAN push list by a mistake: image

What you want is DLMS push 1, which is the one usually used in Denmark.

ArnieO commented 4 months ago

OK, I can see my suspicion is confirmed. 😄

ArnieO commented 4 months ago

I have recieved GPK60 + GPK61 decryption keys from the utility company, which I have entered into the configuration, so when you say it is not encrypted is that because this data is taken after decryption? Or is my utility company just drunk? 🤣

I don't know about "drunk", but apparently they forgot to tick this box: image

As far as I know, encryption is mandatory in Denmark. It is strange that they sent you encryption keys while forgetting to configure your meter for encryption. Anyway, I advise you to contact the grid company and request them to reconfigure your meter.

Would you mind telling us which grid company this is?

pvprodk commented 4 months ago

I have just written them an email to enable DLMS push 1 instead of HAN-NVE

My grid company is "Sunds Forsyning" a small local one, so usually they are very cooperative :)

pvprodk commented 4 months ago

And thanks for all you invaluable help! 😄

ArnieO commented 4 months ago

My grid company is "Sunds Forsyning" a small local one, so usually they are very cooperative :)

That is the nice thing about the small grid companies. The disadvantage is that they might not be so used to doing things like this - so the risk of mistakes increase.

ArnieO commented 4 months ago

And thanks for all you invaluable help! 😄

You're welcome! It would be nice if you could close this thread then (you can reopen it later if needed).

pvprodk commented 4 months ago

And thanks for all you invaluable help! 😄

You're welcome! It would be nice if you could close this thread then (you can reopen it later if needed).

Hereby closed. Thanks