Need help with testing of new firmware on multiple meter models

[gskjold]

I am currently writing new code to interpret data from the meter. The new firmware should support any IEC62056-7-5 (Norwegian and Danish meters with HDLC payload) and IEC62056-21 (Swedish and possibly Dutch meters with RJ12 connector). To confirm that it actually works, I need to have it tested on multiple brands in all possible countries.

A list to start with, I will expand this as new meters are tested:

Norway

• [ ] Aidon 1p
• [x] Aidon 3p IT
• [x] Aidon 3p TN
• [ ] Kaifa 1p
• [x] Kaifa 3p IT
• [ ] Kaifa 3p TN
• [ ] Kamstrup 1p
• [ ] Kamstrup 3p IT
• [x] Kamstrup 3p TN

Denmark

• [ ] Kamstrup 1p
• [ ] Kamstrup 3p IT
• [x] Kamstrup 3p TN

Sweden

• [x] Aidon 3p TN

Any RJ12 meter, please let me know your setup

NOTES

• Testers should be familiar with USB flashing in case this firmware has errors that leads to failure.
• If you revert back to 1.5 series firmware or lower, the device will enter AP mode to be reconfigured

ams2mqtt-esp32-2.0-rc1.zip ams2mqtt-esp8266-2.0-rc1.zip

ams2mqtt-esp32-2.0-rc2.zip ams2mqtt-esp8266-2.0-rc2.zip

ams2mqtt-esp32-2.0-rc3.zip ams2mqtt-esp8266-2.0-rc3.zip

ams2mqtt-esp32-2.0-rc4.zip ams2mqtt-esp8266-2.0-rc4.zip

ams2mqtt-esp32-2.0-rc5.zip ams2mqtt-esp8266-2.0-rc5.zip

ams2mqtt-esp32-2.0-rc6.zip ams2mqtt-esp8266-2.0-rc6.zip

ams2mqtt-esp32-2.0-rc6.1.zip ams2mqtt-esp8266-2.0-rc6.1.zip

[Allram]

Tested on Aidon 3P IT (Norway) on ESP8266, works just like the previous versions :)

[gskjold]

Thanks! Highly appreciated

[madsbg]

Denmark, Kamstrup 3-phase with Pow-K module. How do I see if it is IT or TN model? HAN indicator is red and debug shows: (readHanPort) Invalid HDLC, returned with -1

[gskjold]

Not sure if you can see it on the meter itself, but if you have a 4 pole main breaker you have TN (400V), but if you have a 3 pole breaker it is IT or TT, which is both 230V and will result in the same datagram.

[madsbg]

I guess it must be TN then.

Don't know if it has anything to do with the test firmware but I went from 1.5.6 -> test firmware -> 1.5.7 and now I have lost connetion to the device. It is falshing blue but no wifi connection.

I will test the new firmware once I have recovered :slightly_smiling_face:

[gskjold]

Aha, if you went back to 1.5 series firmware, it have jumped into AP mode as it does not understand the config structure from the test firmware. You need to reconfigure the device, sorry about that, should have put that up as a warning..

EDIT: Removed incorrect file uploaded

[madsbg]

No problem. I have connection again. The files.zip does not contain a firmware file?

[gskjold]

Sorry, a completely different file from same folder :)

Here you go: firmware.zip

[madsbg]

Still red HAN and same message from debug.

(readHanPort) Frame dump: (D) 41 03 13 2F 7A E6 E7 00 DB 08 4B 41 4D 45 01 D1 (D) 22 A4 82 00 E7 30 00 0B 32 77 CE 04 24 86 23 C3 (D) 10 1C 93 EA DB F7 69 89 26 CE E8 7D D9 5F AE 74 (D) DC FD 92 11 01 93 FF FF D4 2F F8 D7 32 B3 EC B9 (D) B3 05 B8 4B 0A B4 E5 8D A9 3D 3D 28 1E E4 6B DE (D) 9F 88 71 8F 20 3A 5F AA D4 36 15 DD FC C5 8A 11 (D) 76 9C E2 71 5D 56 F5 16 D5 86 EF 03 B3 AB 02 1B (D) 93 A5 2F 50 FC 8C 3C BF 19 C4 0D 04 EE B3 E9 A2 (D) F8 0C 03 03 F5 B8 09 41 5C 59 3F B0 FC ED AF 9E (D) 68 A8 C8 61 25 BF 42 CA B8 8A 32 89 06 CF 14 F8 (D) 12 C1 42 CD C0 0F E7 31 85 51 3A E0 58 48 DE 4C (D) 57 0F A5 3A 27 5D EA 81 7D FF 6B D6 1C 24 56 74 (D) CC A5 4D 48 1D 8A A1 C1 1C 10 53 96 D3 98 F2 F3 (D) D7 DA C4 80 81 E1 81 B4 B5 81 D8 0D 42 D1 2E 85 (D) C0 DD C5 46 31 ED 07 1E 42 32 01 5D 35 5C 62 77 (D) A4 91 1C A6 4E 01 2A 4A 69 D9 38 CD 1C FC 7E (D) (readHanPort) System title: (D) 00 00 00 00 00 00 00 00 (D) (readHanPort) Initialization vector: (D) 00 00 00 00 00 00 00 00 00 00 00 00 (D) (readHanPort) Additional authenticated data: (D) 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 (D) 00 (D) (readHanPort) Authentication tag: (D) 00 00 00 00 00 00 00 00 (W) (readHanPort) Invalid HDLC, returned with -1

[madsbg]

Maybe something to do with the meter detection? I cannot select it manually.

[gskjold]

The new code does not need to know what meter type it is, only which format the data is received. I think maybe it is the format detector that is causing this. I will go throught the code and see if I can figure it out

[ArnieO]

Kamstrup / TN / 3-phase in Norway: Same issue as @madsbg

[gskjold]

Updated first post with new firmware files. Made changes to payload autodetect and also fixed a bug that is specific to Kamstrup.

[madsbg]

Looks better now. Thanks.

(I) (readHanPort) Valid HDLC, start at 47

[gskjold]

Awesome! I will check why it did not detect manufacturer properly

[gskjold]

Hang on, I guess you have an encrypted Kamstrup? They do not include any identifying OBIS codes, so identifying the meter manufacturer would be guesswork.

[madsbg]

Yes. Data is encrypted. So maybe you could change manufacturer to: "Unknown or encrypted". At least that would make it more clear.

I am missing the {root}/meter/dlms/timestamp value. Can you bring that back at some point please? Everything else looks fine.

[gskjold]

True, I just realized when I looked through the code, have taken notes to bring that back before major release with this new code

[ArnieO]

Sweet! Working now. I am on unencrypted Kamstrup, manufacturer indicated as «Unknown» here too.

[madsbg]

Regarding manufacturer. I think there is a "Kamstrup_V0001" string in the data, or do you need more than that for a positive identification?

[gskjold]

This is what I need, but if I remember correctly, this is not included in the encrypted datagram. It should exist in the unencrypted datagram though... Have to look closer at this

[madsbg]

In the 1.5.x firmware it was published as data.lv in JSON payload. In this firmware data.lv is empty.

[gskjold]

Got it, then I am mistaken. Have to look into that :)

[ArnieO]

There is an error in my readings. I have 16 A on L3 and 4.3 kW (which is correct).

[gskjold]

Can you elaborate?

[ArnieO]

Sorry... my mistake. Please forget it - all is well.

[madsbg]

I don't normally use {root}/meter/clock (Raw) but I just noticed, that the value is not correct (2875626203 = 2061-02-14). In JSON the clock is "t" but that value is always 0 (zero).

EDIT: I might be mixing up "t" and "data.rtc" 😕 but there is something wrong with the timestamps. I know it's on your todo list, so just for information.

[gskjold]

Thanks, this is something I didn't check properly.

[madsbg]

Another observation: Even though everything seems to work fine with the new firmware, the Pow-K board i blinking blue-green-red-red-red.

[ArnieO]

blinking blue-green-red-red-red.

I confirm that observation. To be exact, it is not a triple-blink, but three single-blinks. Single-blinks signify incorrect data from the meter, so I just think this is part of the "work in progress".

Excerpt from the Pow-K user manual:

[madsbg]

Thanks @ArnieO, Yes that is correct. I should have been more precise. There are about 3 sec. between each red blink.

[gskjold]

Awesome, good testing everyone! Updated first post with new firmware with the following fix:

• Package timestamp
• Meter timestamp correction
• Kamstrup meter detection

[madsbg]

✔️ Red blinks are gone. ✔️ Manufacturer (Kamstrup) is now displayed correctly ✔️ "{root}/meter/clock" (Raw) seems to be correct (1637269255 = 2021-11-18 21:00:52), but I only recieved it once, so it's not conclusive. I will keep an eye on it :-)

❌ "{root}/dlms/timestamp" (Raw) and "t" (JSON) are present but are all over the place 😲 2490307205 = 2048-11-30 01:00:05 972950405 = 2000-10-31 01:00:05 4005072005 = 2096-11-30 01:00:05 1193160709 = 2007-10-23 19:31:49 2707925509 = 2055-10-23 19:31:49 3523887109 = 2081-08-31 19:31:49

[madsbg]

"{root}/meter/clock" (Raw) contains the correct time when it is updated every hour on the hour, but sometimes in between an unexpected value of 0 (zero) is received.

Also "{root}/meter/import/active/accumulated" contains zero values at odd intervals.

[gskjold]

Thanks for thorough testing! :) I have not had time to look at the latest reported bugs, but I will get back to it soon

[Isaksson]

I noticed that my HAN communication goes from green --> yellow --> red --> green in the GUI so I connected with telnet and debug and noticed that I receives a lot of Invalid HDLC (W) (readHanPort) Invalid HDLC, returned with -2 and when its valid then it looks like this (I) (readHanPort) Valid HDLC, start at 18

Edit: Added information that there is no Manufacturer, Model or ID from my meter

Edit: I don't know if this is in scope of this issue but you did mention that the list from my Swedish meter was 27 and not 18, I think this is the list that is provided from my meter

[gskjold]

If it returns with -2 it means that the checksum on the payloads header was not correct. This indicates that there is errors in the received data.

I was also thinking it was probably that list. Since they also changed the payload, are we sure they use 8E1 parity? The document I have does not specify baud or parity for the meters with this payload.

[Isaksson]

I will connect a Logic Analyzer and see what I receive. Maybe this is the same document that you already have, but I cant seem to find the parity but they state 2400 Baudrate.

My D1 Mini crashes after 30 minute or so when running this code, It does not reboot, I have to reset it to get it up and running again. Maybe that has to do with all the checksum errors (if it works for other users with esp8266)

aidonfd-rj45-han-interface-se-v14a-1.pdf

[gskjold]

Keep an eye on "Free mem" on the main page. Maybe it is running out of memory after 30min because of some memory leak. I do not have this problem on ESP32 at least.

[gskjold]

Updated first post with fix for package timestamp, some performance improvements and new graph library.

I have not yet been able to reproduce random zero meter timestamps, still searching for that one..

[Isaksson]

Keep an eye on "Free mem" on the main page. Maybe it is running out of memory after 30min because of some memory leak. I do not have this problem on ESP32 at least.

I was looking at the GUI when the device freezes and there was Free mem: 18.2kb, I switched to your latest snapshot and now I could select my main fuse size :) thanks for that.

[gskjold]

18.2 kib sounds reasonable, so it must be something else. I will try to reproduce it with a d1 mini

[Isaksson]

Have you changed something with the buffer in the latest snapshot? Sometimes I see this when debugging (I) (readHanPort) Buffer overflow, resetting

[gskjold]

Buffer is same size (1024 bytes), but it fills the buffer in a different way. If there is errors in the received data, it may continue to fill the buffer until it is full and reset.

[madsbg]

Updated first post with fix for package timestamp, (...)

I have updated to the latest version (I think) but now "{root}/dlms/timestamp" (Raw) is missing and "t" (JSON) is always 0 (zero).

BTW: Could you perhaps give these beta firmware versions a visible build number in the GUI? That would make it easier to refer to a specific version.

[Isaksson]

18.2 kib sounds reasonable, so it must be something else. I will try to reproduce it with a d1 mini

With the latest build the device have not yet frozen, now it reboots instead.

[Isaksson]

Would it be possible to add a debug option so that it prints out the entire message retrieved from the HAN port to telnet debug? It would be nice to see whats happening on most of my communication if its an actual error from my Mbus to ttl or if the error is that the code cant parse the information.

[gskjold]

If you go to System -> Debugging and set the debug level to "Debug" it will dump the entire frame received in telnet as well as serial.

[Isaksson]

If you go to System -> Debugging and set the debug level to "Debug" it will dump the entire frame received in telnet as well as serial.

Yes, of course :) I have seen this many times know, it just flow above my head.

I notice that there is some different events.

(W) (readHanPort) Invalid HDLC, returned with -2 (I) (readHanPort) Buffer overflow, resetting (I) (readHanPort) Valid HDLC, start at 18 and also nothing reported at all in the debug after a reading and as you suggested its added to the buffer (but no info in debug) and after a couple of readings it gets overflow, but the question is, do you think that this is because of some "trash" from the TTL out or is it because the code cant decode the data? That my Swedish Meter is sending some different data?

[gskjold]

Got a treat for you tonight; first post updated with new firmware with the following updates:

• Improved new graphs
• Added usage graph for last 24hrs and last month
• Added loading indicator to all uploads
• Added more information on firmware upload page
• Added possibility to specify voltage divider for Vcc in GPIO menu
• Added ADC reading for Vcc when possible (ESP32)