Closed mortenmoulder closed 1 year ago
Which hardware are you using in the two cases?
@ArnieO ESP8266 in the first screenshot and ESP32 in the last. Just regular old ESP NodeMCUs powered externally via USB.
My provider just told me he cannot see any new software updates (firmware update) on my meter, nor can he see that the keys have changed or anything like that. But nothing changed on my end.
Someone on Facebook with the same issue told me his provider came to his house, unplugged the meter, and plugged it back in, and then it worked again. Seems rather strange.
I would first check my connectors - but especially the last screenshot indicates that there are too long intervals between anything coming from the meter. The content of the debug has increased lately, so you could update the 8266 just to see if you get the same.
I have heard of several cases where a reboot of the meter was needed. I think they can do that remotely, should not be necessary to unplug/replug it. I would call the grid company and ask if they can do a remote reboot.
@ArnieO I changed the breadboard jumpers just in case. Still the same. Data is being sent every 10 seconds, so everything checks out. It's just.. not recognized for some reason.
I'll call them up and see if they can reboot it remotely. Weird that it has worked for 1+ year without issues, and then it just randomly dies.
Could it be your external power that is no longer working as it should?
@ArnieO Replaced it but nope, it works as intended. The grid company came out to reset the meter today, but it still fails with the same jibberish as shown above.
I've requested new GPK60 and GPK61 keys, hoping it's as simple as that, but if it's not, I've requested that they contact Kamstrup. I just spoke with Kamstrup, and they told me they cannot do support without a ticket, and I cannot create a ticket as an individual - it has to go through the grid company.
The grid company guy told me, my keys have not changed. He also informed me that "your HAN port is set to standard configuration now". I do see a lot more data, but it's still spitting errors:
I'll ask him what default configuration means on the HAN interface. Maybe encryption is now disabled? Not sure.
This does indeed look like jibberish. Valid payloads shall start and end with the same byte "7E", see payload examples in this old thread where the encryption was implemented two years ago.
You will there also see @mikfoo struggling with his hardware (and ended up with a very old version of what is now the Pow-K). If you search the various threads you will find many examples of potential hardware issues that are difficult to understand. Very difficult to say if that is your problem now.
This is a typical case where one of those USB logic analyzers could be very handy. Like one of these cheap clones: https://www.aliexpress.com/item/1005003243227313.html Maybe someone around you that could borrow you one? It can be set up to read serial data and present as byte string. Mine runs on the software intended for the "original": https://www.saleae.com/
@ArnieO I have a logic analyzer (bought specifically for this project over a year ago), but don't ask me how it works.
I got a response from the grid company, and he told me that "default configuration" was "now with encryption - GPK11". So I asked him to change it back to use GPK60 and GPK61, which I think is done in OmniSoft VisionAir by changing the push list to HAN-NVE. No clue, though.
Take a look at this though.. I started getting (W) messages (warning?) with data starting with 7E. I did not get this before he changed my meter configuration:
(W) (errorBlink) No HAN data received last 30s, single blink
(W) 7E 37 C6 68 05 35 51 EF 82 07 02 00 01 D1 8A 8A
(W) 72 1A 5C 9B 88 AB 9D 3B 8B EC B0 19 20 0A DD 7A
(W) 4D 2D 2F B8 27 14 BC A1 BA 5A FF 28 8A 82 9A 93
(W) 56 18 73 77 09 14 0A 75 4E 07 DF 2B 7A 0E 0D AA
(W) 53 F2 BB 94 BB C5 39 21 AC 4D 3B 4B B2 32 26 3B
(W) 0E 3F 62 04 4B 14 CF 08 97 2B 34 29 58 A0 26 6E
(W) 30 B8 C2 7A 14 9A 00 04 21 1E 48 92 12 0E 29 DA
(W) 51 1B A8 FF D4 5A 08 EF 20 20 3D 3C E7 BE ED 9F
(W) C2 08 6A 08 B9 99 08 45 F5 5F 44 13 59 44 41 7E
(W) B2 E5 7D 0D 27 76 00 96 71 C9 DE 71 69 15 1C 84
(W) B7 47 68 55 EA 0A 34 81 A0 A6 C8 5F 5F EA D1 D8
(W) D6 74 82 63 4B 47 B1 B1 63 25 8C C1 5D 55 C1 75
(W) AD 87 AB 2D B6 14 37 7D A0 ED 9D A7 AD FB 92 90
(W) FE
(W) (errorBlink) No HAN data received last 30s, single blink
(W) (errorBlink) No HAN data received last 30s, single blink
(W) 7E 03 13 70 93 3B 00 48 41 7B 82 07 02 00 A1 5C
(W) 78 A7 8E 49 88 05 74 85 57 FF C6 98 28 5D 20 53
(W) B9 70 95 31 05 36 F8 C2 CB D9 90 95 64 15 21 A1
(W) 45 D4 45 57 89 A2 CF 0A 20 95 B5 23 71 59 EA F4
(W) 68 A0 28 95 9A D2 25 46 81 8E 55 D6 C3 21 0E 63
(W) A5 A9 C3 8D D5 90 85 11 38 9B DB 65 B6 0B 04 50
(W) 2C D9 14 BF 21 64 E3 3E 12 07 DE 2C 1C C5 C6 6F
(W) 1C 17 5E EE B9 B5 58 D3 A6 0D 5D 20 53 E8 42 B6
(W) E3 0E 4A 84 64 8E 96 0A 46 DB DE 35 7E C9 2B AF
(W) AB 44 25 1E E6 5C D2 54 A3 51 E7 1D 3E 4C B7 2D
(W) 16 95 DA A2 D5 B3 D7 26 45 C7 92 54 04 18 68 85
(W) 4F 23 DF 62 15 D6 9B 1B CD 20 16 16 77 1E 68 D1
(W) D5 42 67 C7 99 E6 BB 4D 36 1C 1A 35 7B 22 4D 17
(W) 96 F2 91 EF 8C 8A 44 9A 08 36 72 52 FD FF
(W) (errorBlink) No HAN data received last 30s, single blink
(W) 7E 03 13 DC 93 E7 00 28 05 55 51 80 EF AD 82 07
(W) 1E 55 59 B1 EA 03 EA 69 20 B4 F7 55 36 C0 76 AF
(W) 5E 9F 2D 13 68 1E F1 59 D1 3D 53 6C 11 57 CC 19
(W) 93 73 3A 37 DA 4D 40 85 65 73 20 B6 14 56 17 53
(W) 9E C7 3D 93 67 1E 86 1E 94 78 20 5D 2F 18 56 FD
(W) AA EC 8B 4B 2B 76 E4 CC C1 AC 16 9A A4 88 61 6A
(W) B8 31 52 84 95 85 97 95 51 E8 14 BA D6 43 76 32
(W) E3 D7 AC F2 5B 97 98 37 11 6C 5A AD CA 9A 98 6B
(W) B1 3D B0 B9 E4 EC ED 94 E5 41 46 B5 ED 6D 31 0E
(W) 34 AE CB C1 AA 63 3C 22 BC 29 9A 2F F1 19 44 32
(W) 62 0E A2 7A FD A8 68 07 13 5D C3 DE C9 1F 1D 85
(W) 49 C4 52 27 6F 35 B9 24 66 5F 9D 1F 87 8B D9 0D
(W) 87 AD 23 DB EE
(W) (errorBlink) No HAN data received last 30s, single blink
(W) (errorBlink) No HAN data received last 30s, single blink
Just to confirm, your baud and parity is 2400 8N1 ?
@gskjold Yeah. After reading some of the other threads, I switched back to an ESP8266 NodeMCU. Still getting the same errors (and now warnings) as with the ESP32.
Specifically went back because I noticed some had issues with UART1 versus UART2. I switched back and I am now using GPIO4 (D2) which I know was working for 1+ year.
I asked the grid company about another address here in the same city, because another guy uses the same setup as me, and I just got this reply:
Your meter's HAN setup is the same as the HAN setup for XXXXX I dialed up yours and XXXXX to check HAN-push activation, encryption activation, and HAN-push list settings. They are the same and the control showed no errors
So I have a meter that replies with faulty data, and there's another guy in my city with the exact same setup, and his works fine.
Sigh. This sucks.
@ArnieO I have a logic analyzer (bought specifically for this project over a year ago), but don't ask me how it works.
Yeah, I remember that it took a little fiddeling... but it's a great tool when you get it working!
@ArnieO I have both an ESP32 and ESP8266 running the same latest firmware, yeah. Both are showing the exact same debug output.
I have tried 3 external adapters at this point, and I've reverted to the original one simply because I know it works. All 3 adapters showed the same output as well. Some errors, some warnings. Some warnings starting with 7E which I read is HAN data.
@ArnieO I have both an ESP32 and ESP8266 running the same latest firmware, yeah. Both are showing the exact same debug output.
Great - this practically rules out the hypothesis of an issue on your own hardware.
I have tried 3 external adapters at this point, and I've reverted to the original one simply because I know it works. All 3 adapters showed the same output as well.
Great - then the issue is not a faulty power adapter injecting noise.
I'm starting to lean towards this being an issue with the Kamstrup, but I can of course not long-distance diagnosing. 😅
That other person in your city: Do you know him well enough to ask him if he could swap device with you for a test?
@ArnieO I don't know the guy at all - he just replied to my post on Facebook, because I was asking for help (or if anyone has seen anything like this before - which they had).
I could ask him, but I'll keep in touch with Kamstrup. I reached out and got a reply, so hopefully he got a fix!
Hmm seems like I am not the only one with issues. The latest posts in #126 show very similar stuff.
I've got confirmation from Kamstrup technical support, that my setup is the exact same as the other guy in my town using a Pow-K with AmsToMqttBridge. They even tried "resetting" my settings and reapply them with HAN-NVE push (10 second interval). It still just throws errors at my face via telnet.
At this point I might as well buy the Frient or similar. Having to deal with technical support that just shrugs when problems occur is really annoying.
Yes, you are not the first having issues. I try to assist and advise to the best of my ability. As you may know, I have developed and sell the Pow-K+, and my advise is based on the accumulated experience. But long distance debugging is not easy! 😄
And your case is a bit special, I have not seen this before:
If I should guess, I would put my 10 cents on a bug in the meter. But I might be wrong. Your grid company does not want to help? They could try replacing the meter? Or at least bring some reader they know works, and test it on your meter?
@ArnieO Sorry if that last message sounded rude towards you - you have done nothing wrong. Nor has @gskjold. I'm talking about Kamstrup and my grid company, who can't help me further.
When you say "bring some reader", do you mean something like an M-Bus thingy? Because I doubt my grid company knows anything about that.
They replaced my meter last year because it wasn't reporting anything back. Ended up having to talk to Kamstrup directly and then suddenly it worked. I'm talking with the same guy via mail now, and it sounds like he doesn't know what's wrong :(
No worries - I fully understand your frustration! With "bring some reader" I was thinking of for instance the Smart-Me reader.
Another possibility is to see if there is a Pow-K somewhere relatively near you. You are welcome to contact me on post@amsleser.no for that. I could then check my customer list and see if there is one close to you - and if so contact that buyer and ask if he/she would be interested in helping you.
The intention would be to see if a known working device is able to read your meter, to determine if the problem is in your meter.
@ArnieO The guy from my town uses a Pow-K+ so I'll see if I can talk to him. Thanks for the help!
Hey @ArnieO I just got a reply from my grid company - a new person this time. He told me that they are obligated to open up for HAN-PUSH, encryption, and HAN-NVE as the push list (or DLMS PUSH 1) as per the law. They have done that, but since it's not working, they will come and change out my meter with a new one.
Then they will take the old one and ship it to Kamstrup, and have Kamstrup investigate it for errors. If they determine it's not broken, they will bill me for the swap, otherwise they're going to pay for the investigation.
I verified this weekend that the Pow-K+ didn't work either (nice product btw, I will definitely order one if this whole mess works at some point). I got the same errors as with my ESP8266 and ESP32.
I want to accept the offer of them swapping my meter with a new one, but I'm honestly annoyed that if it for some reason works for them, I have to pay. I have no way of verifying that it works or not, because I don't have any official tools. I can only trust their words that the meter is configured correctly.
What I don't understand is that I see valid HAN data (according to other issues here). Warnings starting with 7E
. I am receiving data - it's just not correct. Tried swapping dupont cables. Tried soldering dupont cable directly to ESP. Tried swapping power supplies. Tried a known good Pow-K. I tried essentially everything I can. sigh
Hi @mortenmoulder, thank you for the kind words! Great that you have been able to test also with a known working Pow-K, I think you have done as much testing as you possibly can. Valid payloads shall both start and end with 7E, so there is something strange with what you receive.
I agree - the grid company should have had a way to verify themselves that the meter works correctly. They might need a Pow-K? 😆
Please keep us posted, your experience could be useful for others!
@ArnieO Haha yeah they should 😆
If valid data starts and ends with 7E
, then something is definitely wrong:
(W) (errorBlink) No HAN data received last 30s, single blink
(W) 7E 13 DE 41 80 EF AD E6 D8 73 E8 11 C5 4D 69 71
(W) 65 47 A5 25 C8 AD 92 34 7A A6 4F D0 E8 F4 5B 87
(W) 4A C7 D2 4D A4 58 92 0F 4A 16 55 6E CA 2B 84 60
(W) 61 44 E0 38 DE 0E 81 43 A1 34 47 D4 54 48 CC 06
(W) 41 BD C0 07 BB A3 10 6F 60 3E 6F 9A C1 3B ED 12
(W) 7A 79 5A FF
10 SECOND DELAY
(W) 7E 13 AC 68 41 EF 6D B6 65 15 58 06 9A C3 71 52
(W) C4 11 AC 0A D5 D7 40 0E 21 92 40 42 C7 89 29 B8
(W) A8 04 54 5D 45 90 20 4C 86 8B EB 8C 94 73 76 AD
(W) 7F 50 59 19 53 67 A0 53 ED C8 D7 5E 4C 4E 97 AB
(W) 45 1E 68 B2 0A C2 A0 88 C0 6A AE 64 A5 6A 78 0D
(W) EE FB C8 A9 66 AE E7 8B 2E EB 8D 4F 2F A4
10 SECOND DELAY
(W) 7E 13 49 CC DE 30 56 48 05 50 EB 34 32 56 5B 20
(W) B9 B2 B8 61 65 31 80 84 A3 31 F0 8C 2F AE 1F 60
(W) E8 CF 67 E7 4A A0 F9 79 69 9D 60 BF 93 FD 28 4D
(W) 7B EF CF AB 38 C8 8E 57 70 B4 10 91 39 45 82 B6
(W) E0 C5 18 57 40 60 C4 6B 6B 19 5D 30 7E 00 41 5D
(W) 5B DD 00
If valid data starts and ends with
7E
, then something is definitely wrong:
Yes, this is why I have had a hypothesis of reading error, which could in principle be due to contact problems, connectors, noise due to external power... several things that are difficult to diagnose remotely. But with all the testing you have now done, the main suspect is the meter itself.
@ArnieO I just tried an old firmware I got from @gskjold in a mail back in September 2021. It just says "SNAPSHOT" and it's pretty outdated. However, it was the firmware I was running up until it started failing a few days ago. I see the same corrupt data output, however, it looks truncated:
Another update: I spoke with @porsager (he commented on #126) on Twitter, and he told me that his grid company reset his meter remotely and it worked after that. Not sure how that was done, but I was told by my grid company, that they couldn't do something like that. sigh. The guy at Kamstrup told me, that I should try getting my grid company to contact him directly, so that might be my best shot at this moment.
@mortenmoulder have you tried writing an email to klakj@nexel.dk that I linked above ? He was helpful resetting mine which works now.
@porsager I have not, but I will definitely do that now! Thanks.
Resetting the meter is always a good advise, and it can be done remotely (I have several customer that got a non-working meter running that way. However, the symptoms here are different - and @mortenmoulder has had a reset done by disconnect/reconnect - and it did not help. But it should be an easy thing to try!
@ArnieO Do you know what the configuration in OmniSoft VisionAir should be set to regarding the push list? I've been told it's either HAN-NVE or DLMS PUSH 1.
When I search for DLMS in issues here on GitHub, I see plenty of people getting valid DLMS data back. So it should probably be set to DLMS Push 1, yes?
@ArnieO Do you know what the configuration in OmniSoft VisionAir should be set to regarding the push list? I've been told it's either HAN-NVE or DLMS PUSH 1.
When I search for DLMS in issues here on GitHub, I see plenty of people getting valid DLMS data back. So it should probably be set to DLMS Push 1, yes?
My understanding is that both will work.
Kamstrup meters in Norway push a smaller payload (which does not include accumulated energy [kWh]) each 10 seconds. A larger payload which includes kWh arrives only at each whole hour. I have the impression that Danish Kamstrups push the larger list all the time. This could be the difference between "HAN-NVE" (where the HAN refers to the Norwegian directorate) and "DLMS PUSH 1".
In both cases, the payload is in DMLS format, so maybe just a "somewhat arbitrary" name choice at Kamstrup.
@ArnieO Yeah I just got confirmation from Nexel (thanks @porsager) that it really doesn't matter if it's DMLS PUSH 1 or HAN-NVE. Both send the same "variable names" - HAN-NVE just sends less data.
Hey guys.
I am seeing similar errors, posting them here and hoping that you'll see what's wrong (I don't :)
I have a Kamstrup, I live in Norway (Tromsø) and every 10s I get the following on the telnet console:
(V) (unwrapData)(C1) HDLC frame:
(V) 7E 13 2B
(V) (readHanPort)(C1) Unknown data payload:
(V) 7E 13 2B 48 A2 D2 E6 1E 0F 00 00 E8 E6 E8 03 91
(V) A4 FF 00 02 46 21 68 85 6D 2E 97 D5 70 65 C1 30
(V) 8A 09 50 01 00 41 0A 54 BA 30 53 B2 37 A6 13 82
(V) 38 4C 26 C2 09 50 01 05 01 0A 22 13 A2 31 4C A6
(V) 12 39 32 D3 8A 30 4C A2 30 42 50 01 40 10 FF 10
(V) 00 46 25 06 40 02 10 FF 10 00 00 42 50 01 A0 10
(V) FF 10 00 00 42 50 01 D0 00 06 00 50 42 50 01 74
(V) 00 06 00 C8 09 50 01 A6 10 FF 10 00 41 09 50 01
(V) D4 00 06 00 41 58 C8 01 40 3A 00 12 EA 42 50 01
(V) D3 00 12 E9 42 50 01 EA 00 12 ED 8D F3 FF
The HDLC frame is always 7E 13 2B
.
The payload seems to always start with 7E 13 2B 48 A2 D2 E6 1E 0F 00 00 E8 E6 E8 03
-
Most payloads end with FF
, but not all. I've seen both 7E
and F9
.
The firmware is v2.1.15.
Does this make any sense to you?
Let me know if you need additional data. Thanks!
Does this make any sense to you?
The payload should also end with 7E, this looks like read error to me. What hardware do you use?
Meter serial config 2400 baud 8N1 ?
Meter serial config 2400 baud 8N1 ?
That was it. I had it set to 8E1, but I must admit I had to look at it three times before I realised my mistake.
Embarrassing, but luckily the fix was quick. Thanks, all.
No worries, it's a classic :)
Sorry to revive this issue, but it seems like I'm facing the same issue.
I export the metrics to Prometheus though Home Assistant, and on February 8 I stopped receiving data:
There's been no change to my meter or to my PowK+. At the time I was running AMS reader 2.1.7, but I've just tried flashing with the most recent release, 2.2.8, and that didn't change anything.
The AMS reader web UI still shows data, but it doesn't seem to match the current usage -- I'm guessing that it might show stale data from when it was receiving data from the HAN port last?
Enabling debug logs just show that it's not receiving any data:
$ socat stdout tcp:192.168.1.6:23 | ts %s | tee -a ~/tmp/amsreader.log
1676919347 *** Remote debug - over telnet - for ESP32 - version 3.0.5
1676919347 * Host name: ams-cf58 IP:192.168.1.6 Mac address:58:CF:BA:BA:B8:FE
1676919347 * Free Heap RAM: 113980
1676919347 * ESP SDK version: v4.4.3
1676919347 ******************************************************
1676919347 * Commands:
1676919347 ? or help -> display these help of commands
1676919347 q -> quit (close this connection)
1676919347 m -> display memory available
1676919347 v -> set debug level to verbose
1676919347 d -> set debug level to debug
1676919347 i -> set debug level to info
1676919347 w -> set debug level to warning
1676919347 e -> set debug level to errors
1676919347 s -> set debug silence on/off
1676919347 l -> show debug level
1676919347 t -> show time (millis)
1676919347 profiler:
1676919347 p -> show time between actual and last message (in millis)
1676919347 p min -> show only if time is this minimal
1676919347 P time -> set debug level to profiler
1676919347 c -> show colors
1676919347 filter:
1676919347 filter <string> -> show only debugs with this
1676919347 nofilter -> disable the filter
1676919347
1676919347
1676919347 * Please type the command and press enter to execute.(? or h for this help)
1676919347 ***
1676919350 (D) Serving /data.json over http...
1676919352 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919355 (D) Serving /data.json over http...
1676919358 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919360 (D) Serving /data.json over http...
1676919364 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919366 (D) Serving /data.json over http...
1676919370 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919371 (D) Serving /data.json over http...
1676919376 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919376 (D) Serving /data.json over http...
1676919381 (D) Serving /data.json over http...
1676919382 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919386 (D) Serving /data.json over http...
1676919388 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676919391 (D) Serving /data.json over http...
@mortenmoulder did you find a fix?
Thanks!
@runejuhl Does the LED on the Pow-K blink blue each 10 second?
No, it blinks red every ~6 seconds. This matches the interval in the logs:
1676930099 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676930105 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676930111 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676930117 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
1676930123 (W) (errorBlink)(C0) No HAN data received last 30s, single blink
The first column is Unix epoch time.
The blue part of the LED is directly connected to the incoming data signal. No blue blinking == No incoming data
We have seen several cases where the meter needs to be restarted. If I were you, I would contact the grid company and request a remote reboot of the power meter. (You could also reboot it by turning off and on your main breaker, but in most modern homes that means more hassle.)
Describe the bug At 05:58 PM today, my setup that has been working flawlessly every single day for the past year stopped working.
Got these
Incorrect frame format
errors when I enabled debugging:I thought it was an error with my ESP8266 (like it might have died), so I plugged in an ESP32 and flashed it with the latest firmware and got
Ended up in default case while unwrapping
andGot to end of unwrap method
:My keys are the exact same as on the ESP8266.
What could the issue be here? 05:58 is awfully close to 06:00, which could be a time where my provider has sent out a new set of keys perhaps? Or an update?
Hardware information:
Relevant firmware information: