Add support for encrypted meters

[miafoo]

I've got a Kamstrup Omnipower 3-phase meter and I've connected a Wemos D1 mini (GND to GND and D1 to DATA_OUT - see image below). I've successfully set up the wifi connection and mqtt but I'm not receiving any data. I suspect this is because the power meter data is encrypted here in Denmark, although I would still expect to at least get raw data? I've got an encryption key as well but not sure where to start.

[gskjold]

I think it's a safe bet that the reason is encryption. The code does not dump unknown data to MQTT, although this is something I have been wanting to do. You don't happen to have any information about what type of encryption is used?

Looking through the code, it looks like the LED should flash when it receives data from the meter (as long as meter type is set in config) regardless of it being able to interpret it or not. Is the LED flashing?

From what I can tell from a custom Kamstrup board I have in my collection, your connection is correct.

If you enable telnet and/or serial debugger in system and set the logging level to DEBUG, you should see the raw data in the terminal. Remember to disable the telnet debugger when not in use.

Not sure how we proceed from there, I guess the encryption key is possibly meant to be kept secret, but I am interested in implementing decryption in my code.

Attached is a version of the code that dumps the raw data to MQTT. This is built on the dev-1.3.0 branch and is untested. Also, if you use this binary, you will have to wipe the flash or degrade to earlier version and set it up all over again before upgradring to 1.3.0 when that finally comes out.

firmware.zip

[miafoo]

I tried connecting over telnet with debug level set but I did not see anything in there unfortunately. I will try flashing the firmware you linked and report back later.

Also just found this and this through a Danish forum which seems to be implementing encryption, perhaps that can be of some help. I might give it a try later, although I see they've both got 2 keys (conf key and auth key) and I was only provided 1 key from Radius so I will try calling them again to see if I can get the other one.

Edit: It seems the key they provided me was for the Smart-Me module. They're looking into getting me the other keys now.

[gskjold]

OK cool, for reference, they seem to be both using this library: https://github.com/wolfeidau/mbedtls

[miafoo]

Here's a document with some specifications, section 6. and 6.1 has info about the encryption.

5512-2584_GB_A2_11-2019 OMNIPOWER HAN interface specification.pdf

[miafoo]

I tried flashing the firmware you provided during my lunch break however I can't seem to connect to the AP. If I try to bridge D4 to GND to force AP mode during bootup the LED just stays on but nothing else. If I don't bridge the pins the D1 Mini's built-in LED does blink every few seconds or so at least.

Just to be sure it was an ESP8266/d1mini binary you linked earlier right? I had a similar issue getting into AP mode when I had accidentally flashed lolind32.

[gskjold]

It was the d1mini build, yes. I uploaded it to my own d1 mini before sending it to you. Maybe the conversion from old config to new config is broken. Try erasing flash and starting the image from scratch

[gskjold]

Btw, while running, GND D2 (GPIO 4) for AP mode for 5sec. D4 is the LED pin

[miafoo]

Flashing fixed the AP problem and I've got remote debugging up and running now and also set the meter type to kamstrup. However still no data in remote debugging sadly.

Enabling MQTT seems to completely crash and then it refuses to boot up (tried three times) and then I have to reflash it and start over. I'm not sure if that's related to a very long password which has a mix of different symbols? Although it worked on the previous official release.

Let me know if there's anything else I can do, I'm really keen on getting this to work and would like to help if possible.

[miafoo]

I've just received the two encryption keys as well as documentation now:

DLMS-COSEM.pdf 5512-2584_GB_A2_11-2019 OMNIPOWER HAN interface specification.pdf

[gskjold]

I hope to get around to testing v1.3.0 this weekend so that MQTT will work for you. I find it a bit weird though that you do not see a HEX dump of the payload in debug with v1.2 series firmware.

Looking through the documentation you attached, the payload is formatted differently from what we have in Norway, so after the decryption is solved, there will still be some work to handle the new payload.

[gskjold]

Attaching semi tested firmware for d1 mini. Under meter configuration there is now a checkbox for "Send unknown packets to MQTT" which enables HEX dumping unknown data to MQTT.

Edit: Actually attaching file :)

firmware.zip

[miafoo]

Awesome! I just flashed it and it seems much more stable and I'm also getting dumps when enabling remote debugging via telnet (see an example below). I'm not receiving anything via MQTT though. The raw data also comes through very irregularly.

I also managed to cook up my own simple serial reader that spits the raw data over MQTT during the weekend and noticed that the pattern doesn't match the specification in the document (or maybe I'm doing something wrong?) The header is as described but in the footer the only consistency seems to be the final 0x7E before next frame, the rest appears to be random and likely part of the encrypted data or auth tag.

Edit: ~For some reason the raw data coming in from my reader today doesn't match neither the header or footer...~

Edit2: I've found the problem and am now reliably getting the header again. I will paste a couple of frames soon

(I) (HanReader)Got valid DLMS data (480 bytes)
(D) E6 E7 00 DB  08 4B 41 4D  45 01 AC 4D  6E 82 01 D0
(D) 30 00 00 65  0F 03 88 94  FA 6D 05 8A  B1 FF 38 E5
(D) 2B 93 3A 6E  3A 74 9A C3  86 68 3F 24  80 9B EC 1F
(D) 61 5B 6D 15  2A AC 47 D8  C0 D8 9E 24  A2 7A 87 20
(D) DE 2C 97 FC  5D 8C 1D 6D  1C 94 80 DD  77 99 25 F5
(D) 3D A3 C8 9F  D3 14 5B 7E  C6 E2 BB 8F  EB 10 B1 95
(D) 71 13 CF 4D  EA 06 F5 EF  F6 9E 71 28  E8 8D AC 54
(D) 97 85 E6 B7  AB 8B EA 80  0B 11 EA C6  8D 89 98 9E
(D) 6F 10 44 75  99 E1 5C 96  54 66 5B 3E  94 A1 AF 7D
(D) 14 C2 DF F7  41 F3 B8 6D  C5 2C C5 AC  70 54 4E 65
(D) CD 70 8E 5E  B1 29 9B 50  36 98 94 89  3E 49 B9 15
(D) A5 88 F2 CD  B4 A2 2D AC  B1 9A A4 14  86 82 97 F8
(D) 32 0F BD 71  FB 1C E2 DF  22 20 27 3A  A4 F1 82 54
(D) A0 34 50 51  A9 2D 74 8A  C3 D7 A2 E6  3D 71 4F CB
(D) 15 5A 92 77  B5 B6 E0 60  49 9D C2 E4  3E 95 48 B3
(D) 45 03 98 03  22 A7 58 CC  04 20 11 A5  F4 87 B8 21
(D) AC E5 95 B4  05 5B E3 42  4F AB B5 59  1F E5 51 42
(D) 34 87 03 52  F6 92 E4 26  A3 AF 61 9E  BD 71 4C 20
(D) DA 0A DD FD  E8 47 5C 86  66 15 A2 48  49 25 63 26
(D) F8 FD 27 A1  35 FA E0 65  B2 A7 6F 5C  11 6D AF 7B
(D) 5B F2 FB 33  20 CC 2A 60  49 FC 6D 2C  AB CF F5 E3
(D) A0 A7 E9 15  74 F3 7E E3  57 83 AC 1E  93 AA 4A AA
(D) 4E 9B 33 E6  5B 5C F0 45  D6 2E 0B C0  A1 14 A9 C3
(D) 71 A3 A2 FA  4B 61 32 D2  A6 0F CF 1B  DB 21 58 82
(D) 61 2A 87 03  93 EB C6 59  F6 E6 27 49  73 22 EB 32
(D) 5B 66 06 8C  4A A1 EB E7  27 B3 CD 88  C4 C6 3F BF
(D) 77 52 70 0B  FB 15 A2 95  7A 96 68 C4  4B 98 56 5A
(D) A7 86 17 95  20 2E 91 81  00 B4 52 A7  4E D8 29 AE
(D) D9 E9 34 1B  81 DA 77 68  6D 4C F6 C8  22 46 68 D5
(D) D3 01 21 29  A5 98 61 25  6D F3 A7 4E  3A 02 C9 56
(D)
(W) (HanReader)Invalid HAN data: Start should be E6 E7 00 0F

[miafoo]

Here are a couple of frames from my power meter:

7E A1 E9 41  03 13 C6 37  E6 E7 00 DB  08 4B 41 4D
45 01 AC 4D  6E 82 01 D0  30 00 00 A3  2E 4D F3 55
18 2E A7 73  28 47 05 5C  E8 B6 67 1C  B8 EB DD 36
F6 BD 18 34  03 77 84 C3  7A DF 12 5A  BF 2F 9E 54
95 AB 9D 76  ED D2 62 0D  04 03 77 35  BA 3A 84 EC
F3 E1 9C 4D  43 74 A0 A9  0B C5 AF 96  F1 88 9F 14
41 BD A0 C3  69 A3 5D E4  F8 DC 7E 80  0D 9B 6A 2C
25 97 18 DA  3A 0E 6F 9B  A7 FD 37 A8  40 2E 91 EF
AA 8E F9 CD  32 00 E8 C2  E5 6B DD A0  E4 44 91 90
44 FD 9D 59  13 40 3B 79  21 86 4D FC  3C F4 66 A2
F6 51 42 D2  BD 2E FE 51  37 86 34 14  32 EF 76 F9
01 E3 D2 14  1A 2F 02 D2  A8 D7 DB 3C  EC 0B BD 12
A4 6B 37 F8  35 54 BB E1  C8 3D 1C 80  64 DC DA B5
EB EF 8A 51  23 41 A8 26  33 FA AC FF  38 4D 58 DE
EB BD BA 71  81 67 31 F5  13 90 64 34  F7 AD 7D 90
50 4C 82 54  FA 03 48 27  E4 8B 55 D6  F3 53 EC 96
09 FF 21 DF  DF 52 F3 61  EF CC 55 BA  95 9E 15 26
F8 85 C5 67  6D 4E 99 F1  D5 C5 E6 37  B4 CD D0 03
E3 22 2F BB  76 10 79 CC  58 9F F8 FA  90 B4 C1 81
4B CE CC 6F  29 37 51 53  EF 9C EC 8B  32 F7 B8 2B
A5 7A EA 59  2F 4E 5B 42  1A 7B 4F 0F  BE 33 05 01
C7 55 66 02  76 11 6F 90  B1 99 37 3C  BD 31 86 0D
B6 97 2E DE  06 9A FA DA  37 B3 CB C8  C8 70 CD 4A
D6 04 50 EA  A6 AA D8 AE  B4 D1 E3 8C  8B 5D D5 39
F0 92 68 3E  25 E2 5E 80  DE 18 FC E5  C6 9C 51 3E
2F 5A 07 6E  C8 CB 96 EE  09 F0 3A DA  61 98 68 6A
F1 9B 04 0D  3D C2 A0 52  A2 DB F7 62  7F EA 94 33
98 4E AF E5  09 AD 03 E8  84 D8 16 39  79 55 02 5A
5B 48 5B 82  F3 74 7B 74  BC C0 55 34  2E 04 37 43
10 AC 80 2D  8A 03 FC 3E  91 79 88 8E  33 74 71 01
15 B3 4C 51  C1 AC 45 28  5A 6B 7E

7E A1 E9 41  03 13 C6 37  E6 E7 00 DB  08 4B 41 4D
45 01 AC 4D  6E 82 01 D0  30 00 00 A3  2F 85 B3 61
EB D2 96 E0  4D 08 5D 70  1A B9 C8 B8  F2 8F DA EC
79 8F 57 68  A1 1C 22 23  D7 5C A5 E0  0F 63 DF 1E
F0 95 B8 A6  3B DA C3 12  10 4C 45 47  3A 30 03 5A
E2 6B 4A C3  90 E7 4E A9  3C 65 C1 84  72 13 A5 F5
33 AA 01 D1  47 D5 51 61  5D 0B 30 3A  07 97 27 F0
E1 11 C8 95  9D 4D 2C 93  60 7C 7F 58  6E 1D 43 DE
C4 D5 F9 11  CD 1D AB BB  DB 00 12 D5  3A 18 D5 77
9D 6E 78 FD  BC C9 20 C5  0A 06 A1 00  DA 97 47 6B
86 87 2D 48  18 D4 9A EB  7D 8A C9 DA  F9 BA FF 4F
3D FD 1F 7E  1F 34 CA 82  EE FB 51 F6  59 D1 07 35
D0 5D 7E 2A  D0 C4 01 4C  50 1C CA 9E  E0 9B 9F 85
0C 5C 6D 63  75 D5 51 69  71 90 72 9F  84 77 3B 9E
09 6F 1D B3  67 F3 76 D3  DC F0 1F E0  EE 38 C8 FC
20 25 79 9C  29 87 23 4F  49 5A 46 3D  D7 E0 25 7E
F5 D9 46 E1  3B 21 C5 EE  50 45 21 2E  29 14 CE E5
AE FC 8D 67  24 E3 25 28  41 A6 C6 D5  29 E9 CA 05
D9 8E 16 AC  59 80 93 B5  CE 54 42 67  DF 72 AA 0E
21 61 3B 1F  42 B2 2C 25  6A 1E A6 63  2B 8E 3D 0D
4E A2 80 B2  86 2A CD D6  D1 95 94 41  99 65 FE 10
14 F3 B3 89  13 35 0A 5A  BC D4 FE BA  8F 00 F6 7D
92 EE E5 9D  1A C5 97 47  31 8F ED 63  22 CC E0 6E
C7 D1 CB 1F  CE B5 A2 71  43 C8 0C E3  21 2D 25 8D
18 6A 9B 8C  7B 03 F0 AD  D3 6B AD 8E  95 AC 8A 53
C6 43 99 16  73 4A 89 96  53 68 2C 81  62 35 4D 0C
D2 BC 89 C6  CE FC 39 1E  90 EA D4 B4  E1 6E 5D CC
53 F6 65 93  19 EB AD E9  49 AF C0 D1  3B 88 9E B9
2E 14 42 67  06 96 2F F9  51 FB BE 60  B2 34 1E 6A
F1 DE B1 E9  67 BB 87 63  F5 2E DA 49  5B C3 CD 5E
79 18 18 DA  9F 97 85 FF  5A 84 7E

[gskjold]

Thanks for the frames! Sorry I'm a bit slow here, I've had too many things going on lately. I will concentrate on finishing v1.3.0 over the next week or so, then I'll get back on this one sometime after that. Hopefully I have some time to spare shortly after.

[miafoo]

No problem and no rush! The frames above are encrypted by the way. I can share the encryption keys with you but I'd prefer not to post them publicly here (even if they're not really useful on their own).

[gskjold]

Thanks, I'll let you know when I get as far as needing keys

[gskjold]

Currently have a rough implementation ready, if you are still interested in sharing your keys, email me at gunnar.skjold at gmail.com

I'm still not sure how well all this fits into the project just yet, but if what I have put together at this point works, it should fall in place I think.

[miafoo]

Awesome! I've just sent you an email with the encryption keys. Let me know if you got them.

[gskjold]

Thanks, email received :)

[miafoo]

Just FYI, I will be unavailable for the next 3 weeks and I won't be able to test anything during that time. If you need anything tested I can do it today and maybe tomorrow. No rush though! :)

[gskjold]

Attached is a esp8266 firmware supporting your meter. firmware.zip

[gskjold]

Branch dev-v1.4.0

[miafoo]

I just flashed it and it appears to be somewhat working! The frequency of the blobs are very random and sometimes the han status switches from success to warning or danger in /data.json. The data value in json blobs also sometimes just returns Connected!.

With actual data:

{
  "id": "F4:CF:A2:64:E0:A5",
  "name": "ams-kamstrup",
  "up": 8059,
  "t": 1596618590,
  "vcc": 3.294,
  "rssi": -78,
  "data": {
    "rtc": 1596618590,
    "tPI": 12171.58,
    "tPO": 0,
    "tQI": 47.51,
    "tQO": 6678.59,
    "lv": "Kamstrup_V0001",
    "id": "",
    "type": "",
    "Q": 0,
    "PO": 0,
    "QO": 336,
    "I1": 0.84,
    "I2": 1.37,
    "I3": 0.08,
    "U1": 233,
    "U2": 233,
    "U3": 233,
    "P": 341
  }
}

Just "Connected!" returned:

{
  "id": "F4:CF:A2:64:E0:A5",
  "up": 634,
  "data": "Connected!",
  "vcc": 3.29248,
  "rssi": -81
}

[miafoo]

After the last "Connected!" blob in mqtt (about 15 minutes ago) I haven't received anything else and the web interface doesn't seem to be working at all anymore either.

[gskjold]

Could be that it runs out of memory, does the UI come back if you reset the device? Any output in the serial console? Also, try running with debugger on level ERROR to avoid using memory for debugging

[miafoo]

After restarting it, the web interface is working again however HAN is not. I checked the encryption key and it's not the same as I entered - it's now padded with 0X0000 and the last chars are missing. I just tried reverting it but HAN is still not working. After changing the key a few times the web interface stopped working again, so I think it's possible there might be a memory issue.

I will try checking serial console and running with debugger on ERROR level today if I manage to find some time.

[gskjold]

I've now fixed the problem with zero padding on the keys. Not sure if you used my binary from zip or compiled from source, but if you compiled from dev-1.4.0 yesterday, I have also fixed some other issues that might cause memory issues.

I would try debugger on error and possibly also without mqtt just to see if it could maintain a stable reading.

firmware.zip

[miafoo]

Flashed the firmware you linked and the web interface seems slightly more stable (but still very unstable and at times completely unresponsive) and I've only enabled Telnet debugging on Error level, no MQTT or anything else so far. There's nothing interesting reported in Telnet either and I have yet to try Serial.

HAN readouts seem extremely random at best and consistently goes from success -> warning -> danger. I can sometimes "force" a readout by going to Meter settings and submitting the form again. It almost feels like it only reads the first frame and then stops until I apply a new setting which possibly reboots the D1?

[gskjold]

I have just made some software that sends the two frames I received from you earlier in a loop and hooked it up to my d1 mini i use for development. It receives the data without problem, decrypts and decodes as expected. Could there be some harware issue on your end? Or maybe just try a full chip erase, flash and reconfigure from scratch.

If you are still having problems, I would like a few more frames from you. I would also like to know what hardware you have and what browser you use to configure the device as well as the operating system running your browser.

[miafoo]

I've tried configuring with Firefox and Safari on Mac and Firefox on Windows 10. I tried a full chip erase, flash, and reconfigure but didn't change anything.

As for hardware, I believe this is the ones I ordered (which is likely a clone, so that could possibly be the problem): https://www.banggood.com/Geekcreit-D1-mini-V2_2_0-WIFI-Internet-Development-Board-Based-ESP8266-4MB-FLASH-ESP-12S-Chip-p-1143874.html?rmmds=myorder&cur_warehouse=CN

I'll try and get some more frames for you but I can't promise when I have time. 👍

I really appreciate the time and effort you are putting into this.

[gskjold]

Thanks for the update, I am currently wrapping up v1.4 and will run a thorough test of the UI in that process.

If your board has a ESP12E sitting on top like that, it is a clone. A genuine D1 should look like this:

I think it should probably still work though, so it's hard to tell whats going on... I can try one of my boards with ESP12E chip and see if that behaves in the same way.

I have recently read up on DLMS communication, so I will have another look at the parser in this project and see if there is anything that is incorrect there.

[gskjold]

Merging v1.4 down to master for release. Omnipower support will be announced as beta. I have also added additional debug output in the reader code so that incomplete or damaged data will be written to debugger.

[ArnieO]

IF @mikfoo draws current for the Wemos from the Vcc pin in the 6-pin connector on the meter (see starting post in this thread), the issues he has experienced could potentially be due to overcurrent protection on the Omnipower.

When the ESP transmits over Wifi it will draw some short and quite heavy current pulses, which the Omnipower might not appreciate. In that case, the meter could shut down the output for a period. I advise using a relatively beefy capacitor with low ESR on 3,3V to ensure the ESP has access to enough energy without pulling current spikes from the meter.

In case @mikfoo draws current for the Wemos from a separate power supply, adding a large capacitor between 3,3V and GND on the Wemos should anyway be tested.

[miafoo]

@ArnieO I used an external power supply to power the Wemos, and only used GND and DATA_OUT from the Omnipower.

I did manage to create my own little thing which has been working fine by frankensteining a couple of different open source projects together, so I don't think it's my hardware there's anything wrong with.

I haven't tried this project since v1.4 but I'm happy to test it again if needed.

[ArnieO]

@mikfoo : OK, thank you for the clarification, and it is good to hear that you have something up and running.

It would be great to find out why your setup does not work with V1.4. I note from your postings here above that the RSSI was very low (-78 and -81 dBm). Is this still the case with your setup that is working now, or have you changed anything that improves the Wifi coverage?

EDIT: Wrong thinking; sorry. The problem is HAN readout, not Wifi connection.

EDIT 2 We will soon have feedback on this card in an encrypted Omnipower meter in Denmark (I shipped a card yesterday): https://www.hjemmeautomasjon.no/forums/topic/5894-kamstrup-ams-m%C3%A5ler-wifi-adapter/ It works very well on a number of non-encrypted Omnipower meters in Norway - looking forward to getting feedback on that.

[miafoo]

@ArnieO the card looks really cool, impressive work! I'd love to test one and help debug if needed, are you selling them? Sorry if this is a bit off-topic.

[ArnieO]

@mikfoo Yes I do! Please register user on https://www.hjemmeautomasjon.no/ ... and DM me.

[miafoo]

Just to keep this issue updated, @ArnieO mentioned in a PM on hjemmeautomasjon.no that he got feedback from the other Omnipower user in Denmark and that everything is working fine.

I think we can close this issue. I will test it again once I receive @ArnieO's hardware and re-open it if I have any problems.

[gskjold]

Thanks for the update! Hope it works for you too :)

[miafoo]

I just received and installed @ArnieO's hardware and everything is working perfectly with encrypted meters.

Thanks so much for your time and efforts on this!

[gskjold]

That is great news! Good to hear that it is actually working :)