Data not showing

[petergam]

Hey @gskjold

I have an issue where the data is not shown using my Danish Kamstrup Omnipower meter running on POW-K from @ArnieO. The HAN module is just red.

Here is a dump from the log

(D) Preparing for next frame
(I) (HanReader)Got valid DLMS data (247 bytes)
(I) (HanReader)Full DLMS frame (256 bytes)
(I) 7E A1 00 41  03 13 2F 7A  E6 E7 00 DB  08 4B 41 4D
(I) 45 01 B5 02  D3 82 00 E7  30 00 00 0E  25 29 5A 25
(I) A2 42 2D 6A  32 0A 9F 7C  55 63 35 96  D8 5D 79 A8
(I) 83 D8 96 D9  B9 82 22 D4  29 0A B1 CE  EB 38 2D C3
(I) 59 70 60 4F  89 A8 CB 09  AA 2E D5 2C  DE 1A C0 6F
(I) 71 05 77 A2  CE F5 98 04  5C E0 3C B8  92 1B 66 C5
(I) F5 E7 05 41  C1 30 96 16  21 02 43 3E  A5 9A FD FA
(I) 32 5E 5A 71  80 B8 33 77  9B 95 39 41  F3 53 6F 15
(I) 13 BB 49 F6  01 46 85 59  1B 02 E9 99  9A DC DB 72
(I) BA B0 2C 2C  7C F1 54 46  66 8D 97 1D  A3 4F 56 8C
(I) A1 6E C5 59  3C 39 01 A0  48 B3 E1 E5  E6 57 E2 D3
(I) BD CA D1 BC  8F 89 92 E8  85 4B 9F A1  AD 93 87 3A
(I) 4C C4 D2 D3  84 07 3C 14  12 C0 67 3D  74 1B 7D 98
(I) 91 34 B3 A4  74 DD C6 57  5F E6 3C EE  59 20 09 E7
(I) 6C 96 A9 6E  DB BA 45 44  C2 8D FB 14  06 6C 82 E0
(I) 96 F3 DE 7B  7A 2E 12 65  96 90 C5 08  38 17 DC 0D
(I)
(I) (HanReader)Decrypting frame
(D) (HanReader)System title:
(D) 4B 41 4D 45  01 B5 02 D3
(D) (HanReader)Initialization vector:
(D) 4B 41 4D 45  01 B5 02 D3  00 00 0E 25
(D) (HanReader)Additional authenticated data:
<REDACTED>
(D) (HanReader)Authentication tag:
<REDACTED>
(D) (HanReader)Encryption key:
<REDACTED>
(D)
(D) (HanReader)Data after decryption:
(D) E6 E7 00 DB  08 4B 41 4D  45 01 B5 02  D3 82 00 E7
(D) 30 00 00 0E  25 0F 00 00  00 00 0C 07  E5 08 1E 01
(D) 16 14 1E FF  80 00 80 02  19 0A 0E 4B  61 6D 73 74
(D) 72 75 70 5F  56 30 30 30  31 09 06 01  01 00 00 05
(D) FF 0A 10 35  37 30 36 35  36 37 32 38  36 33 39 39
(D) 35 35 33 09  06 01 01 60  01 01 FF 0A  12 36 38 34
(D) 31 31 33 31  42 4E 32 34  35 31 30 31  30 31 33 09
(D) 06 01 01 01  07 00 FF 06  00 00 00 A9  09 06 01 01
(D) 02 07 00 FF  06 00 00 00  00 09 06 01  01 03 07 00
(D) FF 06 00 00  00 00 09 06  01 01 04 07  00 FF 06 00
(D) 00 01 76 09  06 01 01 1F  07 00 FF 06  00 00 00 24
(D) 09 06 01 01  33 07 00 FF  06 00 00 00  5C 09 06 01
(D) 01 47 07 00  FF 06 00 00  00 58 09 06  01 01 20 07
(D) 00 FF 12 00  E8 09 06 01  01 34 07 00  FF 12 00 E7
(D) 09 06 01 01  48 07 00 FF  12 00 E8 7B  7A 2E 12 65
(D) 96 90 C5 08  38 17 DC
(I) (HanReader)HAN data is valid, listSize: 25

I've been able to decrypt the data on my own machine using the python script below.

from Crypto.Cipher import AES # Library: PyCryptodome
encryption_key = bytearray.fromhex('<REDACTED>') # gpk60
authentication_key = bytearray.fromhex('<REDACTED>') # gpk61
cipher_text = 'DB 08 4B 41 4D 45 01 B5 02 D3 82 00 E7 30 00 00 0E 25 29 5A 25 A2 42 2D 6A 32 0A 9F 7C 55 63 35 96 D8 5D 79 A8 83 D8 96 D9 B9 82 22 D4 29 0A B1 CE EB 38 2D C3 59 70 60 4F 89 A8 CB 09 AA 2E D5 2C DE 1A C0 6F 71 05 77 A2 CE F5 98 04 5C E0 3C B8 92 1B 66 C5 F5 E7 05 41 C1 30 96 16 21 02 43 3E A5 9A FD FA 32 5E 5A 71 80 B8 33 77 9B 95 39 41 F3 53 6F 15 13 BB 49 F6 01 46 85 59 1B 02 E9 99 9A DC DB 72 BA B0 2C 2C 7C F1 54 46 66 8D 97 1D A3 4F 56 8C A1 6E C5 59 3C 39 01 A0 48 B3 E1 E5 E6 57 E2 D3 BD CA D1 BC 8F 89 92 E8 85 4B 9F A1 AD 93 87 3A 4C C4 D2 D3 84 07 3C 14 12 C0 67 3D 74 1B 7D 98 91 34 B3 A4 74 DD C6 57 5F E6 3C EE 59 20 09 E7 6C 96 A9 6E DB BA 45 44 C2 8D FB 14 06 6C 82 E0 96 F3 DE 7B 7A 2E 12 65 96 90 C5 08 38 17 DC'# 0D'
cipher_text = bytearray.fromhex(cipher_text.replace(' ',''))
system_title = cipher_text[2:2+8]
initialization_vector = system_title + cipher_text[14:14+4]
additional_authenticated_data = cipher_text[13:13+1] + authentication_key 
authentication_tag = cipher_text[len(cipher_text)-12:len(cipher_text)] 
cipher = AES.new(encryption_key, AES.MODE_GCM, nonce=initialization_vector, 
mac_len=len(authentication_tag))
cipher.update(additional_authenticated_data)
plaintext = cipher.decrypt_and_verify(cipher_text[18:len(cipher_text)-12], 
authentication_tag)
print(plaintext.hex())

-----
0f000000000c07e5081e0116141eff80008002190a0e4b616d73747275705f563030303109060101000005ff0a103537303635363732383633393935353309060101600101ff0a1236383431313331424e32343531303130313309060101010700ff06000000a909060101020700ff060000000009060101030700ff060000000009060101040700ff0600000176090601011f0700ff060000002409060101330700ff060000005c09060101470700ff060000005809060101200700ff1200e809060101340700ff1200e709060101480700ff1200e8

It seems like the decryption is succesfull since the result I get is similar to what is printed in the logs. Converting this using http://www.gurux.fi/GuruxDLMSTranslator gives

<DataNotification>
  <LongInvokeIdAndPriority Value="00000000" />
  <!--2021-08-30 22:20:30-->
  <DateTime Value="07E5081E0116141EFF800080" />
  <NotificationBody>
    <DataValue>
      <Structure Qty="19" >
        <String Value="Kamstrup_V0001" />
        <!--1.1.0.0.5.255-->
        <OctetString Value="0101000005FF" />
        <String Value="5706567286399553" />
        <!--1.1.96.1.1.255-->
        <OctetString Value="0101600101FF" />
        <String Value="6841131BN245101013" />
        <!--1.1.1.7.0.255-->
        <OctetString Value="0101010700FF" />
        <UInt32 Value="000000A9" />
        <!--1.1.2.7.0.255-->
        <OctetString Value="0101020700FF" />
        <UInt32 Value="00000000" />
        <!--1.1.3.7.0.255-->
        <OctetString Value="0101030700FF" />
        <UInt32 Value="00000000" />
        <!--1.1.4.7.0.255-->
        <OctetString Value="0101040700FF" />
        <UInt32 Value="00000176" />
        <!--1.1.31.7.0.255-->
        <OctetString Value="01011F0700FF" />
        <UInt32 Value="00000024" />
        <!--1.1.51.7.0.255-->
        <OctetString Value="0101330700FF" />
        <UInt32 Value="0000005C" />
        <!--1.1.71.7.0.255-->
        <OctetString Value="0101470700FF" />
        <UInt32 Value="00000058" />
        <!--1.1.32.7.0.255-->
        <OctetString Value="0101200700FF" />
        <UInt16 Value="00E8" />
        <!--1.1.52.7.0.255-->
        <OctetString Value="0101340700FF" />
        <UInt16 Value="00E7" />
        <!--1.1.72.7.0.255-->
        <OctetString Value="0101480700FF" />
        <UInt16 Value="00E8" />
      </Structure>
    </DataValue>
  </NotificationBody>
</DataNotification>

however something must go wrong when parsing the result since it doesn't show any data in AMS or by accessing /data.json directly.

Any help is highly appreciated.

[ArnieO]

Did you tro to enter the encryption keys without the initial «0x»? I think (@gskjold will probably confirm) the code just wants the hexadecimal keys without the preceding 0x that signifies that the string is hexadecimal.

[petergam]

Did you tro to enter the encryption keys without the initial «0x»? I think (@gskjold will probably confirm) the code just wants the hexadecimal keys without the preceding 0x that signifies that the string is hexadecimal.

If I do that and press save the web UI will automatically add the 0x prefix again.

[gskjold]

Your python decoder only shows the decrypted payload, while my firmware shows the entire frame wrapped around the decrypted payload. You will find the result from your python decoder inside the frame starting in the second line.

Anyway, the problem here is that the DLMS decoder only works if the list size of the payload is correct for the type of meter selected. Right now this only supports the frame I was given by the guy who originally asked for this implementation, which had a list size of 65, not 25. But it looks like you have the same frame as used for unencrypted Kamstrup in Norway, so attached is a firmware the hopefully works for you.

firmware.zip

[ArnieO]

Strange case. Around 20 of my cards are running in Denmark now, and I have not seen this before. If other similar cases show up I will direct them to this thread. @petergp : Email me on post@amsleser.no if you need guidance on how to update the firmware. It can be done via the GUI (easiest) or by cable/FTDI.

[petergam]

I updated the firmware and everything seems to be working correctly now. Thanks for the quick help @gskjold and @ArnieO.

[ArnieO]

Good to hear! But I am puzzled as to the reason for the issue you experienced here, so taking due note of this.

[gskjold]

Commited fix, due for next release.

[petergam]

I wrote an email asking my electricity provider if it would be possible to change to a list size of 65 every 10th second to get all the data more frequently. They replied that my meter was setup to use the norwegian list but they have now changed it to the danish list. I guess this explains the confusion.

[ArnieO]

Thank you for that information @petergp - very useful to know in case I run into the same issue again!