search

UtilitechAS / amsreader-firmware

ESP8266 and ESP32 compatible firmware to read, interpret and publish data to MQTT from smart electrical meters, both DLMS and DSMR is supported
Other
390 stars 73 forks source link

Unescaped control character in JSON string in 2.3.9 #867

Open cmock opened 1 week ago

cmock commented 1 week ago

Describe the bug 2.3.9 produces invalid JSON for the "power" MQTT topic:

00000000  7b 22 6c 76 22 3a 22 22  2c 22 69 64 22 3a 22 53  |{"lv":"","id":"S|
00000010  41 47 59 05 e8 94 b6 22  2c 22 74 79 70 65 22 3a  |AGY....","type":|

Specifically, the 0x05 at offset 0x13. In previous versions, the "id" key was empty.

To Reproduce Steps to reproduce the behavior:

  1. Parse the MQTT JSON
  2. See parser throw an exception

Expected behavior I'm not a JSON expert, but "jq" says "control characters from U+0000 through U+001F must be escaped".

Screenshots If applicable, add screenshots to help explain your problem.

Hardware information:

Relevant firmware information:

Additional context Here's the complete JSON string from the power topic attached so nothing gets lost in markdown: ams.json

ArnieO commented 1 week ago

Indeed, your JSON string gives a parse error due to that character. Using https://jsonbeautify.com/, I get this: image

It could look like your meter is providing unexpected input to the fields "id" and "type".

On my Kamstrup meter, I receive:

In order to see what your meter is sending, please run a Telnet debug and post the output here for decoding / verification.

cmock commented 1 week ago

Do I need to do anything special to enable configuration of debug mode? I can't see it in my config page... (using 2.3.9)

ArnieO commented 1 week ago

Do I need to do anything special to enable configuration of debug mode? I can't see it in my config page... (using 2.3.9)

You do not have this at the bottom of the Config screen? image

Noschvie commented 1 week ago

Seems to be an ESP32-C3 https://github.com/UtilitechAS/amsreader-firmware/issues/783#issuecomment-2143769912

ArnieO commented 1 week ago

Seems to be an ESP32-C3

In that case: Try to run debug with an older firmware, I think v2.3.4 might work. We have seen issues with binaries on C3, and since we are not using them in our products, finding the issue does not have high priority

Noschvie commented 1 week ago

Version 2.2.28 is working for me. https://github.com/UtilitechAS/amsreader-firmware/discussions/835#discussioncomment-10477141

cmock commented 1 week ago

No, it's an "original" ESP32, Info page says "Chip: esp32 (240MHz)". The last tile on the config screen is "Hardware" So, back to 2.2.28, and we get

(V) MBUS frame:
(V) 68 FA FA 68  53 FF 00 01  67 DB 08 53  41 47 59 05 
(V) E8 94 B6 81  F8 20 00 EA  75 E7 5C A4  1D 8F 8B 75 
(V) 4F 47 29 91  EC 08 B7 43  DA C8 A8 72  1D 48 13 D3 
(V) BE E1 2C 7C  5A BF 0C 71  BC 54 8C 31  4E 0F 6D B4 
(V) CD 94 52 CB  7B AB 7F B4  34 6C 55 AF  60 1F 0E 9A 
(V) CB EA B4 95  8B FD B9 2F  2D 1D 4F 88  54 EF F1 91 
(V) C6 24 48 F2  49 90 F5 1F  CF F5 9E 20  BB B9 63 86 
(V) 64 D5 C2 4D  A5 5E 67 F4  B1 95 5B 79  E2 9C 14 48 
(V) DD E7 8F 3B  77 7F 6E 5A  C7 B0 12 F7  34 1D C9 49 
(V) 7A 5D BD F7  2D 39 C7 6A  FE 76 EF D5  0C 45 39 3E 
(V) 07 82 23 33  05 EB 3E FD  1E 49 B3 9B  E9 0B AC 72 
(V) 6A B9 29 0B  DC 4B CA ED  30 F8 68 89  39 52 B9 2A 
(V) 7B 38 54 F1  EA A1 BD E2  50 DE F1 3C  A8 78 3E 09 
(V) 41 4C EA 99  4E 35 70 8B  F7 A3 1A 98  78 53 AA C0 
(V) 7B 35 84 E8  C3 24 7F 62  53 8E 7A 82  89 01 33 6A 
(V) D1 83 11 8D  D4 7C AB C3  98 DE B3 AF  81 B1 F3 16 
(V) 
(V) MBUS frame:
(V) 68 14 14 68  53 FF 11 01  67 33 53 DB  55 B1 C5 6D 
(V) B8 F5 B9 61  99 18 7E 72  CC 16 
(V) GCM frame:
(V) DB 08 53 41  47 59 05 E8  94 B6 81 F8  20 00 EA 75 
(V) E7 0F 80 70  DD 27 0C 07  E8 0B 0F 05  0E 2D 2D 00 
(V) FF C4 02 02  23 09 0C 07  E8 0B 0F 05  0E 2D 2D 00 
(V) FF C4 02 09  06 01 00 01  08 00 FF 06  00 48 82 9A 
(V) 02 02 0F 00  16 1E 09 06  01 00 02 08  00 FF 06 00 
(V) 4E B1 99 02  02 0F 00 16  1E 09 06 01  00 01 07 00 
(V) FF 06 00 00  00 00 02 02  0F 00 16 1B  09 06 01 00 
(V) 02 07 00 FF  06 00 00 00  09 02 02 0F  00 16 1B 09 
(V) 06 01 00 20  07 00 FF 12  09 1E 02 02  0F FF 16 23 
(V) 09 06 01 00  34 07 00 FF  12 09 15 02  02 0F FF 16 
(V) 23 09 06 01  00 48 07 00  FF 12 09 11  02 02 0F FF 
(V) 16 23 09 06  01 00 1F 07  00 FF 12 00  02 02 02 0F 
(V) FE 16 21 09  06 01 00 33  07 00 FF 12  00 30 02 02 
(V) 0F FE 16 21  09 06 01 00  47 07 00 FF  12 00 53 02 
(V) 02 0F FE 16  21 09 06 01  00 0D 07 00  FF 12 00 5E 
(V) 02 02 0F FD  16 FF 09 0C  31 37 38 32  31 30 31 32 
(V) 38 35 30 32  
(V) DLMS frame:
(V) 0F 80 70 DD  27 0C 07 E8  0B 0F 05 0E  2D 2D 00 FF 
(V) C4 02 02 23  09 0C 07 E8  0B 0F 05 0E  2D 2D 00 FF 
(V) C4 02 09 06  01 00 01 08  00 FF 06 00  48 82 9A 02 
(V) 02 0F 00 16  1E 09 06 01  00 02 08 00  FF 06 00 4E 
(V) B1 99 02 02  0F 00 16 1E  09 06 01 00  01 07 00 FF 
(V) 06 00 00 00  00 02 02 0F  00 16 1B 09  06 01 00 02 
(V) 07 00 FF 06  00 00 00 09  02 02 0F 00  16 1B 09 06 
(V) 01 00 20 07  00 FF 12 09  1E 02 02 0F  FF 16 23 09 
(V) 06 01 00 34  07 00 FF 12  09 15 02 02  0F FF 16 23 
(V) 09 06 01 00  48 07 00 FF  12 09 11 02  02 0F FF 16 
(V) 23 09 06 01  00 1F 07 00  FF 12 00 02  02 02 0F FE 
(V) 16 21 09 06  01 00 33 07  00 FF 12 00  30 02 02 0F 
(V) FE 16 21 09  06 01 00 47  07 00 FF 12  00 53 02 02 
(V) 0F FE 16 21  09 06 01 00  0D 07 00 FF  12 00 5E 02 
(V) 02 0F FD 16  FF 09 0C 31  37 38 32 31  30 31 32 38 
(V) 35 30 32 
(D) Received valid DLMS at 35
(V) Using application data:
(V) 02 23 09 0C  07 E8 0B 0F  05 0E 2D 2D  00 FF C4 02 
(V) 09 06 01 00  01 08 00 FF  06 00 48 82  9A 02 02 0F 
(V) 00 16 1E 09  06 01 00 02  08 00 FF 06  00 4E B1 99 
(V) 02 02 0F 00  16 1E 09 06  01 00 01 07  00 FF 06 00 
(V) 00 00 00 02  02 0F 00 16  1B 09 06 01  00 02 07 00 
(V) FF 06 00 00  00 09 02 02  0F 00 16 1B  09 06 01 00 
(V) 20 07 00 FF  12 09 1E 02  02 0F FF 16  23 09 06 01 
(V) 00 34 07 00  FF 12 09 15  02 02 0F FF  16 23 09 06 
(V) 01 00 48 07  00 FF 12 09  11 02 02 0F  FF 16 23 09 
(V) 06 01 00 1F  07 00 FF 12  00 02 02 02  0F FE 16 21 
(V) 09 06 01 00  33 07 00 FF  12 00 30 02  02 0F FE 16 
(V) 21 09 06 01  00 47 07 00  FF 12 00 53  02 02 0F FE 
(V) 16 21 09 06  01 00 0D 07  00 FF 12 00  5E 02 02 0F 
(V) FD 16 FF 09  0C 31 37 38  32 31 30 31  32 38 35 30 
(V) 32 
(V) DLMS
(V) (AmsDataStorage) Day data timestamp age 1731678347 - 0 > 3600
(D) Its time to update data storage
(V) (EnergyAccounting) Adding 0.0000 kWh export
(V) (EnergyAccounting)  calculating threshold, currently at 2
(V) (EnergyAccounting)  new threshold 2
(D) Serving /data.json over http...
(D) Serving /data.json over http...
ArnieO commented 1 week ago

Using this tool to decode the DLMS frame gives:

<DataNotification>
  <LongInvokeIdAndPriority Value="8070DD27" />
  <!--2024-11-15 14:45:45-->
  <DateTime Value="07E80B0F050E2D2D00FFC402" />
  <NotificationBody>
    <DataValue>
      <Structure Qty="23" >
        <!--2024-11-15 14:45:45-->
        <OctetString Value="07E80B0F050E2D2D00FFC402" />
        <!--1.0.1.8.0.255-->
        <OctetString Value="0100010800FF" />
        <UInt32 Value="0048829A" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1E" />
        </Structure>
        <!--1.0.2.8.0.255-->
        <OctetString Value="0100020800FF" />
        <UInt32 Value="004EB199" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1E" />
        </Structure>
        <!--1.0.1.7.0.255-->
        <OctetString Value="0100010700FF" />
        <UInt32 Value="00000000" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1B" />
        </Structure>
        <!--1.0.2.7.0.255-->
        <OctetString Value="0100020700FF" />
        <UInt32 Value="00000009" />
        <Structure Qty="02" >
          <Int8 Value="00" />
          <Enum Value="1B" />
        </Structure>
        <!--1.0.32.7.0.255-->
        <OctetString Value="0100200700FF" />
        <UInt16 Value="091E" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.52.7.0.255-->
        <OctetString Value="0100340700FF" />
        <UInt16 Value="0915" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.72.7.0.255-->
        <OctetString Value="0100480700FF" />
        <UInt16 Value="0911" />
        <Structure Qty="02" >
          <Int8 Value="FF" />
          <Enum Value="23" />
        </Structure>
        <!--1.0.31.7.0.255-->
        <OctetString Value="01001F0700FF" />
        <UInt16 Value="0002" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.51.7.0.255-->
        <OctetString Value="0100330700FF" />
        <UInt16 Value="0030" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.71.7.0.255-->
        <OctetString Value="0100470700FF" />
        <UInt16 Value="0053" />
        <Structure Qty="02" >
          <Int8 Value="FE" />
          <Enum Value="21" />
        </Structure>
        <!--1.0.13.7.0.255-->
        <OctetString Value="01000D0700FF" />
        <UInt16 Value="005E" />
        <Structure Qty="02" >
          <Int8 Value="FD" />
          <Enum Value="FF" />
        </Structure>
        <!--178210128502-->
        <OctetString Value="313738323130313238353032" />
      </Structure>
    </DataValue>
  </NotificationBody>
</DataNotification>

We find these fields:

OBIS code        Description
1.8.0       Positive active energy, total
2.8.0       Negative active energy, total
1.7.0       Positive active instantaneous power
2.7.0       Negative active instantaneous power
32.7.0      Instantaneous voltage in phase I
52.7.0      Instantaneous voltage in phase II
72.7.0      Instantaneous voltage in phase III
31.7.0      Instantaneous current in phase I
51.7.0      Instantaneous current in phase II
71.7.0      Instantaneous current in phase III
13.7.0      Instantaneous power factor

I see no data from the meter that could be used in the "id:" or "type:" fields in the MQTT payload. In which case the fields should be empty.

So this seems to be a bug that @gskjold will take care of when he finds time.

cmock commented 1 week ago

I notice the "SAGY..." string at the start (offset 2) of the GCM frame, whatever that may be...

ArnieO commented 1 week ago

I notice the "SAGY..." string at the start (offset 2) of the GCM frame, whatever that may be...

My guess is that it is from a random memory location due to a bug - but I'm just guessing. We'll have to wait until @gskjold finds time to look into this.