search

ndokter / dsmr_parser

Library to parse Dutch Smart Meter Requirements (DSMR) telegrams.
MIT License
110 stars 64 forks source link

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 140: ordinal not in range(128) #80

Open lakovacs78 opened 3 years ago

lakovacs78 commented 3 years ago

Hi @ndokter and @lowdef

I'm from Hungary and my power meter: Sanxing SX631 (S34U18)

I'm trying to use DSMR slimme meter integration in HA but I received an error message:

Traceback (most recent call last): File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 862, in _read_ready__data_received self._protocol.data_received(data) File "/usr/local/lib/python3.8/site-packages/dsmr_parser/clients/protocol.py", line 91, in data_received data = data.decode('ascii') UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 140: ordinal not in range(128)

Here is an example from my telegram data:

/AUX59902730044

0-0:1.0.0(210511102330S) 0-0:42.0.0(AUX1030302730044) 0-0:96.1.0(9902730044) 0-0:96.14.0(0001) 0-0:96.50.68(ON) 0-0:17.0.0(90.000kW) 1-0:1.8.0(000101.821kWh) 1-0:1.8.1(000040.496kWh) 1-0:1.8.2(000061.325kWh) 1-0:1.8.3(000000.000kWh) 1-0:1.8.4(000000.000kWh) 1-0:2.8.0(000653.426kWh) 1-0:2.8.1(000398.181kWh) 1-0:2.8.2(000255.245kWh) 1-0:2.8.3(000000.000kWh) 1-0:2.8.4(000000.000kWh) 1-0:3.8.0(000002.626kvarh) 1-0:4.8.0(000143.011kvarh) 1-0:5.8.0(000002.247kvarh) 1-0:6.8.0(000000.379kvarh) 1-0:7.8.0(000075.324kvarh) 1-0:8.8.0(000067.687kvarh) 1-0:15.8.0(000755.250kWh) 1-0:32.7.0(237.2V) 1-0:52.7.0(235.5V) 1-0:72.7.0(237.1V) 1-0:31.7.0(008A) 1-0:51.7.0(008A) 1-0:71.7.0(008A) 1-0:13.7.0(0.996) 1-0:33.7.0(0.996) 1-0:53.7.0(0.995) 1-0:73.7.0(0.997) 1-0:14.7.0(50.00Hz) 1-0:1.7.0(00.000kW) 1-0:2.7.0(05.868kW) 1-0:5.7.0(00.000kvar) 1-0:6.7.0(00.000kvar) 1-0:7.7.0(00.473kvar) 1-0:8.7.0(00.000kvar) 0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW) 0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????) !6D64

Could you help me pls?

Thank you in advance!

ndokter commented 3 years ago

The field with the question marks seems to be called 'Text message max 1024 characters.'. The parser currently assumes all byte data can be parsed as ASCII but it's not clear to me currently why that would fail

lowdef commented 3 years ago

hi @lakovacs78,

What makes you think this should be parsable with DSMR parser? Nearly none of the lines parses with any of the DSMR Specification variants. What specification is your meter claiming to follow?

In principle I think the telegrams could be made to parse, but that requires adding a new specification type and maybe some new value types.

With following code you can investigate it yourself:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
issue78.py
@author: lowdef
"""

from dsmr_parser import telegram_specifications
from dsmr_parser.objects import Telegram
from dsmr_parser.parsers import TelegramParser
from dsmr_parser.parsers import MBusParser
from dsmr_parser.parsers import ValueParser
from dsmr_parser import obis_references
from dsmr_parser.value_types import timestamp
from decimal import Decimal

telegram_issue  = r"""/AUX59902730044

0-0:1.0.0(210511102330S)
0-0:42.0.0(AUX1030302730044)
0-0:96.1.0(9902730044)
0-0:96.14.0(0001)
0-0:96.50.68(ON)
0-0:17.0.0(90.000kW)
1-0:1.8.0(000101.821kWh)
1-0:1.8.1(000040.496kWh)
1-0:1.8.2(000061.325kWh)
1-0:1.8.3(000000.000kWh)
1-0:1.8.4(000000.000kWh)
1-0:2.8.0(000653.426kWh)
1-0:2.8.1(000398.181kWh)
1-0:2.8.2(000255.245kWh)
1-0:2.8.3(000000.000kWh)
1-0:2.8.4(000000.000kWh)
1-0:3.8.0(000002.626kvarh)
1-0:4.8.0(000143.011kvarh)
1-0:5.8.0(000002.247kvarh)
1-0:6.8.0(000000.379kvarh)
1-0:7.8.0(000075.324kvarh)
1-0:8.8.0(000067.687kvarh)
1-0:15.8.0(000755.250kWh)
1-0:32.7.0(237.2V)
1-0:52.7.0(235.5V)
1-0:72.7.0(237.1V)
1-0:31.7.0(008A)
1-0:51.7.0(008A)
1-0:71.7.0(008A)
1-0:13.7.0(0.996)
1-0:33.7.0(0.996)
1-0:53.7.0(0.995)
1-0:73.7.0(0.997)
1-0:14.7.0(50.00Hz)
1-0:1.7.0(00.000kW)
1-0:2.7.0(05.868kW)
1-0:5.7.0(00.000kvar)
1-0:6.7.0(00.000kvar)
1-0:7.7.0(00.473kvar)
1-0:8.7.0(00.000kvar)
0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)
0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)
!6D64
"""

sample = telegram_issue.replace('\n', '\r\n')

telegram_specification = telegram_specifications.V5
telegram_specification['checksum_support']=False

parser = TelegramParser(telegram_specification)
telegram = Telegram(sample, parser, telegram_specification)

print(telegram)
lakovacs78 commented 3 years ago

hi @lakovacs78,

What makes you think this should be parsable with DSMR parser?

Nearly none of the lines parses with any of the DSMR Specification variants.

What specification is your meter claiming to follow?

In principle I think the telegrams could be made to parse, but that requires adding a new specification type and maybe some new value types.

With following code you can investigate it yourself:


#!/usr/bin/env python3

# -*- coding: utf-8 -*-

"""

issue78.py

@author: lowdef

"""

from dsmr_parser import telegram_specifications

from dsmr_parser.objects import Telegram

from dsmr_parser.parsers import TelegramParser

from dsmr_parser.parsers import MBusParser

from dsmr_parser.parsers import ValueParser

from dsmr_parser import obis_references

from dsmr_parser.value_types import timestamp

from decimal import Decimal

telegram_issue  = r"""/AUX59902730044

0-0:1.0.0(210511102330S)

0-0:42.0.0(AUX1030302730044)

0-0:96.1.0(9902730044)

0-0:96.14.0(0001)

0-0:96.50.68(ON)

0-0:17.0.0(90.000kW)

1-0:1.8.0(000101.821kWh)

1-0:1.8.1(000040.496kWh)

1-0:1.8.2(000061.325kWh)

1-0:1.8.3(000000.000kWh)

1-0:1.8.4(000000.000kWh)

1-0:2.8.0(000653.426kWh)

1-0:2.8.1(000398.181kWh)

1-0:2.8.2(000255.245kWh)

1-0:2.8.3(000000.000kWh)

1-0:2.8.4(000000.000kWh)

1-0:3.8.0(000002.626kvarh)

1-0:4.8.0(000143.011kvarh)

1-0:5.8.0(000002.247kvarh)

1-0:6.8.0(000000.379kvarh)

1-0:7.8.0(000075.324kvarh)

1-0:8.8.0(000067.687kvarh)

1-0:15.8.0(000755.250kWh)

1-0:32.7.0(237.2V)

1-0:52.7.0(235.5V)

1-0:72.7.0(237.1V)

1-0:31.7.0(008A)

1-0:51.7.0(008A)

1-0:71.7.0(008A)

1-0:13.7.0(0.996)

1-0:33.7.0(0.996)

1-0:53.7.0(0.995)

1-0:73.7.0(0.997)

1-0:14.7.0(50.00Hz)

1-0:1.7.0(00.000kW)

1-0:2.7.0(05.868kW)

1-0:5.7.0(00.000kvar)

1-0:6.7.0(00.000kvar)

1-0:7.7.0(00.473kvar)

1-0:8.7.0(00.000kvar)

0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)

0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)

!6D64

"""

sample = telegram_issue.replace('\n', '\r\n')

telegram_specification = telegram_specifications.V5

telegram_specification['checksum_support']=False

parser = TelegramParser(telegram_specification)

telegram = Telegram(sample, parser, telegram_specification)

print(telegram)

@lowdef Thank you for your help!

I tried to parse my "const" telegram data based on your example script (sorry I'm not a Python expert). Now I have a theory what is wrong in my telegram but I need to connect somehow to my DSMR reader in a Python script and then parse my data with dsmr parser. Could you help me in this script (how I should set the serial connection) pls?

lowdef commented 3 years ago

I do not know the reader you are using and how to use it, I am using a P1 Cable directly connected to a Raspberry Pi. Then you can obtain Telegrams directlöy with the DSMR Serial Reader. Alternatively you can use a terminal program and read from the serial connection directly.

On Mon, May 17, 2021 at 9:58 AM lakovacs78 @.***> wrote:

hi @lakovacs78 https://github.com/lakovacs78,

What makes you think this should be parsable with DSMR parser?

Nearly none of the lines parses with any of the DSMR Specification variants.

What specification is your meter claiming to follow?

In principle I think the telegrams could be made to parse, but that requires adding a new specification type and maybe some new value types.

With following code you can investigate it yourself:

!/usr/bin/env python3

-- coding: utf-8 --

@.***: lowdef"""

from dsmr_parser import telegram_specifications from dsmr_parser.objects import Telegram from dsmr_parser.parsers import TelegramParser from dsmr_parser.parsers import MBusParser from dsmr_parser.parsers import ValueParser from dsmr_parser import obis_references from dsmr_parser.value_types import timestamp from decimal import Decimal

telegram_issue = r"""/AUX599027300440-0:1.0.0(210511102330S)0-0:42.0.0(AUX1030302730044)0-0:96.1.0(9902730044)0-0:96.14.0(0001)0-0:96.50.68(ON)0-0:17.0.0(90.000kW)1-0:1.8.0(000101.821kWh)1-0:1.8.1(000040.496kWh)1-0:1.8.2(000061.325kWh)1-0:1.8.3(000000.000kWh)1-0:1.8.4(000000.000kWh)1-0:2.8.0(000653.426kWh)1-0:2.8.1(000398.181kWh)1-0:2.8.2(000255.245kWh)1-0:2.8.3(000000.000kWh)1-0:2.8.4(000000.000kWh)1-0:3.8.0(000002.626kvarh)1-0:4.8.0(000143.011kvarh)1-0:5.8.0(000002.247kvarh)1-0:6.8.0(000000.379kvarh)1-0:7.8.0(000075.324kvarh)1-0:8.8.0(000067.687kvarh)1-0:15.8.0(000755.250kWh)1-0:32.7.0(237.2V)1-0:52.7.0(235.5V)1-0:72.7.0(237.1V)1-0:31.7.0(008A)1-0:51.7.0(008A)1-0:71.7.0(008A)1-0:13.7.0(0.996)1-0:33.7.0(0.996)1-0:53.7.0(0.995)1-0:73.7.0(0.997)1-0:14.7.0(50.00Hz)1-0:1.7.0(00.000kW)1-0:2.7.0(05.868kW)1-0:5.7.0(00.000kvar)1-0:6.7.0(00.000kvar)1-0:7.7.0(00.473kvar)1-0:8.7.0(00.000kvar)0-0:98.1.0(210501000000S)(000041.191kWh)(000015.425kWh)(000025.766kWh)(000274.705kWh)(000189.336kWh)(000085.369kWh)(000001.034kvarh)(000059.658kvarh)(000000.866kvarh)(000000.168kvarh)(000031.678kvarh)(000027.980kvarh)(000315.899kWh)(03.644kW)(03.188kW)(03.644kW)(06.872kW)(06.872kW)(06.544kW)0-0:96.13.0(????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????)!6D64"""

sample = telegram_issue.replace('\n', '\r\n')

telegram_specification = telegram_specifications.V5 telegram_specification['checksum_support']=False

parser = TelegramParser(telegram_specification) telegram = Telegram(sample, parser, telegram_specification)

print(telegram)

@lowdef https://github.com/lowdef Thank you for your help!

I tried to parse my "const" telegram data based on your example script (sorry I'm not an Python expert). Now I have a theory what is wrong in my telegram but I need to connect somehow to my DSMR reader https://www.zuidwijk.com/product/slimme-lezer-smart-reader/ in a Python script and then parse my data with dsmr parser. Could you help me in this script pls?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ndokter/dsmr_parser/issues/80#issuecomment-842103328, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCJDWEWTZMWOVTBF46DWCTTODEBFANCNFSM44W55PNA .

Zolli commented 2 years ago

Hey, @lowdef i encountered the same problem using this specific meter, also in Hungary, if i'm able to help somehow please let me know, to receive the data i use a SlimmeLezer falsed with esp-link so i able to acces the data using telnet

Bubi504 commented 2 years ago

@Zolli jutottál ezzel valamire ?

Zolli commented 2 years ago

@Bubi504 Sure, but not with esphome sadly. I ended up with the DSMRReader project. This project discards unicode characters, but obviously makes the CRC check fail, but fortunately you can disable CRC checks. I configured it as a DSMR5 compatible meter. It works like a charm

Bubi504 commented 2 years ago

@Zolli Now it's working without DSMRReader, the solution is here : https://github.com/esphome/issues/issues/2393#issuecomment-960644346

Aeroid commented 2 years ago

I had the same issue with my tcp-bridge. Adding errors="ignore" to decode("ascii") helped see #92

p0macs commented 5 months ago

Hi! This is an old issue, but I have a similar error now, using homeassistant (Core: 2024.1.6) and trying to add the DSMR Slimme meter integration. After filling in the configuration (using network with port 23) the integration cannot be added. Looking into the logs, I can see error messages like this: File "/usr/local/lib/python3.11/site-packages/dsmr_parser/clients/protocol.py", line 132, in data_received telegram = telegram.encode("latin1").decode("ascii") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1371: ordinal not in range(128) I have a hungarian EON meter too. I can parse the telegram using DSMR Reader without issues. I guess the problem is very similar to the original issue, at the last telegram message: 0-0:96.13.0(ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ) This version of homeassistant should use the 1.3.1 version of dsmr-parser. Do you have an idea how to add this integration?

p0macs commented 5 months ago

The problem with the above telegram message is, that the 0xFF characters are not a trailing character in the message, but they are part of the last 0-0:96.13.0 OBIS ID code. This is some kind of placeholder for a missing text message. One problem is, that the telegram.encode("latin1").decode("ascii") will not accept the 0xFF codes. The other problem is, that parser regexp will accept only letters and number and some special characters like .*(). One possible workaround could be to simply remove this placeholder from the telegram message - after the checksum calculation, but before parsing the lines - using something like replace(chr(0xff),'')