OBIS reference filter help

[Proxy5]

hello,

it is possible to filter this out of the data sent by the meter 0-0: 96.13.0.255, it is not relevant data and its processing is problematic, because the service provider, 0xff, fills in the location of the message.

Thanks, Tomas

[lowdef]

Hi Tomas, can you send an example message?

[lowdef]

In principle you can delete the line, before you offer it to the parser. Make sure you switch of CRC check in this case.

[Proxy5]

Hi,

yes of course

dsmr_console --host 192.168.1.71 --port 23 --version 5 --verbose

@.***:/# dsmr_console --host 192.168.1.71 --port 23 --version 5 --verbose DEBUG:asyncio:Using selector: EpollSelector DEBUG:dsmr_parser.clients.protocol:connected DEBUG:dsmr_parser.clients.protocol:received data: /AUX59940192257

0-0:1.0.0(220212153250W) 0-0:42.0.0(AUX1020340192257) 0-0:96.1.0(9940192257) 0-0:96.14.0(0002) 0-0:96.50.68(ON) 0-0:17.0.0(24.000kW) 1-0:1.8.0(003970.901kWh) 1-0:1.8.1(001827.309kWh) 1-0:1.8.2(002143.592kWh) 1-0:1.8.3(000000.000kWh) 1-0:1.8.4(000000.000kWh) 1-0:2.8.0(000000.000kWh) 1-0:2.8.1(000000.000kWh) 1-0:2.8.2(000000.000kWh) 1-0:2.8.3(000000.000kWh) 1-0:2.8.4(000000.000kWh) 1-0:3.8.0(000081.870kvarh) 1-0:4.8.0(000353.778kvarh) 1-0:5.8.0(000081.870kvarh) 1-0:6.8.0(000000.000kvarh) 1-0:7.8.0(000000.000kvarh) 1-0:8.8.0(000353.778kvarh) 1-0:15.8.0(003970.901kWh) 1-0:32.7.0(232.0V) 1-0:31.7.0(002A) 1-0:33.7.0(0.848) 1-0:14.7.0(49.99Hz) 1-0:1.7.0(00.572kW) 1-0:2.7.0(00.000kW) 1-0:5.7.0(00.000kvar) 1-0:6.7.0(00.000kvar) 1-0:7.7.0(00.000kvar) 1-0:8.7.0(00.239kvar) 0-0:98.1.0(220201000000W)(003574.321kWh)(001624.224kWh)(001950.097kWh)(000000.000kWh)(000000.000kWh)(000000.000kWh)(000072.292kva rh)(000317.886kvarh)(000072.292kvarh)(000000.000kvarh)(000000.000kvarh)(000317.886kvarh)(003574.321kWh)(05.248kW)(05.248kW)(04.2 28kW)(00.000kW)(00.000kW)(00.000kW) 0-0:96.13.0(\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff DEBUG:dsmr_parser.clients.protocol:received data: \xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x ff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff!3CE6

ERROR:asyncio:Exception in callback _SelectorSocketTransport._read_ready() handle: <Handle _SelectorSocketTransport._read_ready()> Traceback (most recent call last):         File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run                 self._callback(*self._args)         File "/usr/lib/python3.6/asyncio/selector_events.py", line 721, in _read_ready                 self._protocol.data_received(data)         File "/usr/local/lib/python3.6/dist-packages/dsmr_parser/clients/protocol.py", line 124, in data_received                 telegram = telegram.encode("latin1").decode("ascii") UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1177: ordinal not in range(128) ^CINFO:dsmr_parser.clients.protocol:disconnected because of close/abort.

Tamás On 2022. Feb 12. 12:34 +0100, lowdef @.***>, wrote:

Hi Tomas, can you send an example message? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[lowdef]

In principle recent versions of the parser can handle this, the offending line is just ignored:

P1_MESSAGE_TIMESTAMP:    2022-02-12T15:32:50+01:00  [None]
ELECTRICITY_IMPORTED_TOTAL:      3970.901   [kWh]
ELECTRICITY_USED_TARIFF_1:   1827.309   [kWh]
ELECTRICITY_USED_TARIFF_2:   2143.592   [kWh]
ELECTRICITY_DELIVERED_TARIFF_1:      0.000  [kWh]
ELECTRICITY_DELIVERED_TARIFF_2:      0.000  [kWh]
ELECTRICITY_ACTIVE_TARIFF:   0002   [None]
CURRENT_ELECTRICITY_USAGE:   0.572  [kW]
CURRENT_ELECTRICITY_DELIVERY:    0.000  [kW]
INSTANTANEOUS_VOLTAGE_L1:    232.0  [V]
INSTANTANEOUS_CURRENT_L1:    2  [A]
EQUIPMENT_IDENTIFIER_GAS:    9940192257 [None]

ignore line with signature \d-\d:96\.13\.0.+?\r\n, because parsing failed.
Traceback (most recent call last):
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 59, in parse
    telegram[signature] = parser.parse(match.group(0))
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 201, in parse
    return CosemObject(self._parse(line))
  File "/home/hanserik/projects/dsmr_parser/dsmr_parser/parsers.py", line 151, in _parse
    raise ParseError("Invalid '%s' line for '%s'", line, self)
dsmr_parser.exceptions.ParseError: ("Invalid '%s' line for '%s'", '0-0:96.13.0(\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\\xff\r\n', <dsmr_parser.parsers.CosemParser object at 0x7ff488d029a0>)

[apulai]

Hello,

I also have the same problem. 0-0: 96.13.0.255 is filled with 0xff. Provider's manual says it is reserved for further use and is filled with '????'s. In reality it is 0xff.

I understand that it is ignored when parsing as lowdef wrote.

However I would like to use it from Home Assisstant.

This is the log I see in Home Assisstant:

File "/usr/local/lib/python3.12/site-packages/dsmr_parser/clients/protocol.py", line 132, in data_received telegram = telegram.encode("latin1").decode("ascii") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 1371: ordinal not in range(128)

The dsmr_parser/clients/protocol.py Tries to convert it to an ASCII string and rightfully fails, since 0xff cannot be converted to ASCII.

So I am proposing to change this code below. I think a try block would help:

        for telegram in self.telegram_buffer.get_all():
            # ensure actual telegram is ascii (7-bit) only (ISO 646:1991 IRV required in section 5.5 of IEC 62056-21)

             try:
                telegram = telegram.encode("latin1").decode("ascii")
            except UnicodeDecodeError as e:
                pass
                or maybe ???:
                telegram = telegram.encode("latin1").decode("latin1")

            self.handle_telegram(telegram)

Please comment on it. Is there an elegant way to get around this issue? Thanks, Andras

[apulai]

Hello,

Actually after modifiying /usr/local/lib/python3.12/site-packages/dsmr_parser/clients/protocol.py around line 132 in Homassistant (inside the container) from the original to

        for telegram in self.telegram_buffer.get_all():
            # ensure actual telegram is ascii (7-bit) only (ISO 646:1991 IRV required in section 5.5 of IEC 62056-21)

             try:
                telegram = telegram.encode("latin1").decode("ascii")
            except:
                telegram = telegram.encode("latin1").decode("latin1")

            self.handle_telegram(telegram)

It works even with the non-comformant telegram (0xff bytes under last part of the message).

Dear @ndokter, please advise or comment.

Thanks, Andras

[ndokter]

I don't feel like i'm knowledged enough to comment on it. I'd think that maybe just processing it as latin1 might be better then? I think it will be fine, but am unable to predict side effects.