[P1] warning: Ignoring invalid telegram

napthemax commented 3 years ago

Have been struggling a bit with my brand new Sagemcom T211 (with the P1 interface activated). Don't know if it's the meter that is not supported or anything else (I'm not a programmer).

lsusb reports: Bus 001 Device 002: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC Set to default /dev/ttyUSB0 on my Pi W.

So it seems to start fine, but the only thing that comes out of the Raspberry is "Ignoring invalid telegram". Tried 2.2, tried different timeout. At 9 it stops after 9 seconds stating "no data", at 10 it just continues delivering "invalid data". Any ideas to proceed?

Log as follows:

25/03/2021, 10:13:37 Initializing HAP-NodeJS v0.9.3... 25/03/2021, 10:13:43 Loaded plugin 'homebridge-p1' 25/03/2021, 10:13:43 [3/25/2021, 3:13:43 AM] Registering platform 'homebridge-p1.Lib' 25/03/2021, 10:13:43 [3/25/2021, 3:13:43 AM] Registering platform 'homebridge-p1.P1' 25/03/2021, 10:13:44 Loading 1 platforms... 25/03/2021, 10:13:44 [P1] Initializing P1 platform... 25/03/2021, 10:13:44 [P1] homebridge-p1 v1.2.0, node v14.16.0, homebridge v1.3.3, homebridge-lib v5.1.0 25/03/2021, 10:13:44 [P1] warning: config.json: plugin_map: invalid key 25/03/2021, 10:13:44 Preparing Advertiser for 'HOOBS 9F12' using bonjour-hap backend! 25/03/2021, 10:13:44 Starting to advertise 'HOOBS 9F12' using bonjour-hap backend! 25/03/2021, 10:13:44 Bridge is running on port 51826. 25/03/2021, 10:13:46 [P1] warning: heartbeat 1, drift 261 25/03/2021, 10:13:47 [P1] warning: heartbeat 2, drift 701 25/03/2021, 10:13:47 [P1] connected to /dev/ttyUSB0 25/03/2021, 10:13:53 [P1] warning: ignoring invalid telegram ... and so on ...

ebaauw commented 3 years ago

So e ideas:

What DSMR version does your meter have?
Are you running on a Pi Zero W? Not sure if that’s powerful enough.
Do you have a P1 cable with micro-USB plug, or do you use a micro-USB to USB-A female conversion cable?
Did you get the P1 cable to work with another program or with cu (see link in the README)?
Run Homebridge in debug mode to see whether Homebridge P1 receives any partial data. Not sure how to do that on hoobs.

napthemax commented 3 years ago

DSMR 5.0.2
Running four plugins on a different one without problem, and on this only P1, so I doubt it
Female conversion cable between the P1/USB cable and the Pi
One caveat of the W (or maybe it's HOOBS) is that cu is not available in the cli (must admit my Dutch is non-existent, but searched the page ;-) )
Debug mode below

Might need to get that Pi 3/4 instead. Tested with the W that I already had, hoping for the best ... Meter sends in "Mode D" (whatever that means). Format defined as “8N1”: - 1 start bit - 8 data bits - No parity bit - 1 stop bit · 115200 baud

hhoobs@hoobs:~ $ hoobs -d Error: listen EADDRINUSE: address already in use :::8080 at Server.setupListenHandle [as _listen2] (net.js:1318:16) at listenInCluster (net.js:1366:12) at Server.listen (net.js:1452:7) at Function.serverListen [as listen] (/usr/local/lib/node_modules/@hoobs/hoobs/node_modules/express-ws/lib/index.js:42:40) at API.start (/usr/local/lib/node_modules/@hoobs/hoobs/server/api.js:191:17) at module.exports (/usr/local/lib/node_modules/@hoobs/hoobs/server/cli.js:172:21) { code: 'EADDRINUSE', errno: -98, syscall: 'listen', address: '::', port: 8080 } /usr/local/lib/node_modules/@hoobs/hoobs/bin/hoobs bridge -u /home/hoobs/.hoobs/etc -p /home/hoobs/.hoobs/node_modules -r -d [HOOBS] Stopping server.

ebaauw commented 3 years ago

You cannot start a second instance of Homebridge. Either set debug mode in the Homebridge UI, or stop the service before starting Homebridge manually.

napthemax commented 3 years ago

Of course. Stupid me. Blame brainfreeze after ice cream ...

hoobs@hoobs:~ $ hoobs -d HOOBS listening on port 8080. /usr/local/lib/node_modules/@hoobs/hoobs/bin/hoobs bridge -u /home/hoobs/.hoobs/etc -p /home/hoobs/.hoobs/node_modules -r -d "GET" /log "GET" /lib/codemirror.js "GET" /lib/chart.js "GET" /lib/javascript.js "GET" /lib/lint.js "GET" /lib/json-lint.js "GET" /api/config "GET" /api/config "GET" /api/config "GET" /api/auth/validate "GET" /api/status "GET" /css/config~layout~login~profile~users.2bfa9c25.css "GET" /js/config~layout~login~profile~users.976ab349.js "GET" /css/config~login~profile~users.3904ba73.css "GET" /js/config~login~profile~users.8ff83a6c.js "GET" /js/login.a2fd37d9.js "GET" /css/login.018db737.css "GET" /api/config "GET" /api/auth "GET" /img/snapshot.80428b94.jpg Initializing HAP-NodeJS v0.9.3... Loaded plugin 'homebridge-p1' [3/25/2021, 9:55:43 AM] Registering platform 'homebridge-p1.Lib' [3/25/2021, 9:55:43 AM] Registering platform 'homebridge-p1.P1' Loading 1 platforms... [P1] Initializing P1 platform... [P1] homebridge-p1 v1.2.0, node v14.16.0, homebridge v1.3.3, homebridge-lib v5.1.0 [P1] config.json: {"platform":"P1","plugin_map":{"plugin_name":"homebridge-p1"},"name":"P1","timeout":10,"dsmr22":false,"serialPort":"/dev/ttyUSB0"} [P1] warning: config.json: plugin_map: invalid key [P1] config: {"dsmr22":false,"port":8088,"timeout":10,"platform":"P1","name":"P1","serialPort":"/dev/ttyUSB0"} [P1] starting heartbeat for ["P1Platform"] Preparing Advertiser for 'HOOBS 9F12' using bonjour-hap backend! api_launched Starting to advertise 'HOOBS 9F12' using bonjour-hap backend! Bridge is running on port 51826. Setup URI: X-HM://0023ISYX07PJA Bridge started: Thu Mar 25 2021 09:55:44 GMT-0600 (Mountain Daylight Time) "GET" /fonts/material.0509ab09.woff2 "GET" /img/snapshot.80428b94.jpg [P1] npm registry: request 1: GET /homebridge-p1/latest [P1] connected to /dev/ttyUSB0 [P1] npm registry: request 1: 200 OK [P1] latest version: homebridge-p1 v1.2.0 [P1] warning: ignoring invalid telegram [P1] warning: ignoring invalid telegram [P1] warning: ignoring invalid telegram [P1] warning: ignoring invalid telegram

napthemax commented 3 years ago

If you use Google Translate, some info on the Swedish meters here: https://hanporten.se/svenska/protokollet/

ebaauw commented 3 years ago

Damn, it doesn't log the invalid telegram (sometimes, I lose track of what I programmed ;-)

Didn't know the use a similar scheme in SE; just learned that BE uses a similar scheme as well. From the description, Homebridge P1 should recognise the telegram; the invalid telegram means Homebridge P1 doesn't find the header (/ XXXZ Ident CR LF CR LF or footer (! CRC CR LF).

I would really be helpful if you could connect the P1 cable to a system supporting cu, or a similar terminal emulation programme, and capture a telegram. You should be able to install it on a Pi Zero by sudo apt install cu (at least under Buster).

The cu command is:

$ cu -l /dev/ttyUSB0 -s 115200 --parity=none -E q

It should log the telegrams on the screen. Type ~ to quit cu.

ebaauw commented 3 years ago

No day vs night tariff in SE? Not sure if it makes sense to expose reactive energy and power?

napthemax commented 3 years ago

No day vs night tariff in SE? Not sure if it makes sense to expose reactive energy and power?

No, but I have hourly tariff. And thanks for the cu help. Just gives me strange things:

hoobs@hoobs:~ $ cu -l /dev/ttyUSB0 -s 115200 --parity=none -E q Connected. h]?V??3Y3,6VRkek% -ek% ##/ )Q/-ek% /#+)Q/-ek%

/#+)=ek%

                       ##/#+)=ek%
                                 ##/#+)Q-ek%
                                            /#+)Q-ek%
                                                     ##/#=k%
                                                            ##/#+)=k%
                                                                     /#+)Q-ek%
                                                                              ##/#+)Q-ek%
                                                                                         ##/#+)Q-ek%
                                                                                                    /#+)Q-ek%
                                                                                                             /#+)Q-ek%
                                                                                                                      /#+)Q-ek%
                                                                                                                               /#=k%
                                                                                                                                    ##/#+)=k%
                                                                                                                                             ##/#+)=k%
                                                                                                                                                      /#+)=k%
                                                                                                                                                             ##/#+)=k%

+)=k% ##/# /-ek% /S-ek% /-ek%

/#+}-ek%

                           ##/}-ek%
                                   ##/}-ek={{wekh]?V??Vv,6V*kek%

-ek% ##/

/Q/-ek%

         /#+)Q/-ek%

... and so on ...

ebaauw commented 3 years ago

Ok, that’s probably an issue with the serial parameters, baudrate, data bits, parity, etc. Have you tried the dsmr22 setting with Homebridge P1?

thomas-svrts commented 3 years ago

Sure that it is DSMR? It mentions on your referenced website that the DLMS protocol is used…

Also, here I don’t find it: https://www.buildup.eu/sites/default/files/content/mj0220176enn.en_.pdf

@ebaauw Luxembourg is also using DSMR (version4) 🙂

napthemax commented 3 years ago

Sure that it is DSMR? It mentions on your referenced website that the DLMS protocol is used…

hmmm. Other user on Swedish forum said DSMR 5.0.2, but looking at the standard here makes me confused too… https://en.m.wikipedia.org/wiki/IEC_62056

napthemax commented 3 years ago

Ok, that’s probably an issue with the serial parameters, baudrate, data bits, parity, etc. Have you tried the dsmr22 setting with Homebridge P1?

Yep. Didn’t see anything when I did (nothing in the normal log). But I have not tried with cu on dsmr22. Will try in an hour – out driving for a while. BTW the communication is not encrypted according to Swedish standard. Wrote this earlier too: Format defined as “8N1”: - 1 start bit - 8 data bits - No parity bit - 1 stop bit · 115200 baud

ebaauw commented 3 years ago

Sure that it is DSMR? It mentions on your referenced website that the DLMS protocol is used…

DSMR = Dutch Smart Meter Requirements, as defined by the association of network operators here in NL. It is mentioned in the doc you linked:

In the Netherlands, a national companion specification has been defined and is supported by several manufacturers (DSMR). In that context, the P1 port is covering the functional specification of interfaces H1 H2 and H3. The Dutch Smart Meter Architecture defines ports (P1 to P4) as a mean on which communication takes place between two instances. • P1 is used for connecting the smart meter to third party hard-/software. • P2 is used to connect to a gas- or water meter. • P3 connects (most commonly via GPRS) with the DSO. • P4 is on the DSO's site and allows suppliers and/or third parties to connect to and to gather data from a customer. Serial protocol (115 kbaud) is the chosen technology for the interfaces. G1 relies on GSM and PLC. DSOs have defined the standards for H2 and were free to choose G1.

The doc mentions that Flandres (BE) and LU are using the P1 port from DSMR as well.

I'm unfamiliar with DLMS and cannot find it on any of the web sites I link to. It is mentioned on the Wiki you linked. Looks like an older standard to read the meters electronically by the network operator, but still needing physical access to the meter (i.e. house calls).

Homebridge P1 uses 8N1 at 115200 baud by default, as defined by DSMR. However the first version (DSMR 2.2) uses 7E1 at 9600 baud. This is also mentioned in the SE doc:

Man läser av data via ett seriellt gränssnitt. Till skillnad på IEC62056-21 så skall teckenformatet vara 8N1 (8 databits, ingen paritet, en stop bit). Standard IEC62056-21 är 7E1

But I have not tried with cu on dsmr22.

dsmr22 is a config.json setting for Homebridge P1, to use 7E1 at 9600 baud. Not sure how to set the data bits to 7 on cu.

Other user on Swedish forum said DSMR 5.0.2

Afaik that's the most recent version, marked "final". You might try a lower baud rate first. Did you try cu on the Pi Zero, or on a Pi 3 or 4?

napthemax commented 3 years ago

Read up a little on DLMS. It is supposed to send plain ASCII over the port (according to the Swedish page I linked above), so that should have been seen by cu in a good manner even if Homebridge-P1 can't read it, am I right? Checked to see if there are any different commands or variables for cu to work with, but can't figure it out.

dsmr22 is a config.json setting for Homebridge P1, to use 7E1 at 9600 baud. Not sure how to set the data bits to 7 on cu.

Other user on Swedish forum said DSMR 5.0.2

Afaik that's the most recent version, marked "final". You might try a lower baud rate first. Did you try cu on the Pi Zero, or on a Pi 3 or 4?

Yeah, but he could very well have been missing the difference between DSMR and DLMS. Seems like every company implements their own standard here ... DOH! Ran cu on the Pi Zero, works well.

It's a very new meter in Sweden (installed by Ellevio) and the Sagemcom-meter sends a CRC16-checksum with every telegram. Maybe I just have to give up if you can't make DLMS work and wait for someone to figure it out.

napthemax commented 3 years ago

There seems to be an open source project on the DLMS though (but only for Mode C, not Mode D): https://github.com/pwitab/iec62056-21

ebaauw commented 3 years ago

The CRC16 checksum is part of the DSMR standard. Try specifying different baudrates to cu. If that doesn’t result in anything legible, try cu on a more powerful server.

napthemax commented 3 years ago

The CRC16 checksum is part of the DSMR standard. Try specifying different baudrates to cu. If that doesn’t result in anything legible, try cu on a more powerful server.

Tried them all, from 7200, 9600 and up to 230400. 9600 gives same crap and cu does not support many baud rates. The meter supports 115200 officially. Running it does not affect cpu on the Pi (~ 9% with or without running cu). Grateful for your support, but it seems like a dead end because of the meter using DLMS not DSMR. I will try to install Homebridge on a different machine and run P1 & cu from there later today (if I can find the time for it). Otherwise it will have to wait for next week. It is strange that the data coming out of cu is scrambled though – it should be plain ASCII.

Thousands of these meters are now being installed in Sweden, so hopefully you will find a way to support also DLMS. I know many will pop into this project in search for some cool home automation projects (and that are at the same skill level as me - not knowing how to electronically build your own cable and set things up manually).

ebaauw commented 3 years ago

hopefully you will find a way to support also DLMS

I will not be supporting DLMS (if that's indeed what your meter is using for reporting). That's a different protocol, best supported by a different plugin.

some info on the Swedish meters here: https://hanporten.se/svenska/protokollet/

I'm happy to support this protocol, irrespective of how they name it. It looks like DSMR to me, and I can probably support it with minor changes, as I've done for the BE protocol.

However, you need to get proper data from the meter first. I'm no export on serial port communications, but it would seem the Pi and the meter aren't communicating correctly. In my limited experience, that's most likely related to the serial port communication settings. Maybe it could also be caused by faulty cabling, but I doubt this, since the Pi is receiving something.

Tried them all, from 7200, 9600 and up to 230400.

57600? 28800? 14400?

napthemax commented 3 years ago

hopefully you will find a way to support also DLMS

I will not be supporting DLMS (if that's indeed what your meter is using for reporting). That's a different protocol, best supported by a different plugin.

Completely understand. Really disturbing that every grid company seem to invent their own protocol in Sweden.

some info on the Swedish meters here: https://hanporten.se/svenska/protokollet/

I'm happy to support this protocol, irrespective of how they name it. It looks like DSMR to me, and I can probably support it with minor changes, as I've done for the BE protocol.

That is really good news. I will really try to figure out why my Pi receives only garbage. Probably need to experiment with the other settings in cu to get something else out. Have even ordered a Pi4 only to explore if it could be the power of the Pi W that messes things up, but it will take a week or so to receive it and to set it up.

57600? 28800? 14400?

57600 gives garbage 28800 unsupported baud rate 14400 unsupported baud rate 7200 unsupported baud rate 3600 unsupported baud rate 1800 connects but gives garbage ...

I remember my first modem. So tested 300 baud. It connects. But garbage in return.

ebaauw commented 3 years ago

Really disturbing that every grid company seem to invent their own protocol in Sweden.

From what I read, DLMS is really on older generation protocol, from the time the grid companies still send people on house calls to take the meter readings.

Probably need to experiment with the other settings in cu to get something else out.

cu seems broken alright. You might want to try https://www.npmjs.com/package/@serialport/terminal, but it might take a long time to install on a Pi Zero.

Have even ordered a Pi4 only to explore if it could be the power of the Pi W that messes things up

Indeed, I wouldn't completely rule out that it's caused by the limited power on the Pi Zero.

napthemax commented 3 years ago

cu seems broken alright. You might want to try https://www.npmjs.com/package/@serialport/terminal, but it might take a long time to install on a Pi Zero.

hoobs@hoobs:~ $ serialport-terminal -l /dev/ttyAMA0
/dev/ttyUSB0 usb-FTDI_FT232R_USB_UART_AQ59FFX5-if00-port0 FTDI hoobs@hoobs:~ $

So I tested this: hoobs@hoobs:~ $ serialport-terminal -p /dev/ttyUSB0 -b 115200 --parity=none

... but only garbage, unfortunately (also the parity flag odd/even) ... Any other setting to try?

napthemax commented 3 years ago

Nothing happens on this setting. Just opens the port and nothing more.

hoobs@hoobs:~ $ serialport-terminal -p /dev/ttyUSB0 -b 9200 --parity=none Opening serial port: /dev/ttyUSB0 echo: true

ebaauw commented 3 years ago

I would try the data bits (7, 8), stop bits (0, 1, 2), and party settings (none, even, odd), initially at 115200 baud.

napthemax commented 3 years ago

I would try the data bits (7, 8), stop bits (0, 1, 2), and party settings (none, even, odd), initially at 115200 baud.

115200 baud, same result on none/even/odd: 8:2 garbage 8:1 garbage 8:0 error (stopbits is invalid) 7:2 garbage 7:1 garbage 7:0 error (stopbits is invalid)

9600, 28800 and 57600 baud, same result on none/even/odd: 8:2 nothing happens on 9600, garbage on 28800 & 57600 8:1 nothing happens on 9600, garbage on 28800 & 57600 8:0 error (stopbits is invalid) 7:2 nothing happens on 9600, garbage on 28800 & 57600 7:1 nothing happens on 9600, garbage on 28800 & 57600 7:0 error (stopbits is invalid)

You may Google-translate this thread. I don't understand much of the tech they discuss there – even though it's in Swedish. :-)

Reply #45 in this thread describes what Ellevio sends out on the port. I will call Ellevio on Monday and ask them to "restart" the P1 port. It's activated, but I wanted to test and deactivate and activate again, but that didn't work (it's something I can do on Ellevio home page).

But this is what one user get from the same meter (using MQTT???) that others have integrated into Home-Assistant (see the thread above).

12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"/ELL5\253833635_A\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1019120127W)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"varh)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"0000.167kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"61.7.0(0002.226kW)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:23000kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"3.7.0(0000.000kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:63.7.0(0000.167kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:24.7.0(0000.107kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:44.7.0(0000.061kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:64.7.0(0000.000kvar)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:32.7.0(231.0V)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:52.7.0(230.2V)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:72.7.0(230.5V)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:31.7.0(001.5A)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:51.7.0(001.3A)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"1-0:71.7.0(009.6*A)\r"} 12:01:16 MQT: tele/powermeter/RESULT = {"SerialReceived":"!7AA3\r"}

Anyway. It's Saturday evening, and I have some champagne to open. I'll dig into this more tomorrow.

napthemax commented 3 years ago

I have now tested with a Pi3 and unfortunately I get the same result. It seems like the port of the new meter sends out something the P1-cable and the Pi can't read. I guess you may close this request for now – I don't think we will come any further without knowing why serialport-terminal does not report anything useful. Could be a faulty cable, but feels far fetched. Something is coming through, it's just that it's garbage ...

I know thousands of homes will have this meter soon, so hopefully someone more skilled than me can start to hack whatever is coming out of it and provide you with some useful data.

ebaauw commented 3 years ago

OK closing for now. Please re-open (or open a new issue) if you have more info.

napthemax commented 3 years ago

Bringing in some more info on the Swedish meter, if you have the possibility to give it another try:

https://grodansparadis.com/wordpress/?p=5039

https://github.com/psvanstrom/esphome-p1reader

ebaauw commented 3 years ago

The info behind those links is very promising. I might have to make some minor changes (I don't think I've seen kvar as a unit before), but that would be doable. If memory serves, the issue was that you didn't get any legible data from the P1 port. Once you do, I'll be more than happy to pick this up.

napthemax commented 3 years ago

Yes, it’s garbage coming out as you can see earlier in the thread. But I read the first link that it might be a hardware thing too (I’m neither knowledgeable in HW or SW though):

The serial format is unencrypted with speed and format 115200, N, 8.1. Just standard. However, it is sent inverted and with TTL levels so you have to take care of that.

I guess I’ll have to wait for more info (or that someone builds me a different cable if it’s HW …).

ebaauw / homebridge-p1