Switch to hardware serial for DSMR5 compatibility?

[daniel-jong]

First off, thanks for your work on this library 👍

Using it I was running into issues with my Landys and Gyr E360 meter. The problem I ran into was that all received telegrams would have some form of corruption (often a couple of missing characters) causing the CRC validation to fail. I think this is due to the 1 second interval of telegrams in DSMR5 just being to much.

Some background

After a lengthy investigation I can conclude that my problem is caused by the software serial simply not being able to keep up with the barrage of incoming messages.

So far I have learned:

• Disabling everything (Wifi, MQTT, etc) allows me to receive about 30% of all telegrams using SoftwareSerial. But this limits the usefulness of this library 😜
• Connecting the RTS pin to a ESP8266 IO pin and only activating the P1 meter on interval allows me to receive about 5% of all telegrams correctly. I think this helps the SoftwareSerial to keep up pace. But I am afraid that users with longer telegrams (i.e. when using a 3 phase power supply) will have a much lower success rate.
• Removing delays or watchdog calls allowed for small improvements at the cost of overall stability

This finally made me give up on SoftwareSerial for DSMR5.0.

Hardware Serial

Luckily it is has become possible to run the Esp8266's HardwareSerial with inversion. One major limitation is that inversion is only supported for Uart0, which is also used for USB serial logging (at least on my D1 mini).

I have created a fork where I got the HardwareSerial working, and all DSMR5.0 telegrams now pass CRC validation. But this is at the cost of no longer being able to read serial log data from the USB serial monitor in the Arduino IDE.

Now I have arrived at the point (not just the point of this lengthy message 😉) where I want to offer these changes back to your library as a PR. But think I first need to fix the logging.

This leads me tot he following questions:

• Are you interested in changing the library to use the Hardware serial, or would you rather I keep and maintain it myself as a fork?
• I'm thinking of changing the logging to use some form of storage and some form of network messaging (basic UDP, Telnet, Mqtt, HTTP webserver) to deal with the unavailability of the USB serial monitor. Do you have any preferences or opinions on this matter before I start work on this?

Update 2020-06-12: I have been able to get the USB serial monitoring working by doing some crazy cool esp8266 register call

[FrankZZ]

I see you've fixed the problem of reading log data while using hw serial. Awesome! Using your fork atm

[geerdkakes]

Thanks. Works great. I received the same version meter today. I have made some very little changes, mainly to be able to compile it with platform.io: https://github.com/geerdkakes/esp8266_p1meter

[geerdkakes]

@daniel-jong or @FrankZZ do you have readings for gas? It stays at 0 in my case:

sensors/power/p1meter/consumption_low_tarif 14314 sensors/power/p1meter/consumption_high_tarif 0 sensors/power/p1meter/returndelivery_low_tarif 0 sensors/power/p1meter/returndelivery_high_tarif 0 sensors/power/p1meter/actual_consumption 511 sensors/power/p1meter/actual_returndelivery 0 sensors/power/p1meter/l1_instant_power_usage 511 sensors/power/p1meter/l2_instant_power_usage 0 sensors/power/p1meter/l3_instant_power_usage 0 sensors/power/p1meter/l1_instant_power_current 2000 sensors/power/p1meter/l2_instant_power_current 0 sensors/power/p1meter/l3_instant_power_current 0 sensors/power/p1meter/l1_voltage 227000 sensors/power/p1meter/l2_voltage 0 sensors/power/p1meter/l3_voltage 0 sensors/power/p1meter/gas_meter_m3 0 sensors/power/p1meter/actual_tarif_group 1 sensors/power/p1meter/short_power_outages 15 sensors/power/p1meter/long_power_outages 2 sensors/power/p1meter/short_power_drops 5 sensors/power/p1meter/short_power_peaks 3

[FrankZZ]

Hi,

Yeah I get gas readings! Do you see the gas reading on the power meter display when you scroll through them? If not they might not be paired properly through M-bus

On Fri, Jun 19, 2020, 09:58 geerdkakes notifications@github.com wrote:

@daniel-jong https://github.com/daniel-jong or @FrankZZ https://github.com/FrankZZ do you have readings for gas? It stays at 0 in my case:

sensors/power/p1meter/consumption_low_tarif 14314 sensors/power/p1meter/consumption_high_tarif 0 sensors/power/p1meter/returndelivery_low_tarif 0 sensors/power/p1meter/returndelivery_high_tarif 0 sensors/power/p1meter/actual_consumption 511 sensors/power/p1meter/actual_returndelivery 0 sensors/power/p1meter/l1_instant_power_usage 511 sensors/power/p1meter/l2_instant_power_usage 0 sensors/power/p1meter/l3_instant_power_usage 0 sensors/power/p1meter/l1_instant_power_current 2000 sensors/power/p1meter/l2_instant_power_current 0 sensors/power/p1meter/l3_instant_power_current 0 sensors/power/p1meter/l1_voltage 227000 sensors/power/p1meter/l2_voltage 0 sensors/power/p1meter/l3_voltage 0 sensors/power/p1meter/gas_meter_m3 0 sensors/power/p1meter/actual_tarif_group 1 sensors/power/p1meter/short_power_outages 15 sensors/power/p1meter/long_power_outages 2 sensors/power/p1meter/short_power_drops 5 sensors/power/p1meter/short_power_peaks 3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fliphess/esp8266_p1meter/issues/7#issuecomment-646497597, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH5EV4CR2XKVK65GALVWNDRXMLBXANCNFSM4NGXQW3A .

[fliphess]

@daniel-jong Sorry for my late response, but I'd love to merge a PR with your changes :)

I'm still waiting for an electrician (yeah I'm sorry..... I'm that lazy).... so it's hard for me to test your PR, but there might be some other users willing to test another branch before merging it into master :)

[fliphess]

Also, you added the other phases I didn't have in my previous home and structured the mqtt topics :) That's so cool!

[fliphess]

@geerdkakes Do you perhaps have some time to create a PR of the changes you made for platformio? I'd love to merge it! 👍

If you are busy or otherwise engaged, let me know so I can merge it in myself :)

[daniel-jong]

@daniel-jong Sorry for my late response, but I'd love to merge a PR with your changes :)

I'm still waiting for an electrician (yeah I'm sorry..... I'm that lazy).... so it's hard for me to test your PR, but there might be some other users willing to test another branch before merging it into master :)

I'm more than happy to create a PR and will do so after a quick review, so give me a few days .

One thing that I might need to work on first is that the pinout has changed on my branch. I'm not sure if the existing userbase would be happy to resolder whatever they have made due to a software update 😆.

[fliphess]

@daniel-jong If you can't work around it let me know, I'm not against publishing a breaking change if it's for the better :) As long as I mention it with bold letter in the readme :)

[marcboon]

I have one question regarding the use of the D1 mini: RX of the ESP8266 is wired to the CH340 TX output, and also to the IO pin labeled RX. If you connect an external source (eg P1 output) to this RX pin, won't you get a hardware conflict (ie short) between the CH340 output and the external signal? I doubt the CH340 will be happy if it sees it's TX output pulled to GND while it's high, or is it high-Z while USB is disconnected?

[geerdkakes]

Impedance of the Rx (input) port will be very high, probably > 100kOhm so the current will not exceed 0,03mA.

[daniel-jong]

I think geerdkakes is right about how it's implemented👍

To be honest I just went for it, figuring there where no warnings in the documentation. The chips being as cheap as they are didn't help my recklessness either 😂

[marcboon]

Impedance of the Rx (input) port will be very high, probably > 100kOhm so the current will not exceed 0,03mA.

It's not about the input of the ESP, but about the outputs of the CH340 and the smart meter. When the output of the CH340 is high (idle state), the external input is driven low by the P1 port by means of the (opto-coupler) output of the smart meter (it's idle state is inverted). This will effectively short the output of the CH340 to GND, and pulling maximum current out of if. It could damage both the CH340 and the internal opto-coupler of the smart meter. The other way around (CH340 output low and P1 output high) is not a problem, because the P1 port is not driven high actively, but pulled up by a resistor, so the current is limited to 3.3V/R, which is just 330uA when R is 10k.

[marcboon]

I measured the current flowing from the ESP RX pin (actually from the CH340 TX pin, because that's where the current comes from) to the TX output on the P1 socket. In idle state (RTS low or disconnected), the open (no connection) voltage level on P1 TX is zero, and the voltage level on ESP RX is 3.3V (coming from the CH340). If you connect the two as in your suggested circuit, with a pull-up resistor to 3.3V, the voltage drops to 0.55V, and the current from RX to P1 is 15mA.

This is probably not too much for the opto-coupler in the smart meter, but it's quite high for a logic output such as the CH340. It clearly can't deliver it's high state, since it dropped from 3.3V to 0.55V, effectively shorted. I'm not confident that the CH340 will be able to sustain this shorted output, so I will modify my D1 mini board by cutting the trace from the CH340 TX pin to the RX pin of the board (which is connected to the ESP RX input) and then place a 3k3 resistor between those two pins instead. When the RX pin of the board is left unconnected, USB input will work as normal, and when USB is disconnected, the resistor will act as a pull-up for the RX input, since the output of the CH340 is at 3.3V when idle. This means that you can connect an external input (P1 in this case) without even having to add a pull-up. It's already there!

You can still enjoy logging output to USB while having the board listen to external input With OTA I don't even need the USB for firmware updates, so it's hardly an issue.

[zigurana]

@marcboon : This is a nice example of "better save than sorry", thanks for the write-up! Could you maybe add a picture of the trace you have cut, and the final board after adding the 3k3 resistor? It would help me out a lot! Thanks!

[marcboon]

Here's where I cut the trace to the CH340 TX output (connected to the RX pin of the Wemos board). I soldered a 1k resistor between the RX pin and pin 2 of the CH340. After this modification, the CH340 TX output is no longer shorted when the RX pin is driven externally, and at the same time it provides a pull-up resistor for the RX pin. I choose a 1k value as a compromise between output current (now limited to 3.3 mA when shorted) and speed. Higher values will slow the edges of the serial data too much, and might compromise code uploads (which are done at 460800 baud). With this value, the output of the CH340 is still above 3.0 V when the RX pin is shorted to GND, so that's safe.