Wired AmsToMqttBridge?

[attah]

Is your feature request related to a problem? Please describe. There are some places without wifi, but that may well have wired ethernet (e.g. basements in multi-tenant houses). And there are even some nice PoE-powered, wired, ESP32s like https://shop.m5stack.com/products/esp32-ethernet-unit-with-poe?variant=42675684966657

Describe the solution you'd like Would you be interested in enabling AmsToMqttBridge to use the built-in ethernet MAC on the ESP32? I'm currently a bit low on spare time so haven't got far, but i have bought the hardware, set up the environment and programmer, and am slowly starting to look at what it would take.

So i just wanted to gauge how, if at all, involved you'd want to be. If you are anything like me with open projects, there might be some parts and/or some prep that you'd want to get out of the way first. Or you maybe already thought of this and have some ideas/suggestions... or you might even want more involvement. (I'll gladly sponsor you for the hardware)

Or if you don't, i hope you'd at least accept pull requests for this so that it doesn't end up being a fork.

What do you say? :)

Describe alternatives you've considered One can always sprinkle wifi access points everywhere... but this is cheaper and options are always nice.

Additional context

[ArnieO]

Thank you for interesting proposition and link!

I represent amsleser.no, and we have had several requests for AMS-readers with Ethernet/PoE connection option.

For the M5stack ESP32 PoE device: If you want to use the exposed UART port for the signal from the AMS meter, you'll need to add some interface hardware towards the meter to condition the incoming signal, depending on the physical interface on the meter.

1. The simplest case would be a Kamstrup meter, if you connect to the 6-pin connector in the CCC bay; the serial signal there is from a 0/3,3V push/pull output.
2. The "Swedish/Dutch" P1 interface is a bit easier, but has an Open Collector output, which must be handled correctly, and preferably including inverting the signal.
3. More complex: Variants of M-bus-type signalling. (Norwegian "NVE standard" or European meters with higher signal voltage). The incoming signal must be scaled down from 24/12V or 32/18V (depending on interface) to 3,3/0V for a ESP32 GPIO.

Then of course the firmware needs to be modified, as you indicate - to send data over Ethernet in stead of Wifi.

[attah]

Very interesting to hear!

Are there any plans for a wired amsleser? The interfacing bits are indeed another reason for purpose-built hardware. For where i want this, we have a P1 interface, so should be doable. From what i could find online the ESP32 has a setting for inverting the UART. Or do you consider that not usable for some reason?

For my own meter, i have a "Norwegian" version, and there the POW-U was a godsend since i would not manage to power anything from that with my limited electronics experience.

Speaking of which; PoE is probably overkill since there shouldn't be reason to run Ethernet and WiFi at the same time. You are quite right in that i should look at the electrical part first. Otherwise, what will i even test with. :)

Should you consider making purpose-built hardware, consider me interested in 4 units. And in the meantime i'll keep tinkering.

[ArnieO]

Wired amsleser is not on the shortlist, but it depends on the signalled interest. 4x duly noted! 😉 For your P1 port situation, you could maybe consider sending data out via mobile phone network using an NB-IoT connected access point? Using Pow-P1 devices on the meters. Tidy install without all those cables.

You’re right, the ESPs can invert UARTs. If you look at your config/meter menu the option is already implemented.

I hav not gotten around to checking how much current ethernet driver circuitry draws, so not sure of PoE is needed. Will see when/if I start looking into it.

And keep on tinkering! 😄

[attah]

For your P1 port situation, you could maybe consider sending data out via mobile phone network using an NB-IoT connected access point? Using Pow-P1 devices on the meters. Tidy install without all those cables.

Just for completeness (it's my problem after all); That would have been a good solution, except it is in 4 different buildings, but they all just happen to have Ethernet with PoE a meter away.

[ArnieO]

That would have been a good solution, except it is in 4 different buildings, but they all just happen to have Ethernet with PoE a meter away.

Ah, OK. In that case not a good solution. I had in mind one of those technical rooms in basements of commercial buildings with many meters in the same room.

[bardahlm]

Speaking of which; PoE is probably overkill since there shouldn't be reason to run Ethernet and WiFi at the same time.

Wifi is not the only reason for PoE. Another use-case is surveilance cameras. There are also cheap power injectors available.

I'd prefer PoE over a separate power supply.

Do ethernet use more power than wifi?

[ArnieO]

Do ethernet use more power than wifi?

I don’t know yet! 😄

[mrkaasa]

Slightly offtopic: I had the same problem, no wifi (due to metal enclosure outside) or power where the AMS-reader is.

I pulled an ethernet cable and extended the AMS signal to a place where I have both good wifi signal and USB power. I power the POW-U (from ArnieO) with USB, while it also has good wifi signal.

The lenght of the cable was around 30 meter.

In my other residence I am using a POW-K OG. With similar issue for the wifi. So I am using a TP-Link powerline with built in wifi hotspot for the POW-K to have sufficient signal.

Of course PoE would be a cleaner solution, but I like having the firmwares and everything released and compatible with my hardware.

[ArnieO]

The ability to handle long cables is one of the few positive things I can say about the standard chosen for the Norwegian AMS meters: High voltage levels (24/12V) and moderate bit rate (2400 baud).

The standard adopted by Sweden is a more modern standard. Data delivered from an Open Collector output, nominally powered at 5V and at 115 200 baud bit rate. Much easier to interface, but don't expect it to handle more than a few meters of cable.

[ThorAlex87]

I'll add my vote for a wired version! I'm moving to a new house soon and expect to have to buy a second AP to get good enough coverage for the reader. Or run an ethernet wire from the meter and put the reader in my livingroom...

[PTRSC]

I would also like to be able to run it on an ESP with Ethernet and PoE, for example Olimex ESP32 Power-over-Ethernet ISO

[GEAG]

It is quite feasible to run an ethernet socket together with an ESP32, with the energy solely supplied trough a P1-DSMR version 5 port. P1-DSMR ports must be able to supply up to 1.25 W of power, according to its public specification. There is no need for POE. We have built a bunch of such devices running already for several months with no major problems. We currently use a different firmware than AMSreader. You might want to look at the Wt32-ETH01 board (about 15 EUR). This still needs the additional P1's adaption to serial circuit and maybe an USB port for easy flashing and debugging.

[ArnieO]

It is quite feasible to run an ethernet socket together with an ESP32, with the energy solely supplied trough a P1-DSMR version 5 port. P1-DSMR ports must be able to supply up to 1.25 W of power, according to its public specification. There is no need for POE. We have built a bunch of such devices running already for several months with no major problems. We currently use a different firmware than AMSreader. You might want to look at the Wt32-ETH01 board (about 15 EUR). This still needs the additional P1's adaption to serial circuit and maybe an USB port for easy flashing and debugging.

Thank you for sharing your knowledge!

The P1-DSMR port is clearly the less challenging regarding ethernet, as there is 1.25W power available. (Our Pow-P1 board is designed for P1-DSMR port; Wifi only.)

The challenge is our best selling boards:

- Pow-U (for meters with M-bus interface) There is no clear specification for this standard, the implementations vary between meter manufacturers, even the M-bus voltage level varies. And available power is generally not documented by the manufacturers. For the meters used in Norway we have: . Aidon: 700 mW . Kaifa: 500 mW . Landis & Gyr: We have not found documentation on available power. M-bus meters used in Europe (Sagemcom etc): We have not found documentation on available power.

- Pow-K (a plug-in board for Kamstrup Omnipower meters) Available power around 300 mW.

So here you see the reasons we have been thinking PoE, as there in general is significantly less power available on those devices compared to the 1.25 W available on a P1-DSMR port.

Also complicating this a bit is the fact that the current consumption of such a device is varying a lot depending on what the device is doing. While transmitting on Wifi, the ESP32 current draw is substantially higher than what is allowed as maximum in the P1-DSMR specification. Some meter designs are more tolerant than what is said in the specification, which means that you are not necessarily compliant to the spec even if things do work correctly. We have seen this on one of the newest meter models (Kamstrup Omnia E-meter): It is less tolerant to short current peaks, so we recently found that our Pow-P1 design needs a modification to be able to run on that meter.

We do not know yet how peak current draw is on an ethernet system, and whether this could be a problem; just pointing out that this is an issue to take into account in addition to overall average consumption.

EMC We are in the process of CE-marking our products. As part of that process, the designs will be submitted to an EMC test. In order to pass such an EMC test, an Ethernet socket will have to be sufficiently protected regarding EMC issues. The potential of a (very) long ethernet cable being connected makes the test regime more severe compared to our current design, where there is only a short cable between the reader and the power meter.

The Wt32-ETH01 from Seeed Studio does indeed look very interesting, and could be an interesting starting point for a design for P1-DSMR - however it seems to be inadequately documented. And it is does not seem to be CE marked, so we cannot expect it to hold up in an EMC test. (I also note that they do not seem to use original Espressif ESP32 module; making it even more difficult.) If you find more documentation on this board, please let me know!

Was it this board you have used for your implementations?

[gskjold]

I have made this work with LilyGO T-Internet-POE. Does anyone out there have any other boards available for testing?

[attah]

I still have those M5stack ones with PoE that i linked in the first post. (But i haven't really got any further than setting up the build environment - other projects taking my limited time).

[gskjold]

No guarantee here, but I have added a hardware profile for M5 PoESP32 and corresponding ethernet config. Hopefully it works, and if you want to try it, here is the firmware:

esp32.zip

[attah]

Excellent! Thank you! I just need to A) find them again, and B) do the hardware side of things (perhaps it was just a pull-up, need to re-research that).

Edit: found. And after turning down the baudrate to 38400, i finally managed to flash one. I really need to build a programming clip, so that i don't have to pretend to be one... this was miserable. Unfortunately it does not seem to boot as far as initializing the ethernet side of things; i.e. i get no lights in at least 10x the time of the stock firmware. The console to check (if it even says anything of value?) is the one i programmed over presumably?

(And if this discussion should move somewhere, do tell me)

[gskjold]

Sorry, I should have been more clear on how I (for now) have implemented this. You will still need to connect to the AMS2MQTT access point to set up the device. You will first get the hardware menu where you get to choose the hardware profile. Choose M5 PoESP32 and continue. You will then get the option to activate ethernet

[attah]

Very nice! That seems to have worked - it shows up with hostname "tasmota" though. Now i know what i'm doing this weekend :) Will it want/need the data line just pulled up, or done properly with inverting it? (Settings containing pullup and inv setting suggests neither would be needed?)

[attah]

Success! While i did mess with the settings, ~this should be back what it started with after selecting the correct config~. (UART2, pullup, inv) Edit: UART2(GPIO16) i apparently selected during setup - default was UART0. I did end up needing an external pull-up though (10k to be at the safer end of normal values).

[Wupsman]

Awesome what you share with us here 🍾🎉👍

I tried it right away and it works perfectly.

Only the description settings for Austria didn't work for me.

Cheers

[ArnieO]

Only the description settings for Austria didn't work for me.

Exactly what is it that does not work with this setting?

[Wupsman]

I have a Smartmeter Sagemcom T-210-D from the grid operator EVN, Baud fits with 2400 but Party I had to set 8E1 and like the Kaifa MA309M I only needed the Encrypton key (as in the picture above) and also the format is MBUS.

[ArnieO]

I have a Smartmeter Sagemcom T-210-D from the grid operator EVN, Baud fits with 2400 but Party I had to set 8E1 and like the Kaifa MA309M I only needed the Encrypton key (as in the picture above) and also the format is MBUS.

The "Austria" settings are only for reading the electricity prices. Data format is linked to the type of meter, disregarding which country you are in. 2400 baud / 8N1 is normal for meters with M-bus output.

So you are using this with an Mbus-to-ethernet adapter?

[Wupsman]

I can hardly believe what you have created with the AMS Reader, thank you so much.

I had tried to read the smart meter with the Python script from greenMikeEU, but it didn't work for me, it kept stopping.

Then I tried an Olimex PoE + Farley TTL to MBUS with the currently released amsreader firmware and only the 8E1 setting worked.

I then came across "Wired AmsToMqttBridge" here and tried the firmware posted by gskjold and this version is simply awesome on the M5 PoESP32 + Farley TTL to MBUS!

I also tried the 8N1 setting today and it works fine too, sorry for my misreport.

What I would still like to know is, where can I find the firmware code with the options for darkmode and eth?

cherrs

[gskjold]

Excellent, thank you for the positive feedback!

edit: No source code for this is available at the moment. Will get back to this later

[ArnieO]

@Wupsman : Thank you for your kind words!

You are maybe aware of this, but in case someone else from Austria finds this thread:

Our Pow-U device with an RJ-12 cable plugged into its R-J45 connector directly reads the Austrian meters (also used in some other mid-European countries) that have M-bus signal on pins 3 and 4 of their RJ12 connector.

(And I just realized we have not yet updated the description in the webshop to provide that information regarding ... so I have to fix that!)

But of cause - that will be a wifi connection from the meter - and not cabled Ethernet, which is what some of you following this thread are looking for.

[egiljae]

I got a LilyGo T-Internet-POE board working over WiFi, but wired would be preferred. Will the M5 PoESP32 firmware posted here also work with the LilyGo board?

[gskjold]

Yes, there is a profile for that board already, I have the same board.

[markuspeter]

Hi,

I did end up needing an external pull-up though (10k to be at the safer end of normal values).

Is that necessary to get it working ? Where exactly would I put that pull-up ?

I finally received a m5stack poesp32 and Farley TTL to mbus converter yesterday, managed to load the firmware from this thread and start up Asmreader, However, I get "No HAN data received for at least 30s" no matter what configuration I choose.

What options did you set under "Hardware"? I'm sure about UART2, but what about the other fields and checkboxes ("pullup" for example) ?

I do get ~29V on the MBus dropping regularly to around 19/20V, therefore I guess the L&G 450 does output data. I connected the MBUS to the converter, and the converter to the Poesp32, using Rx (Rx pin on converter & poesp32 white cable).

For a brief moment, probably when auto-detect was initially activated, I've been getting a different message ("Unknown data received, check meter config") - but the UI doesn't let me put auto-detect back.

I was hoping to get some data back, since you @attah and @Wupsman were able to make it work with the same components.

Btw - awesome project, I'm blown away, even though I didn't get everything working yet!

[attah]

Yes needing means it was necessary to have it. Pulling up implies the data line up to something of higher voltage; i.e. +5V in this case. But i have the Swedish standard with TTL signals already available, though "open collector", i.e. needing to be pulled up.

If you are running a MBUS converter it should probably not need that (and whether you do would depend on the specific converter, and not be a general truth).

My post seems to say i have uart2, pullup and inv compared to with whatever is default. Fairly sure pullup does nothing, and i'm not sure inv does either. Inv(ert) sounds like something you shouldn't need with a MBUS converter.

If you, as you say, only have connected one cable - that's your first problem. You also need ground, or there will be no way for the esp to see the signal. I would have expected that it is the TX on the converter that should be connected to the RX of the PoESP. But it definitely looks like the PoESP is mislabeled (aka labeled from the point of view of what you connect to it and not itself). Yellow is its data-in. No idea how i figured that out previously, must have looked up the respective GPIO functions in my sleep or something.

If that still doesn't work; have you checked the baud rate? MBUS should be 2400 but the looks of it.

@gskjold Is this mainlined yet BTW?

[markuspeter]

Thanks for pointing out the RX/TX issue - now that you said it, it seems obvious that it's the yellow cable (there is a label "G16" on the other side). Of course I also connected Gnd and 5V between the Poesp32 and the converter.

Unfortunately, this still doesn't work, I still get no data.

I use the same converter as @Wupsman, the Harley TTL to MBUS converter. RX seems to be the correct pin, it's identical to the picture of above post (and in this post using the same module). Nevertheless, I tried the other pin as well, without any luck.

I tried 2400/8N1 as well as other settings. I assume I should get something even if the data were encrypted (which I don't know yet). Telnet debugging does not show any incoming data neither ("no HAN data received last 30sm single blink").

Running out of ideas what to try next, maybe I've got a faulty converter.

[gskjold]

@gskjold Is this mainlined yet BTW?

Not yet, working on some changes I want to include in 2.3 before releasing it for full testing.

[gskjold]

Added profile for WT32-ETH01, based on schematics from @gellfuen in discussion #673. Untested, hope it works. Remember you need to preconfigure the board via WiFi AP before hooking it up to ethernet. esp32.zip

[gellfuen]

Many THX, module should arrive tomorrow, will start testing tomorrow. THX for the WiFi AP info!

[egiljae]

I just received a LilyGO T-ETH-Lite board, https://www.lilygo.cc/products/t-eth-lite. It uses an W5500 ethernet controller, and comes in a small package, even with the POE shield attached. Is it possible to add support for this board?

[gellfuen]

Hello, (discussions #673) got the WT32-ETH01 today and was able to flash the firmware from esp32.zip from yesterday with serial connection. Test output/steps:

1. Disconnected power, removed flash jumper and reconnected power.
2. Connected to the WiFi AP.
3. Selected profile for WT32-ETH01 and configured AP at first boot.
4. I let the RS232 converter connected and saw at serial terminal the IP the WT32 got from DHCP.
5. Next step was to switch at configuration the connection from WiFi to Ethernet.
6. Device performed a reboot. -- At this point, the serial terminal output stopped! -- So I looked at DHCP for a new MAC address and found a device called tasmota :-) - I do not have any tasmota devices. -- I'm able to connect over that new IP to the WT32 again, but I often lost the connection, what I mean here is that I lost pings. -- If I try to change from DHCP to fixed IP, that change will not happen, but the configured values are stored on the webpage and I can still see my entered values even after a reboot. -- if I check settings at status the network section is empty and the MAC addresses at device information are not the same I can see on my DHCP. Tried to add a picture of that:

Do you need something from my devices?

BR

[gskjold]

Thanks, I see there are some things I forgot to implement for wired, including network status and fixed IP :) Will have a look at it when I have available time

[markuspeter]

Running out of ideas what to try next, maybe I've got a faulty converter.

I finally received a new converter, and it is working now. Looks like I wrecked the first one by inverting the pins to the poesp32, or it was bad from the start.

This works great with the L&G E450 in Switzerland (provider Romande Energie) with above settings (2400baud 8N1), no key needed. I get readings every 5 seconds using MQTT with the HomeAssistant payload config, which works fine, except for some reason I get 1 reading per minute which is all empty/zero.

[GEAG]

This works great with the L&G E450 in Switzerland (provider Romande Energie) with above settings (2400baud 8N1), no key needed.

Swiss L&G E450 from RE by default use 8E1, not 8N1.

except for some reason I get 1 reading per minute which is all empty/zero.

L&G E450 push different sets of data values (I call it a telegram) at different intervals. The 1/minute telegram does not contain the respective power value, but other data values like energy counters. AmsReader sets a 0, if the value (assumed to be always delivered) is not present in the telegram. That's a known bug.

Most other SM do not issue different telegrams at different intervals. Because of the low throughput at 2400 baud 8E1 and a low S/N ratio of the DLMS data encoding, the data is partioned into different telegrams.

Some other SM use higher baud rates (up to 115200), 8N1 asynch coding (saves 10% bandwith) and may not repeat the constant COSEM meta-data in every telegram, which dosen't add anything useful but clogs up a very large part of the channel's capacity.

[markuspeter]

Swiss L&G E450 from RE by default use 8E1, not 8N1.

Just changed the configuration to 8E1, and this works as well. Not sure if there is a difference, from a user point of view I can't see any - is there ?

except for some reason I get 1 reading per minute which is all empty/zero.

L&G E450 push different sets of data values (I call it a telegram) at different intervals. The 1/minute telegram does not contain the respective power value, but other data values like energy counters. AmsReader sets a 0, if the value (assumed to be always delivered) is not present in the telegram. That's a known bug.

Good to know, because I was looking through the open issues and didn't see any related to that. Thank you for the explanation. I guess I'll filter that on the Home Assistant side of things for now.

[gskjold]

AmsReader sets a 0, if the value (assumed to be always delivered) is not present in the telegram. That's a known bug.

I thought I had that covered by now :thinking: I should have a look at that again

[gellfuen]

Thanks, I see there are some things I forgot to implement for wired, including network status and fixed IP :) Will have a look at it when I have available time

Hello, no problem! I have time, I can wait until it's finished. BR

[gellfuen]

Hello just kindly asking for update about the missing stuff for wired profile for WT32-ETH01? BR

[gskjold]

Here is an update, fixing hostname and fixed IP, as well as IP display on status page: esp32.zip edit: Found a fault in the firmware above, use this: esp32.zip

[gskjold]

Btw, I may have changed some things, so back up config, erase flash before flashing new firmware and set it up from scratch, then load config

[gskjold]

I just received a LilyGO T-ETH-Lite board, https://www.lilygo.cc/products/t-eth-lite. It uses an W5500 ethernet controller, and comes in a small package, even with the POE shield attached. Is it possible to add support for this board?

@egiljae I have one of these actually. Have you ever made a arduino sketch for it? Looks like the ethernet implementation is slightly different

[egiljae]

@egiljae I have one of these actually. Have you ever made a arduino sketch for it? Looks like the ethernet implementation is slightly different

I haven't had a chance to code a arduino sketch yet, but I successfully set it up in ESPHome with the following config for ethernet:

ethernet:
  type: W5500
  clk_pin: GPIO10
  mosi_pin: GPIO12
  miso_pin: GPIO11
  cs_pin: GPIO9
  interrupt_pin: GPIO13
  reset_pin: GPIO14
  clock_speed: 30

Edit: there seems to be a bunch of arduino example sketches here: https://github.com/Xinyuan-LilyGO/LilyGO-T-ETH-Series/tree/master/examples

[gellfuen]

Hi, THX for update just saw your info now - I will test esp32.zip tomorrow! BR

[gellfuen]

Hi, uploaded software from esp32.zip(10th December) but sorry no success: ESP (WT32) is only cycling:

I can see that text at terminal: ... rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:2 load:0x3fff0030,len:184 load:0x40078000,len:12732 ho 0 tail 12 room 4 load:0x40080400,len:2908 entry 0x400805c4 (I) AMS bridge started (I) Voltage: 0.00V (D) ESP32 LittleFS (D) size: 1441792 (I) No configuration, booting AP (I) Entering setup mode ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:2 load:0x3fff0030,len:184 load:0x40078000,len:12732 ho 0 tail 12 room 4 load:0x40080400,len:2908 entry 0x400805c4 (I) AMS bridge started (I) Voltage: 0.00V (D) ESP32 LittleFS (D) size: 1441792 (I) No configuration, booting AP (I) Entering setup mode ets Jun 8 2016 00:22:57 .... At first I erased the flash like you wrote and loaded your files. >> ESP is only cycling... Then I saw, that in the zip file the file "bootloader_dio_40m.bin" was not in place like at the older zip file, so I copied that and flashed again, but the behavior at the ESP is the same :-(.