Brultech Greeneye Energy Monitor

[adamalli7]

The problem

Integration works fine, but we need to add the new Energy Integration piece. The energy configuration in hass states to notify developer: https://developers.home-assistant.io/docs/core/entity/sensor#long-term-statistics

What is version of Home Assistant Core has the issue?

core-2021.8.8

What was the last working version of Home Assistant Core?

core-2021.8.8

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Greeneye

Link to integration documentation on our website

https://www.home-assistant.io/integrations/greeneye_monitor/

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

Hey there @jkeljo, mind taking a look at this issue as it has been labeled with an integration (greeneye_monitor) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

---

greeneye_monitor documentation greeneye_monitor source _{^{(message by IssueLinks)}}

[jcam]

This would be great! It would take care of the "That’s a lot of data, and the databases used by the recorder integration for history don’t do well with that much data" problem, and make using this integration much more plug and go! @emontnemery looks like you've been a frequent contributor, are you a code owner now as well?

[ludeeus]

@jcam please do not ping members of our organization. Thanks :+1:

[jcam]

Oops, I didn't realize that wasn't allowed here

[jkeljo]

I finally upgraded my local installation, and I can hopefully play with this soon.

[jkeljo]

Okay, I played around with this a bit. Unfortunately, it looks like it is more involved than just adding some device class/state class metadata to the SensorEntitys. :( The energy integration requires entities with DEVICE_CLASS_ENERGY, and all of the entities that this integration creates for the CTs would be DEVICE_CLASS_POWER. (They also have energy information, but it's stuffed into "extra state" where it's only accessible to automations.)

So, I think what needs to happen is I need to pull that energy info out of "extra state" and into new entities. As the component is currently designed that would require YAML changes, which are no longer allowed, so I'll need to move this integration to use the UI config flow instead. (I've been meaning to do this at some point anyway, so it's nice to have some additional motivation.)

I started digging in to the UI config flow stuff, and I think I see how to do it. I'll update this issue as things progress.

[adamalli7]

It would be nice if the revamped version of this could show solar, grid and house usage to match up with the new energy dashboard. To do this with the greeneye some math will need to be done since most typical installation will have CTs on the main (which can have power flowing either direction) and another set of CTs on the solar lines (this line can also have power flowing either direction. My inverter uses about 2 watts when no solar power is being produced).

Not sure if you have a greeneye yourself, but I use BTMON.py to get my data into influxDB and then use grafana to do the math above and display some charts. I wonder if BTMON.py would be of any help to you or at least some of the code. I believe that codes collects two readings from the greeneye and then does some math to output W, Wh, dWh (differential watt hours), and amps.

[jkeljo]

Yep, I do have a GEM myself, as well as solar panels. I will definitely make sure that there's a way to hook that up properly. I did look at BTMON while writing the GEM library that Home Assistant uses, which does the math to convert from the Watt-seconds reported by the GEM to Watts.

I'm not sure yet what the cleanest way to connect it all to the house/grid/solar stuff is. My setup has CTs on the main lines in the panel and on the solar lines, but the way mine is wired only the solar CTs can have power flowing both ways and I need to do math to figure out the grid usage. Sounds like your setup is measuring grid and solar and needs to do math to figure out the house usage. I suspect this can all be done with some kind of derived sensor in Home Assistant.

[jkeljo]

As far as I can tell, by the way, enabling the entities for long-term statistics doesn't stop recorder from wanting to log every single state change, so I think the recommendation to keep these out of recorder will likely remain.

[sdwilsh]

I'm not sure yet what the cleanest way to connect it all to the house/grid/solar stuff is.

I've been grappling with this as I'm trying to build a Brultech to MQTT gateway. I think the easiest way is to get the data into HA, and then use a template sensor to do the math.

As far as I can tell, by the way, enabling the entities for long-term statistics doesn't stop recorder from wanting to log every single state change, so I think the recommendation to keep these out of recorder will likely remain.

One other thing that might be worth doing is either a) adding built in rate-limiting, or b) user-configurable rate-limiting with a safe default for most users. While that won't give as-real time of information for power and current, it would let folks get setup right away with safe defaults to get the built-in integration. On the MQTT side of things, I figured I was going to rate-limit before sending to the broker so I don't spam home assistant with too many updates as well.

[jkeljo]

Yeah, template sensors were what I had in mind. Too bad they're in YAML, I was kinda looking forward to being able to configure this without a restart.

If the integration is becoming configurable in the UI (aside from users adding template sensors when they're needed), I think having rate-limiting with safe defaults makes a lot of sense. For myself, I'd probably want rate-limiting per sensor type, letting the power/current/temperature/pulse counter sensors be chatty (and ignored by recorder) and throttling the energy ones down to a rate that's more manageable by SQLite/MariaDB. Really seems like something that HomeAssistant should provide rather than having to do it per-integration, though. :(

[sdwilsh]

...or c) user-configuration that asks you about your setup and makes the math sensors automatically. We could programmatically create the SensorTemplate and avoid YAML all together. The user-config flow would get complicated (thankfully not too hard to write tests for), but it would make this component much easier to use. Basically, we'd want to ask what channel(s) are from solar (and maybe battery?), and what channel(s) is your main. Since the library that is in use already flags things for net-metered or not, we should be able to use that information to figure out how things flow.

Hypothetically, for @jkeljo's setup, assuming channel 1 is solar and net-metered, channel 2 is main and not net-metered, we don't actually need to create a template sensor at all, since the energy monitoring wants total power consumption and what solar is producing (at least, that's my read).

Whereas for @adamalli7's setup (which sounds like mine as well), assuming channel 1 is solar and net-metered, channel 2 is main and net-metered, we'd make a template sensor for the total energy consumption which would basically be something like (but in template language that we can figure out later):

if solar.polarized > 0:
  total_consumption = main.polarized + solar.polarized
else:
  # If solar is negative, that means it is consuming more than it is producing,
  # and therefore should already be counted in the main
  total_consumption = main.polarized

This gets a lot trickier with wrap-around numbers between the multiple sensors though, alas.

[sdwilsh]

To further elaborate, we should be able to do this:

from homeassistant.components.template.sensor import SensorTemplate

# In setup, when creating our entities
SensorTemplate(...)

The worst part will be constructing the template programmatically. We could also just program a sensor in python that does this too, which, in hindsight, might be easier :D

[sdwilsh]

Hypothetically, for @jkeljo's setup, assuming channel 1 is solar and net-metered, channel 2 is main and not net-metered, we don't actually need to create a template sensor at all, since the energy monitoring wants total power consumption and what solar is producing (at least, that's my read).

I was wrong about this, since the inverter will consume some meter power off-hours too, so we'd have to do some math.

But, given the new sensor state class total, maybe this is something that's worth engaging the core folks on since I imagine other implementations (such as zwave CTs) would suffer from the same problems.

[jkeljo]

Quick update: I've been working to bring greeneye_monitor up to modern Home Assistant component standards (adding typing (#58571), tests (#58661) and config flows (future PRs since the code depends on the tests one and the HA folks request that we not put up PRs that depend on one another). Once those make their way through the backlog, I'll get to adding statistics/energy integration support. There will likely be some breaking changes, but on the other side things should be much nicer.

I've been working with @sdwilsh on updates to the Python library that this integration uses, and a side effect of that is that the config flow experience will end up being much nicer than the current YAML config is. It will be able to auto-detect net metering channels and temperature units, and possibly also do most of the configuration of the GEM itself.

[peteraclausen]

Wow, this is very exiting news! I have 3 GEMs, 2 are connected via STS cables to a Dashbox and the 3rd has an Ethernet module, which in turn sends packets to the Dashbox.

I just recently started messing with HA and plan to retire the Dashbox (other than forwarding packets to HA from the 2 direct connected GEMs).

I also have solar (a 50 kWh install), and have a total of 8 split-200 CTs:

2 on 400A main feeder from meter base 2 on 400A feeder to 2x200A main house panels 2 on 400A feeder to shop building 200A sub-panel 2 on 200A feeder from 200A solar sub-panel

So in my case, I don't need to do any "calculations" since I have dedicated CTs for all my main panels.

Can't wait to start testing this!

[emontnemery]

@jkeljo Can this issue be closed since #58764 has been merged, or are more changes needed?

[jcam]

Config flows and statistics/energy integration support were both mentioned a couple weeks back, are those both now complete?

[emontnemery]

Config flows are not done, but this issue mentions lack of energy dashboard support which was fixed by #58764 as I understand it

[sdwilsh]

One thing I have learned, @peteraclausen, is that the Ethernet module doesn't seem to be able to send the same packets that the WiFi/Ethernet module can. Consequentially, I've been working on https://github.com/sdwilsh/brultech-serial2mqtt to get that data into Home Assistant with a Raspberry Pi attached to the serial port on the device.

[chrismaki]

My GEM showed up yesterday and I've connected 9 channels via 11 CTs from my load centre so far. While I'm not a programmer, I am technically minded and would love to see this fully integrated into HA so if you need any feedback or tests performed, please let me know.

[peteraclausen]

@sdwilsh My setup with 3 GEMs has been running nice and stable for a few weeks now since Ben @ Brultech added the ability to run all 3 of my GEMs with a packet format of Bin-48-NET-Time, including the 2 that are connected directly to my DashBox, which is then forwarding the packets to my HASS.

@jkeljo Are there any plans to also support the Brultech ECM-1240 (basically a 7-channel version of the GEM) device with a direct attached EtherPort gateway? I have 2 of those as well for monitoring my smaller sub-panels.

I'm also running a Sense monitor and was impressed with how easy it was to setup in the new Energy Integration. Having that type of integration for the GEM is going to be awesome!

Any idea of when we might see this in the beta track? Would be an awesome Christmas surprise! ;)

[jkeljo]

@jkeljo Are there any plans to also support the Brultech ECM-1240 (basically a 7-channel version of the GEM) device with a direct attached EtherPort gateway? I have 2 of those as well for monitoring my smaller sub-panels.

No active plans, and neither Shawn nor I have one to test with. The first step would be to add support for the packet format in siobrultech-protocols. We could do that from the spec, but having an actual packet along with an idea of what values it's reporting would be better.

I'm also running a Sense monitor and was impressed with how easy it was to setup in the new Energy Integration. Having that type of integration for the GEM is going to be awesome!

Any idea of when we might see this in the beta track? Would be an awesome Christmas surprise! ;)

Christmas is looking unlikely; the PRs are moving pretty slowly right now. :(

[sdwilsh]

@jkeljo, maybe we could throw together a HACS integration that has all the changes you want to beta test it with interested folks, and then back-port them into core?

[jkeljo]

@jkeljo, maybe we could throw together a HACS integration that has all the changes you want to beta test it with interested folks, and then back-port them into core?

Yeah that is looking more and more attractive, though if iteration in HACS is way faster I suspect it would be a bell we can't un-ring. (Getting the changes back into core would be harder than just putting them there in the first place.)

[sdwilsh]

Iteration speed in HACS is as fast as you want to go. We can un-ring the bell, but we'd probably want to wait until it is fairly stable again.

[peteraclausen]

@jkeljo I'd be happy to ship you a spare ECM-1240 along with an EtherPort gateway and Brultech AC adapters for both. Are you in the US?

[scyto]

@jkeljo, maybe we could throw together a HACS integration that has all the changes you want to beta test it with interested folks, and then back-port them into core?

i would be willing to test this, i actually only got data flowing into the production integration today so don't mind loosing data i see folks making assumptions solar is on channel 1 i dont use solar so my total is on channel 1 (aka the main feed CTs) and the other channels are all different breakers, not sure this matters, just want to make sure no one makes assumptions about what is on channel 1.

[jkeljo]

i would be willing to test this, i actually only got data flowing into the production integration today so don't mind loosing data

Cool, thank you! I will probably start looking into this closer to the holidays.

i see folks making assumptions solar is on channel 1

No worries. I think mine's on 31 or something. No assumptions will be made. :)

[peteraclausen]

@jkeljo

I would like to think there will be complete flexibility on what CT's are used for what measurements. In my case I have 3 GEMs deployed. 2 of them are next to my 200A sub-panels at my main dwelling and the 3rd one is at my shop building which is where my solar arrays are deployed.

My 200A Split CTs are as follows:

GEM1 CH1 Meter Base L1 (Grid Net) GEM1 CH2 Meter Base L2 (Grid Net) GEM2 CH1 House Feeder L1 GEM2 CH2 House Feeder L2 GEM3 CH7 From Solar Sub-Panel L1 (Solar Net) GEM3 CH8 From Solar Sub-Panel L2 (Solar Net) GEM3 CH9 Shop Feeder L1 GEM3 CH10 Shop Feeder L2

I know I could have connected the above CT's is pairs and used a single Channel on the GEMs for each of my 4 main measurements, but I like to measure each leg individually to make sure my loads are balanced across both phases and then sum them up for each total as needed.

My instantaneous readings as I type this are:

Grid L1 -4210W (this is what I pushing back into the grid right now on L1) Grid L2 -4281W (this is what I pushing back into the grid right now on L2) House Feeder L1 4675W House Feeder L2 4619W Solar L1 9806W Solar L2 9864W Shop Feeder L1 693W Shop Feeder L2 715W

Scenarios like the above will be supported, right?

[BenK22]

Ben from Brultech here. If there's anything we can provide to help with integration (hardware/firmware/etc) just send me an email and I'll see what we can do.

[jkeljo]

@peteraclausen should be. I looked back up at the comments and the assumptions were being made just for the sake of examples. As far as I understand the energy integration, it doesn't care about channel numbers, you can specify any entity that provides the right kind of data. I think you could even pull solar from one integration and grid from something else entirely.

[sdwilsh]

The biggest blocker right now is getting @jkeljo's PRs reviewed here and landed by the core team. I'm reviewing code as a first-pass to try and help, but neither of us has the ability to land the code.

If folks really want/need this now, I've setup a serial-to-mqtt project that has an easy-to-use docker image that uses the same underlying library that the Home Assistant integration uses. I'm currently using this in my home with a Raspberry Pi attached to the second COM port because I wanted to get stuff into the energy integration after getting my GEM in August.

[jkeljo]

@BenK22 that's awesome. We're all set hardware wise for getting the energy integration working with one or more GEMs. We don't have any ECM's to test with, though, so if we ever want to support those it would be helpful to have the hardware.

[sdwilsh]

As far as I understand the energy integration, it doesn't care about channel numbers, you can specify any entity that provides the right kind of data. I think you could even pull solar from one integration and grid from something else entirely.

This is true, although I found it a bit confusing at first. You can add as many entities as you want to each category, and Home Assistant will track each contribution. My setup is similar to @peteraclausen where I have split my mains to see how balanced they are (although I only have one GEM). I could add each one manually to Home Assistant, but the serial2mqtt program I wrote does the aggregation for me so it was easy to know what to put into the energy dashboard.

[scyto]

If folks really want/need this now, I've setup a serial-to-mqtt project that has an easy-to-use docker image that uses the same underlying library that the Home Assistant integration uses.

Can this connect over IP instead of com - for example to my dashbox-ip:8001 to get to comport 1 on the gem?

[peteraclausen]

For those of us with separate CTs on each phase, I wonder if the Energy Integration supports that directly, or if one would be better off building Entity Templates that sum up L1/L2 for each consumer/producer you want to track, and then pick that from the Consumed Entergy (kWh) / Entergy returned to the grid (kWh) drop downs? Not even sure Entity Templates are supported from within the Energy Integration module...

[BenK22]

@BenK22 that's awesome. We're all set hardware wise for getting the energy integration working with one or more GEMs. We don't have any ECM's to test with, though, so if we ever want to support those it would be helpful to have the hardware.

@jkeljo NP, just email me a shipping address and we can send one out.

The ECM-1240 uses a similar binary packet to the GEM with header identifier 0x3, the biggest difference is the 5 AUX channels are 4-byte counters. After a quick look at siobrultech-protocols, I think the packet side could be implemented in there with a few conditional statements (unless it's preferred to have it separate from the GEM code).

[jkeljo]

@emontnemery sorry, I somehow missed your earlier question.

Can this issue be closed since #58764 has been merged, or are more changes needed?

Config flows are not done, but this issue mentions lack of energy dashboard support which was fixed by #58764 as I understand it

No, we still have a ways to go in order to implement long-term statistics for this integration, as well as throttle down the events so they don't blow up the recorder. That will require configuration changes, so I need to first add a config flow. #59871 is the next step in that direction. Once that's in we can add the UI and then make the changes needed to get the sensors set up properly for long term stats.

[sdwilsh]

Can this connect over IP instead of com - for example to my dashbox-ip:8001 to get to comport 1 on the gem?

Sorry to say, @scyto, that it would require a number of changes to support, but maybe if @jkeljo doesn't get any traction on reviews by the end of the year, I'll figure out how to make that happen.

For those of us with separate CTs on each phase, I wonder if the Energy Integration supports that directly, or if one would be better off building Entity Templates that sum up L1/L2 for each consumer/producer you want to track, and then pick that from the Consumed Entergy (kWh) / Entergy returned to the grid (kWh) drop downs? Not even sure Entity Templates are supported from within the Energy Integration module...

You can add multiple entities, @peteraclausen, so you could have them separate. They would be tracked separately, but summed up for each category (from grid, from solar, etc). I do not know if entity templates work or not, but you don't need them.

The ECM-1240 uses a similar binary packet to the GEM with header identifier 0x3, the biggest difference is the 5 AUX channels are 4-byte counters. After a quick look at siobrultech-protocols, I think the packet side could be implemented in there with a few conditional statements (unless it's preferred to have it separate from the GEM code).

I remember skimming the ECM packet document and suspecting it'd be an easy port to add, which is why I named the library like I did. Everything is under the gem sublibrary though because I wasn't sure how we'd structure stuff once we actually incorporated support for other devices.

I've been meaning to post about that library to your company's forum, @BenK22, as the intent was to allow more folks to easily build helpful python programs that work with your devices, especially since the btmon script isn't well-maintained these days. I'm also happy to transfer sdwilsh/siobrultech-protocols/, sdwilsh/aiobrultech-serial, and sdwilsh/brultech-serial2mqtt to a Brultech GitHub organization so these (hopefully) helpful libraries can have owners if something were to happen to me. Feel free to reach out if y'all are interested (you and I corresponded in early July 2020 if you need my email address).

[peteraclausen]

No, we still have a ways to go in order to implement long-term statistics for this integration, as well as throttle down the events so they don't blow up the recorder.

@jkeljo I was curious about this. I added InfuxDB and excluded all the GEM entities from recorder in my installation as per the recommendation on the main GEM Integration page here. So do I need to back out those exclusions again once the events have been tamed?

[jkeljo]

So do I need to back out those exclusions again once the events have been tamed?

The long-term statistics used by the energy dashboard are gathered by recorder, so we have to be sending at least some events to it. What you'll likely see when this is done is that there will be some new entities that report energy (kWh) on a slower cadence (user-configurable, but defaulting to something like once every 30 mins or once an hour). The power entities will continue to report as fast as your GEM is set up to send information.

So you won't want to back out your exclusions entirely, but depending on how you specified them you might need to tweak them a bit. E.g. on my setup I've just excluded the entire sensor domain; I'm going to need to change that to be more specific about just excluding the really chatty sensor types.

[sarahmva]

One of the issues I have seen with GEM in the past is that it stopped reporting for unknown reasons. It did get better in recent releases and by increasing the reporting interval (1 minute now, I would prefer 15 sec.) I only have the ethernet module and I do get all the data. I am looking for.

I agree, writing so much data into recorder is not the best option even when migration to MariaDB. Even InfluxDB is sometimes struggling with the amount of data.

I wonder if there could be two statistics per GEM channel: A short term reporting as often as GEM sends data and only retain the data for 48 hours and a long term where the data is reported to recorder every 30 minutes. Maybe derive the data from the other sensor?

[BenK22]

One of the issues I have seen with GEM in the past is that it stopped reporting for unknown reasons. It did get better in recent releases and by increasing the reporting interval (1 minute now, I would prefer 15 sec.) I only have the ethernet module and I do get all the data. I am looking for.

Check Flow Control on COM1/2 and make sure they haven't become enabled, especially if an older unit. The Flow Control pins are left floating so if enabled they can get in a state where the GEM doesn't think it's clear-to-send.

[sarahmva]

@BenK22 : Thanks for the suggestion - I think I did figure out the issue: I had multiple DS18b20 connected to the GEM via a fairly long cable. Ever since I have disconnected them, the GEM has been stable. I assume that this caused issues. I will connect them via ESP, much easier.

[jkeljo]

@BenK22 I received the test ECM device in the mail a while ago, thank you! (I didn't want to leave you wondering.) Hopefully I'll be able to set it up and work on parser support for it soon!

[peteraclausen]

That's great to hear. Very much looking forward to testing out what you come up with!

[BenK22]

@BenK22 I received the test ECM device in the mail a while ago, thank you! (I didn't want to leave you wondering.) Hopefully I'll be able to set it up and work on parser support for it soon!

I was actually going to ask about that, thank for the update. I think my emails might be hitting your spam (or I have the wrong email), we've been having some issues with our mail provider over the last few months.

Let me know if you need any help on that front.

[42Network]

Hello. I am very interested in this topic, especially the ECM-1240 part. I have three ECM-1240 units monitoring my 42 circuit panel. I have no solar nor do I have a Greeneye. I'd be very willing to test anything done for HA and ECM-1240 on my equipment.

I've worked with BenK22 over the years. I use a slightly-customized version of Matt Wall's btmon.py that Ben converted from Python 2 to 3. btmon is running on a dedicated RPi3 with a quad serial cable connected to the three ECM-1240s. Then btmon.py publishes the power metrics to a local instance of Emoncms running in a VM on the same server (Mac Mini) where I have HA running. Then I use the existing Emoncms HA integration to put the data on various dashboards.

I would love to eliminate Emoncms from the above workflow.

I will also separately check out the suggested serial-to-mqtt project.

[sdwilsh]

@42Network, your best bet for starting out would probably be in https://github.com/sdwilsh/aiobrultech-serial, which is shared by the serial2mqtt and this integration.