search

esphome / feature-requests

ESPHome Feature Request Tracker
https://esphome.io/
419 stars 26 forks source link

Integration for ADE7880 (3-phase power monitor, used in Shelly 3EM) #1579

Closed michaelpiron closed 8 months ago

michaelpiron commented 2 years ago

Describe the problem you have/What new integration you would like

I have a Shelly 3PM 3-phase energy monitor: https://shelly.cloud/products/shelly-3em-smart-home-automation-energy-meter/ Inside the product is an ESP8266 SoC and a ADE7880 power monitoring chip. I’d like to put ESPhome firmware on my Shelly 3PM for two reasons:

Therefore I see an added value in having an ADE7880 component in ESPhome, next to the existing ADE7953 (used in the single-phase Shelly PM) -> https://esphome.io/components/sensor/ade7953.html

Please describe your use case for this integration and alternatives you've tried:

use case: being able to do 3-phase energy monitoring

Additional context

I started developing a ADE7880 component, based on the existing ADE7953 component and taking into account the ADE7880 datasheet specifications. The majority of the work is done and is available in a fork of the esphome repository: https://github.com/michaelpiron/esphome/tree/dev/esphome/components/ade7880 However, some outstanding issues remain at this moment, for which I would need some help:

furthermore: in the Tasmota Github community, people also raised the idea of having a Tasmota component for the ADE7880: https://github.com/arendst/Tasmota/discussions/13515 So there are also other people interested in having an ADE7880 integration.

michaelpiron commented 2 years ago

Setting the GPIO linked to the ADE’s IRQ in a correct way is important, as wrong configuration can lead to heating of the device and potential fire hazard! Refering to this related issue: https://github.com/arendst/Tasmota/issues/7991#issuecomment-698252659

michaelpiron commented 2 years ago

A Tasmota component for the Shelly 3EM is being built as people started contributing photos of the PCB's and investigated code dumps. All information is in this discussion: https://github.com/arendst/Tasmota/discussions/13515.

As I'm not that proficient with coding in Python and C (yet), would someone be willing to complete my draft code for an ADE7880 component in ESPHome? Link to my fork: https://github.com/michaelpiron/esphome/tree/dev/esphome/components/ade7880

kroimon commented 1 year ago

Hi Michael! Did you ever get to the finishing touches of your component? I would love to run ESPhome on my Shelly 3EM :⁠-⁠) If not, I could probably pick up where you left. What were your plans with the component going forward?

michaelpiron commented 1 year ago

Hi @kroimon , I didn’t progress further on this, unfortunately. My fork of esphome is still the most recent one. In the meantime, I see quite some activity in the Tasmota discussion (see link in my previous post) with interesting info on how to download the device calibration parameters from the original firmware, before putting new firmware on it. If you can help out, that would be very welcome. I won’t have time in the next months, due to other priorities. Feel free to take the code from my fork. If you can get it into the esphome codebase, mentioning my name as co-author would be kindly appreciated.

majuss commented 1 year ago

It's standard and known for a long time, if you're not downloading the calibration data for your 3EM and flash it, then it's worthless electronic waste.

See: https://templates.blakadder.com/shelly_3EM.html

stich86 commented 1 year ago

@michaelpiron what's the current status on support 3-phases system without neutral? Shelly OS doesn't support this type of installation...

thx!

kpfleming commented 1 year ago

Since I have four 3EMs and a strong desire to run ESPHome on them, I've begun working on this component. Hopefully I'll get some early progress posted in the next day or two.

kpfleming commented 1 year ago

Development is happening here.

Current status:

Tasks left (at least):

kpfleming commented 1 year ago

Major progress... I now have most of the sensors reporting as planned, and the numbers match very closely to the Shelly firmware. Tomorrow I'll start working on docs!

The image below has the top three channels from a 3EM running Shelly firmware, and the bottom three channels from a 3EM running ESPHome, with the current clamps on the same wires.

image

kpfleming commented 1 year ago

The PRs to merge this into ESPHome are up for review now.

If anyone would like to help test this before it is merged, the docs are here. You can add the component to your configuration like this:

external_components:
  - source: github://kpfleming/esphome/ade7880
  - components: [ ade7880 ]

I'll be opening a PR later today for the esphome-devices repo with a full configuration example for the Shelly 3EM.

kpfleming commented 1 year ago

Example config for Shelly 3EM.

dinamitemic commented 1 year ago

The PRs to merge this into ESPHome are up for review now.

If anyone would like to help test this before it is merged, the docs are here. You can add the component to your configuration like this:

external_components:
  - source: github://kpfleming/esphome/ade7880
  - components: [ ade7880 ]

I'll be opening a PR later today for the esphome-devices repo with a full configuration example for the Shelly 3EM.

Hi guys, my Shelly 3EM is on its way so I started to prepare everything for the firmware conversion. First of all thank you for the custom component. I tried to follow your infos for the manual integration of the external component but I had to make some adjustements. Using ESPHome 2023.7.1

\\config\esphome.esphome\external_components\c4a2a214\esphome\components\ade7880

(Please note that c4a2a214 can be different, be sure to edit the right one) Re-adding the following lines:

CONF_PHASE_A = "phase_a"    
CONF_PHASE_B = "phase_b"    
CONF_PHASE_C = "phase_c"`

Put them just above CONF_NEUTRAL = "neutral"

kpfleming commented 1 year ago
  • Try to compile once and wait for the custom components to be downloaded. It will fail with errors finding some constants in const.py

Ahh, you are correct, thanks for the details. This branch won't work completely standalone as an external component because it contains a patch to const.py. Your workaround is reasonable until the component has been merged.

dinamitemic commented 1 year ago

Ciao @kpfleming, I can confirm you that everything is working correctly. I just want to report 2 infos:

  1. Esphome logs are reporting a high execution time for the ade component:

    [17:47:07][D][ade7880:159]: update took 180 ms
[17:47:07][W][component:204]: Component ade7880.sensor took a long time for an operation (0.18 s).
[17:47:07][W][component:205]: Components should block for at most 20-30ms.

  2. In my calibration data I got _"currents": 266881794 that is strange and cannot be accepted since the maximum is 8388607. Have you any idea abou it?

kpfleming commented 1 year ago
  1. Esphome logs are reporting a high execution time for the ade component

Which ESP board are you using, and how many of the ade7880 sensors do you have enabled?

2. In my calibration data I got _"currents": 266881794 that is strange and cannot be accepted since the maximum is 8388607. Have you any idea abou it?

Unfortunately I do not; I had the same situation on my units but since I don't have clamps installed on the neutral conductors I didn't care about it. Someone will eventually need to talk to Shelly about those calibration constants to find out what the correct way to use current_n and current_s are.

dinamitemic commented 1 year ago

Which ESP board are you using, and how many of the ade7880 sensors do you have enabled?

It is a Shelly 3EM, I took your config example and just added some minor changes. If you want to check I can attach the full config.

Someone will eventually need to talk to Shelly about those calibration constants to find out what the correct way to use current_n and current_s are.

Hmm, interesting, I will try to find out more and eventually I'll write to the shelly team.

kpfleming commented 1 year ago

It is a Shelly 3EM, I took your config example and just added some minor changes.

Yes, of course... it was a silly question I asked :-)

Most likely the cause of those warnings is that the logger level is set to debug (either in your config, or because it is the default). When that is the case, the sensor core code emits a "Sending state..." message for every sensor value change, and it takes 3-6ms for that message to be emitted. As a result, if a single component has more than 7 or 8 sensors enabled, the total time required to update them all will exceed 50ms and the warning will be triggered.

You can ignore the warning, as there is no actual problem. You can also set the log level to info and the warnings will probably go away.

dinamitemic commented 1 year ago

Hi @kpfleming this morning I switched to Tasmota in order to make some comparison. With ESPHome I noticed some wrong calculation on the energy side, let me add some details:

Let me provide you an example with some numbers: From 17:58 to 18:58 I had an average power consumption of around 240W, I would expect a computed energy of 240Wh. But if I look at the forward energy channel it reports the following data 17:58 --> 0,02Wh 18:58 --> 0,07Wh

Have you experienced something similar? Maybe I'm missing something. With Tasmota and (of course) the same calibration parameters I can find a correct correlation between power and energy computation.

kpfleming commented 1 year ago

Thanks for the report! I have not seen this sort of problem on my own devices; I ran two 3EMs in parallel (one with the Shelly firmware and one with ESPHome) for about a week to confirm that the reported numbers were close enough to be acceptable. There were minor differences, but I assumed those were caused by CT clamp and/or calibration differences.

This component does gather the Wh data differently than the Shelly firmware and Tasmota do; they both read the Wh registers every second, and then report the sum periodically. This component reads the registers at each update_interval, under the assumption that the ADE7880 itself won't overflow the registers during that time. What is your update_interval set to?

kpfleming commented 1 year ago

Also, are only the Wh sensors incorrect (meaning the other sensors are correct)?

dinamitemic commented 1 year ago

What is your update_interval set to?

I have not specified it so should be 60s as default. I can confirm that also in the logs the data were retrieved every minute.

Also, are only the Wh sensors incorrect (meaning the other sensors are correct)? I cannot test both fw in parallel like you (since I have only one device) but the data that I've seen were the same.

You're idea about the register overflow and the update time can be the point. Are you updating the sensors faster?

kpfleming commented 1 year ago

No, I'm also using the default update interval of 60s.

kpfleming commented 1 year ago

In order to avoid spamming everyone else subscribed to this issue, please open one in https://github.com/kpfleming/esphome and we'll work through this there. I'll report back results to this issue once we've reached a conclusion.

kpfleming commented 1 year ago

@dinamitemic I've just pushed an update to the PR branch. It resolves the issue with not being able to provide the current_s calibration value... Shelly reports those differently than the voltage calibration values (negative values are reported as large positive values). I don't expect that this will address your Wh reporting problem, but there's a chance it could.

I've also removed the commit which moved around the CONF_PHASE_A/B/C constants, so now the branch can be used as a normal 'external component' with no patching required, and no need to suppress refreshes.

kpfleming commented 1 year ago

Since the weirdness with the current_s values is specific to the Shelly 3EM and not to the ADE7880 in general, I've reverted those code changes and replaced them with a note in the 'Devices' guide for the Shelly 3EM documenting the procedure for converting the value to the necessary format.

ghost commented 1 year ago

Is this component dependent on having all three phases connected? I'm testing it with just neutral and A wired - and I'm not getting the expected readings.

[12:24:41][D][sensor:093]: 'Neutral Current': Sending state 0.01175 A with 2 decimals of accuracy
[12:24:41][D][sensor:093]: 'Phase A Current': Sending state 0.01353 A with 2 decimals of accuracy
[12:24:41][D][sensor:093]: 'Phase A Voltage': Sending state 0.11360 V with 1 decimals of accuracy
[12:24:41][D][ade7880:159]: update took 21 ms

Using the https://deploy-preview-499--esphome-devices.netlify.app/devices/Shelly-3EM yaml with my calib.dat entries merged (and a lot of the other entities commented out, same result with the full config for that matter).

kpfleming commented 1 year ago

No, the phases (all four) are independently read and reported. Would you be willing to test with Tasmota to see if it reports the same readings (or correct readings) using the same calibration values?

ghost commented 1 year ago

Would you be willing to test with Tasmota

When pasting the template it ignores the ADE7880 IRQ, adding both back after and running the calib rule in the console gives me all 0 values.

Restoring the stock firmware from backup gives me a good voltage reading.

Guess its time to see if they've made a new revision or something. Will compare available pictures to my hardware. Edit: hardware looks identical to pictures from tasmota thread, except different batches of the ADE7880. Will poke it some more when I have more time.

r10r commented 1 year ago

@kpfleming I have issues compiling a firmware for the shelly3em because my calibration data does not validate. I followed the instructions on https://deploy-preview-499--esphome-devices.netlify.app/devices/Shelly-3EM#calibration-data

The power values are to low and return the error value must be at least -8388608 on compilation. The phase_angle values are negative and return the error value must be at least 0 on compilation.

This is my calibration data:

{
  "state": 0,
  "rms": {
    "current_a": 3115853,
    "current_b": 3145899,
    "current_c": 3118537,
    "current_n": 55507790,
    "current_s": 3683735,
    "voltage_a": -754486,
    "voltage_b": -745905,
    "voltage_c": -768669
  },
  "angles": {
    "angle0": -10091,
    "angle1": -10096,
    "angle2": -10099
  },
  "powers": {
    "totactive": {
      "a": -15427176,
      "b": -15431191,
      "c": -15434854
    },
    "apparent": {
      "a": 1383297,
      "b": 1383213,
      "c": 1383263
    }
  },
  "energies": {
    "totactive": {
      "a": -56297,
      "b": -56294,
      "c": -56297
    },
    "apparent": {
      "a": 146044,
      "b": 146123,
      "c": 146229
    }
  }
}

This is my esphome sensor configuration:

sensor:
  - id: ade7880_device
    platform: ade7880
    irq0_pin:
      number: GPIO13
    irq1_pin:
      number: GPIO5
    reset_pin:
      number: GPIO16
    frequency: 60Hz
    phase_a:
      name: Phase A
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3115853
        voltage: -754486
        power: -15427176
        phase_angle: -10091
    phase_b:
      name: Phase B
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3145899
        voltage: -745905
        power: -15431191
        phase_angle: -10096
    phase_c:
      name: Phase C
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3118537
        voltage: -745905
        power: -15434854
        phase_angle: -10099
    neutral:
      name: Neutral
      current: Current
      calibration:
        current: 55507790
kpfleming commented 1 year ago

Thanks for the report! I'm investigating this now, it appears that the 3EM calibration data may be the 'raw' values from the I2C bus and not the documented values from the datasheet. I'd already dealt with that for the current sensor calibration, but I'll check it for the others too.

r10r commented 1 year ago

I‘ve already disabled the checks and now I‘m getting correct power values. So it seems the calibration values are valid and must not be converted. 😀

Am 21.10.2023 um 14:11 schrieb Kevin P. Fleming @.***>:

 Thanks for the report! I'm investigating this now, it appears that the 3EM calibration data may be the 'raw' values from the I2C bus and not the documented values from the datasheet. I'd already dealt with that for the current sensor calibration, but I'll check it for the others too.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

kpfleming commented 1 year ago

Yes, that's what I suspected. It's not clear how the Shelly firmware is deriving these calibration values as they do not align with what the ADE7880 datasheet indicates, but since the Tasmota firmware blindly accepts them and writes them to the chip (and that produces the expected results), this component could do the same. I'll rework the code to not care about the interpretation of the calibration data at all.

kpfleming commented 1 year ago

All done; the three PRs have all been updated to accept any integer value for the calibration registers.

Alibloke commented 9 months ago

@kpfleming I get a compilation error for the neutral channel:

INFO ESPHome 2023.12.8
INFO Reading configuration /config/heatpumpmonitor.yaml...
INFO Detected timezone 'Etc/UTC'
INFO Generating C++ source...
INFO Compiling app...
Processing heatpumpmonitor (board: esp01_1m; framework: arduino; platform: platformio/espressif8266@3.2.0)
--------------------------------------------------------------------------------
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
Dependency Graph
|-- ESPAsyncTCP-esphome @ 2.0.0
|-- ESPAsyncWebServer-esphome @ 3.1.0
|-- DNSServer @ 1.1.1
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- noise-c @ 0.1.4
|-- Wire @ 1.0
|-- ArduinoJson @ 6.18.5
Compiling .pioenvs/heatpumpmonitor/src/main.cpp.o
src/main.cpp: In function 'void setup()':
src/main.cpp:833:27: error: 'struct esphome::ade7880::NeutralChannel' has no member named 'set_current_calibration'; did you mean 'set_current_gain_calibration'?
  833 |   ade7880_neutralchannel->set_current_calibration(44329829);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~
      |                           set_current_gain_calibration
*** [.pioenvs/heatpumpmonitor/src/main.cpp.o] Error 1
========================= [FAILED] Took 10.79 seconds =========================

If I comment out the neutral: block it compiles and installs fine, though voltage is 0.

edit: voltage is fine, I got my phases mixed up

kpfleming commented 9 months ago

The calibration constant config entries recently changed; current is now called current_gain. I'm not sure why your build didn't report a failure when parsing the config, as it should have failed validation if it still has current in the calibration section. Can you post the ade7880 section of your YAML config?

Alibloke commented 9 months ago 
sensor:
  - id: ade7880_device
    platform: ade7880
    irq0_pin:
      number: GPIO13
    irq1_pin:
      number: GPIO5
    reset_pin:
      number: GPIO16
    frequency: 50Hz
    phase_a:
      name: Phase A
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3164337
        voltage_gain: -789668
        power_gain: -1310795
        phase_angle: 188
    phase_b:
      name: Phase B
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3177488
        voltage_gain: -813346
        power_gain: -1305703
        phase_angle: 204
    phase_c:
      name: Phase C
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3072438
        voltage_gain: -789750
        power_gain: -1304045
        phase_angle: 136
#    neutral:
#      name: Neutral
#      current: Current
#      calibration:
#        current_gain: 44329829
kpfleming commented 9 months ago

Well, that certainly looks correct :-) Can you run an esphome clean for that YAML config before compiling again? I want to make sure there aren't any old build files left. I've also just rebased the branch onto current dev so your next build should re-clone the branch.

kpfleming commented 9 months ago

Ahh... never mind. I just reproduced with the updated PR branch. I'll fix it now.

kpfleming commented 9 months ago

The PR branch has been fixed and CI tests now pass, your config should compile properly. Thanks for the report, I didn't catch this locally as I don't use the 'neutral' phase monitoring.