Question about data sampling

[elyorkhakimov]

Hi, I read through Microchip ATM90E32AS datasheet and the respective ESPHome page, but can't seem to understand how this energy meter is sampling data. I am currently running 6 channel meter (v1.4 rev1) with only 4 channels in use (waiting for more CT Clamps for the remaining 2 channels).

Question is really about the values that are send via MQTT. In yaml file update_interval is set to 10 seconds. Does this mean that meter reports instant value of Voltage, Wattage, Current, Power factor, etc, at a given instant "t", and then after 10 seconds reports the instant value at "t+10sec" ? What about Wh readings?

Or, rather, does the meter sample with a specific sampling rate (what is it?) values for each of the sensors and takes an average of all values over the configured 10 seconds?

If there is a better place to ask these questions, please let me know and I will move this discussion there.

[CircuitSetup]

That's a good question! The ATM90E32AS can write to registers several times a second. But from what I understand this can only happen if the register is read first. I have to double check this for the energy registers. I know this is the case for the cumulative energy registers (which aren't used in ESPHome anyway).

Either way, with a 10 second interval it would either be taking the energy reading at that moment or it would be reading the value from 10 seconds prior, then allowing a new value to be written. There is probably a way to configure an average in ESPHome if you wanted to do that, but I haven't looked into this yet.

If you want to calculate watt hours, this can be done in ESPHome as well.

[elyorkhakimov]

Thanks for response! I just did a non-scientific experiment:

• set "update_interval" to 20 seconds
• subscribed to MQTT messages for "totalmain/power" and "totalmain/dailyenergy".

Now the idea is to check each of the values at time t1, then right after the value is reported and within 20 seconds interval turn power up something really power hungry (how about 1.3KW kettle?), and then turn it right off. That means there will be no current going through monitored CT clamp at time t1+20sec.

So, my thought process is - if the reported values average the array of all data points gathered within that 20 second time interval, I should see some non zero values in MQTT messages at time t1+20sec.

And I am glad to report that indeed, daily_energy value appears to show correct data. But power does shows what seems to be the max value in the interval.

[elyorkhakimov]

[elyorkhakimov]

However, after some more experiments, I see that 30 seconds update interval changes the abovementioned behavior. No reported values show any changes. As if nothing happened within that 30 seconds interval

[elyorkhakimov]

ok, scratch all of the above. After I changed the interval to 30 seconds and then back to 20 seconds or even 10 seconds, the "cumulative" behavior stopped and now all values report immediate values. This means that if any power load comes one for brief moment during the interval, it will not be accounted for. This is a problem.

[elyorkhakimov]

Did some more tinkering. Sorry for spamming here. I don't know why esphome was sending MQTT messages with what seemed to be cumulative Wh reading, but since now that's gone, I set the update interval to 10 seconds, and with the kettle "turned on" can see that the total_daily_energy sensor in ESPHome takes the power sensor reading in watts and integrates this instant reading value over the specified "update_interval" time value (10 seconds in my case). That's good, and I think it provides a reliable energy metering with known error margin. The source of this error is that in this case we are relying on ESPHome to calculate the Wh energy values based on instant power readings at specified interval. Rough error margin calculation could be:

• at 10 seconds update interval, the Wh reading can produce error by as much as 0.2778% (10s/3600s*100=0.2778)
• for a 6 channel board with fully loaded 2x100Amp +4x20Amp CT clamps, this could translate to ~5.6KWh of unaccounted energy per hour. (280Amp120V0.2778/100 *60min= 5600Wh) I think this is a good level of precision, especially accounting that most households won't be consuming this much energy. And I also can't think of loads that consume high amounts of energy in such short <10sec bursts (except maybe tankless water heaters with electric fans).

But I think the used M90E32AS chip could yield really high precision results.

Based on the section 3.5 in datasheet for M90E32AS (http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-46003-SE-M90E32AS-Datasheet.pdf), this chip samples values at 1µs, and "accumulates" them. Even on the board itself there are contacts for pulses for each phase and also the blue LED blinks every so often (I actually don't know if it counts only Active energy or total sum of Active+Reactive+Apparent).

The energy accumulation runs at 1 MHz clock rate by accumulating the power value calculated by the DSP processor.
The power accumulation process is equivalent to digitally integrating the instantaneous power with a delta-time of about
1us. The accumulated energy is used to calculate the CF pulses and the corresponding internal energy registers.

Anyways, I am looking forward to see if it is possible to add per-phase energy sensors exposed in ESPHome, which pull data directly from chip registers at a fixed interval (in my opinion once every 30 or even 60 seconds should be sufficient). This way we won't have to rely on "approximated" math in ESPHome.

[elyorkhakimov]

After more research, I came to conclusion that the ESPHome reports instantaneous sensor values via MQTT messages.

I see there are registers in esphome/components/atm90e32/atm90e32_reg.h that is forked here: https://github.com/esphome/esphome/compare/dev...CircuitSetup:dev ; but the energy sensors do not seem to be implemented in ESPHome.

/* ENERGY REGISTERS */
static const uint16_t ATM90E32_REGISTER_APENERGYT = 0x80;  // Total Forward Active
static const uint16_t ATM90E32_REGISTER_APENERGYA = 0x81;  // A Forward Active
static const uint16_t ATM90E32_REGISTER_APENERGYB = 0x82;  // B Forward Active
static const uint16_t ATM90E32_REGISTER_APENERGYC = 0x83;  // C Forward Active
static const uint16_t ATM90E32_REGISTER_ANENERGYT = 0x84;  // Total Reverse Active
static const uint16_t ATM90E32_REGISTER_ANENERGYA = 0x85;  // A Reverse Active
static const uint16_t ATM90E32_REGISTER_ANENERGYB = 0x86;  // B Reverse Active
static const uint16_t ATM90E32_REGISTER_ANENERGYC = 0x87;  // C Reverse Active

@CircuitSetup , how feasible it is to implement the reading from these registers into individual ESPHome sensors we could use in yaml? I recognize the ESP32 might run out of memory, but I wonder if we could sacrifice web server component and only keep ota and mqtt. Might even sacrifice ota in favor of getting energy sensor data from chip, rather than relying on ESPHome to calculate approximated values.

[CircuitSetup]

Wow, you've done a lot of tinkering! I'm glad you're getting to know the IC!

The energy registers listed above wouldn't be hard to add to ESPHome, but they would require some additional coding to determine what the number being read means over time. I haven't experimented much with these registers, but from what I gather, as a 16 bit integer, the max they can hold is around 0.2kWh. I do know that they are reset once read though.

The Arduino library divides the output of APENERGYT by 100, then by 3200 to get kWh.

[elyorkhakimov]

Fascinating! Please don't mind my excitement. I haven't coded in over 10 years, and this whole thing makes me want to come back.

I see where that division by 100 (resolution 0.01CF) and then 3200 (MC of 3200imp/kWh) comes from. (from http://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-46003-SE-M90E32AS-Datasheet.pdf)

It appears as though Arduino library only implemented registers for Total active/reactive/apparent energy values. The per-phase values are commented/not implemented.

I am now trying to understand what would it take to add the support for energy sensors in ESPHome. I can't seem to make a logical connection between "sensor" definition in YAML and how/where it gets interpreted and used to pull data from chip via atm90e32.h/atm90e32.cpp. Where would be a good place to start research?

[elyorkhakimov]

this commit is probably all bits of data I need to add support for pulling energy data from chip. https://github.com/esphome/esphome/commit/5a67e723895297f78bae960cbfd38c787192a85e#diff-a545a9e0e9b82494c44c9cdadf63a9be

what IDE would you recommend using? I would like to contribute, and recognize I need lots of guidance. Thanks for your patience!

[CircuitSetup]

Yes, that commit is exactly what I did to add other registers.

For development I would recommend Atom with the platformio plugin installed. You'll want the dev build of ESPHome. Getting it to compile is a bit tricky at first because all of the dependencies have to be loaded.

[elyorkhakimov]

Thanks! I tried installing platformio plugin for atom, but that did not seem to work. Then tried VSCode, but when compiling getting lots of errors. Did not spend time troubleshooting it. In the meantime, forked dev branch and made several changes, but then don't seem to figure out how to upload it with "esphome xx.yaml run" command. I will troubleshoot more, but in the meantime thought I'd ask if you could try to run it and see if it works for you. https://github.com/esphome/esphome/compare/dev...elyorkhakimov:energy-add

[elyorkhakimov]

OK, I think I'm more or less done with the implementation. Folks over at ESPHome Discord helped me ascertain where I made mistakes.

I don't have a bench or some type of high-precision Wh-meter/Joule meter to estimate a Wh reading error, and therefore can't calibrate; but comparing to Kill-A-Watt (only measures to hundredth's of kWh) the reading is spot on. I tested by boiling a whole electric kettle of water several times. I initially thought to create new unit UNIT_KWATT_HOURS, but then decided to just keep base Wh unit.

What's next? How can I get this into dev branch? Would someone want to test this code some more?

Oh, and one more observation from the way it's implemented - upon reading the register, it is cleared. I might need to figure out a way to re-use or implement some type of sensor that "accumulates" the readings and then reports at less frequent intervals. At the moment, with default "update_interval=10s", I just end up seeing a very small "increments" in consumed energy measured in Wh.

My use case is to log this data in influxDB, but I recognize to make it usable more broadly, the integrated value over an hour or a day might be needed

Code is in "energy-add" branch: https://github.com/elyorkhakimov/esphome/tree/energy-add

This is an excerpt from the YAML:

      power_factor:
        name: ${disp_name} CT3 Power Factor
        state_topic: ${disp_name}/ct3/power_factor
      current:
        name: ${disp_name} CT3 Amps
        id: ct3Amps
        state_topic: ${disp_name}/ct3/current
      power:
        name: ${disp_name} CT3 Watts
        id: ct3Watts
        state_topic: ${disp_name}/ct3/power
      forward_active_energy:
        name: ${disp_name} CT3 WattHours
        id: ct3FAWatts
        state_topic: ${disp_name}/ct3/forward_active_energy
      reverse_active_energy:
        name: ${disp_name} CT3 WattHours
        id: ct3RAWatts
        state_topic: ${disp_name}/ct3/reverse_active_energy
      gain_voltage: ${voltage_cal}
      gain_ct: ${current_cal}```

[CircuitSetup]

Excellent work! From here you can open a pull request with the dev branch of esphome. You'll have to do the same for the documents page, being sure to add examples of the new parameters and how they're used.

[elyorkhakimov]

I had a chance to run my setup for few weeks now, and without any calibration of CT sensors (YHDC SCT-013 100A/50mA) spliced/extended with DIY unshielded twisted copper wire extension are giving about 1.51% error. For example, Solar inverter says it generated 22.48kWh, and my setup says 22.14, and from few days ago - 24.37 from sensor vs 24.74 from solar inverter. Both CTs are connected to IC2 (ct5 and ct6).

The only global ESPHome adjustment I made was of the voltage_cal:, set it to 7280 vs stock 7305, as it seemed that reported voltage was off. Should I do any type of calibration for CTs ? By the way, IC1 and IC2 are reporting slightly different voltages. What could be a cause of that ?

The DIY cable extensions are made using 2x 16AWG for first CT, and 2x 18AWG for the second CT, both stranded copper TFFN wires from HomeDepot. I locked two wires of 30ft length in drill chuck and then spun it. Really, eyeballed the number of twists. Then wires untwisted a bit, and I believe the effective extension length is about 28ft. I solder-spliced in this extension cable into existing 24AWG cable that was already attached to CT sensor. I thought the noise would be unacceptable, but, it seems that there are no issues.

[CircuitSetup]

Glad to hear you're getting fairly accurate results, and that the extensions worked!

For the differences in IC1 and IC2, this is likely because of variance in the resistors. They are 1%. If you need to, you can calibrate each voltage input differently.

For the CTs you can also adjust the calibration for a more accurate result. It may be easier to do all of the CTs at once, clamped around the same wire, with the same load.

[smurfix]

@elyorkhakimov I assume you're no longer working on this?

I'd really like to log "real" energy use instead of instantaneous loads.