Moving average for Power Measurement

[rasti70]

Have you look for this feature in other issues and in the wiki? => YES

Is your feature request related to a problem? Please describe.

=> I have a few power measurement devices (Blitzwolf/Sonoff/Gosund) and noticed that some of them show really big fluctuations, e.g. I measured a 15W bulb and get measurements values of 13-19 Watts. This happens in particular with Gosund 111 / Blitzwolf SHP6 . On the contrary, Sonoff Pow, Gosund1, Gosund SHP2 seem to be pretty stable (fluctuations of +/- 1 Watt only)

Describe the solution you'd like A clear and concise description of what you want to happen.

=> I would like to have a moving average filter. The current sample rate or update rate of the values shown in the webinterface seems to be around 1-2 Hz. I would suggest a moving average filter which calculates the average of e.g. 10 measurements before displaying or publishing it. Obviously the refresh rate of the data shown in the webinterface will be lower, but getting an update e.g. every 10 or 20 seconds would be acceptable for me. This filter could be adjustable through the console, e.g. "movingaverage 0" or "movingaverage 50" . Maybe an IIR filter also will do the job.

Describe alternatives you've considered

=> I tried to implement a moving average filter in home automation software FHEM. This works, however I had to set the teleperiod to a very low value (10s) which imposes a lot of traffic on my mqtt/fhem server

Additional context

=> I posted this on fhem forum, too : https://forum.fhem.de/index.php/topic,90220.msg875650.html#msg875650 In the subsequent posts and elsewhere on the same forum I found at least 3 users more which had the same problem I had.

(Please, remember to close the issue when the problem has been addressed)

[ascillato]

Hi,

Your idea of doing some type of data processing for the power measurement is nice, but there are some points to be taken into account:

If the 15W bulb you are measuring is a cheap one, it will have a lot of noise that it is being injected in your mains AC. Also if you have other devices of that type, you will be having much more noise. There are some of these cheap ones that have a noise switching power supply and I saw others that have a capacitive power supply that produces a huge amount of noise that it is just an additive noise, not a random noise.

So, the moving average filter won't work in an additive noise environment. If the noise is just white noise type or pink noise type, a moving average filter will do the job. In an environment where you have an asymmetrical noise, a moving average filter will give you all the time a higher value than the real one.

An IIR or FIR filter will do the job but only if it is tuned to be a low pass filter to filter those noise spikes. Anyway, the sampling rate for power measurement needs to be adjusted to be a multiple of the switching frequency (this needs to be adjusted experimentally).

So, as you have explained that this power fluctuations only happens in those devices but not in for example a Sonoff Pow, and that this needs a little more complex processing rather than just a moving average in order to have a little more accurate measurement, I think it is not going to be added soon. I would like to know other opinions also.

Anyway, your idea and your request is nice.

A workaround for now so as to have more precise power measurements is to use devices like sonoff pow R2 or the PZEM sensor supported by Tasmota.

Thanks for sharing your ideas :+1:

[rasti70]

Hi, thanks for your swift reply. The bulb I have is an old fashioned conventional 15W light bulb with filament, thus almost a 100% ohmic resistor which doesn't add noise at all. I think it's the wifi socket switch itself which causes trouble. Of course I can use another device but I'd really like to have a solution for this Gosund 111 / Blitzwolf SHP6 , because these are really tiny and so nice to look at. Maybe its small form factor is the reason for the fluctuations because it favours inductive or capacative coupling. I immediatly flashed them with tasmota so I didn't try the original firmware and app at all but I suppose the original manufacturer firmware or app also uses some kind of data value smoothing algorithm because they should have the same raw measurement values.

[arendst]

I notice the same behaviour as you do. Most energy sensors are kind of stable but my shp6 acts the same as yours.

[rasti70]

I notice the same behaviour as you do. Most energy sensors are kind of stable but my shp6 acts the same as yours.

OK so what would you suggest ? Do you think implementing some kind of prefiltering in tasmota is an option or do I simply need to live with the poor measurement result of the SHP6 ?

[arendst]

As a low power measurement takes some seconds you would need to calculate the average over 12 seconds. In that time the load could have been switched off or changed so that makes it very dificult to calculate.

I'm afraid we have to live with it unless some clver dicky finds a quick filter option.

[rasti70]

Yes of course that's the price for it. Nobody will find a quicker filter. I already wrote it before : "Obviously the refresh rate of the data shown in the webinterface will be lower, but getting an update e.g. every 10 or 20 seconds would be acceptable for me"

So I currently see 3 options

• wait for better hardware versions (delivering better raw data) of SHP6 like devices or
• accept the poor data without any filtering or
• implement a slow software filter

[MaceMoneta]

I do this with my own code on my server. There's no need for a delay to collect measurements; just use a rolling average.

set lastMeasurement to saved value, or zero if none. measurement = (( 2 * lastMeasurement ) + newMeasurement) / 3 lastMeasurement = measurement

This will cause a ramp-up for the measurement over the measurement period, and a ramp-down when usage returns to zero. Fluctuations are smoothed over the samples. I use this with temperature and humidity readings as well, since they can similarly fluctuate.

An example in bash:

$ lastMeasurement=0
$ newMeasurement=100 ; # <=== 100W device turned on
$ for i in $(seq 1 14); do measurement="$(echo "scale=2;measurement=(((2*$lastMeasurement)+$newMeasurement)/3);measurement"|/bin/bc -q)" ; lastMeasurement=$measurement ; echo $(printf "%0.f" $measurement) ; done

33                                                                                                                                                     
56                                                                                                                                                     
70                                                                                                                                                     
80
87
91
94
96
97
98
99
99
99
100

$ newMeasurement=0 ; # <=== Device turned off
$ for i in $(seq 1 14); do measurement="$(echo "scale=2;measurement=(((2*$lastMeasurement)+$newMeasurement)/3);measurement"|/bin/bc -q)" ; lastMeasurement=$measurement ; echo $(printf "%0.f" $measurement) ; done                                                                         

66                                                                                                                                                     
44
30
20
13
9
6
4
3
2
1
1
0
0 

Edit: decreased the number of samples to speed up the convergence, and added an example of the ramp-up/down.

[ascillato]

Your method is really good for smoothing data with white noise.

If you have unbalanced noise, that method will show a value greater than the reality.

For example if you have a 15w bulb, so 15 should be the real value, but if that bulb is noisy with spikes only greater than 15w like

15, 18, 15, 15, 16, 14, 19, 16, 15, 15

Your smooth value will be higher than the real 15 because of this spikes in the measurement.

If you have a IIR tuned as low pass filter, you should have a more accurate value for that case.

Anyway, your algorithm will show a more stable value than the actual raw measurement.

[curzon01]

15, 18, 15, 15, 16, 14, 19, 16, 15, 15

to eliminate spikes in a test series do not use average calculation, use the median:

14, 15, 15, 15, 15, 15, 16, 16, 18, 19 (=15)

and everything is fine. Works also with raising/falling value.

Disadvantage of all statistical methods is the time delay of between real value and displayed one.

[rasti70]

Disadvantage of all statistical methods is the time delay of between real value and displayed one.

Correct. And (at the moment) in order to apply any statistical method (smoothing), all the data needs to be transfered from the front end (socket with tasmota) to the server which performs the filtering.

Instead of transfering raw data through e.g. MQTT to do the filtering on a separate server, this should be done right at the frontend.

[andrethomas]

A resistive load such as a 15W light bulb may also not remain 15W due to fluctuations in temperature and of course the source voltage. Also, calibration with a 15W load on a device that can measure 2.2KW may not give an accurate enough value for calibration purposes.

15W of 2200W = ~0.6% so it would be nearly impossible to get an accurate calibration figure from this.

[rasti70]

@andrethomas : Calibration at a single load and the resulting precision within different load ranges is a completely different issue. Again: I opened this issue because of high fluctuations of the raw data and I asked for a software filter at the frontend (i.e. integrated in tasmota). I implemented such a filter at the backend (home automation system FHEM) and it works as it should. I want the same filter or something similar in tasmota in order to avoid the massive transfer raw data to the mqtt/fhem server. First raw data processing is one of the most inherent tasks of an intelligent front end - and should not be done by the home automation system.

[ascillato]

As @andrethomas said, this is a hardware limitation, so any data processing added to this will make a marginal improvement to the measurement and also to be used only for lower power measurement values. At higher power values, this noise should not be present.

Anyway, the median approach from @curzon01, I think that will result in a more stable and real measurement and also faster than IIR or moving average methods.

I don't know if that processing will be better to be added in Tasmota or in the home automation software itself.

[andrethomas]

@rasti70

Calibration at a single load and the resulting precision within different load ranges is a completely different issue

Technically incorrect - they are related.

I do not think implementing filtering on Tasmota side is viable as filtering strategies will change depending on use case - i.e. resistive load vs. inductive loads, just to give a simple differential example... lets not forget about switch mode loads :)

So my recommendation is not to implement filtering on Tasmota side at all.

[ascillato2]

@rasti70

From the technical side and calibration measurement theory, a one point calibration should be done close to the maximum rating in order to have the minimal error all over the range. So, if you calibrate any device with a low value, the error is much bigger than calibrating from a bigger value.

Anyway, the spikes in the raw values is indeed as you explained a hardware limitation.

[rasti70]

I don't know if that processing will be better to be added in Tasmota or in the home automation software itself.

In very inert systems (temperature for example) it is not a problem to do first transmit the data to the home automation system and then do the filtering. Power measurement is different. Of course loads can change quickly, however, I think that a time resultion of 10-20 seconds is an acceptable tradeoff to get rid of these nasty fluctuations.

[andrethomas]

I think that a time resultion of 10-20 seconds is an acceptable tradeoff

Your opinion. I reject this from a technical perspective though :)

[rasti70]

From the technical side and calibration measurement theory, a one point calibration should be done close to the maximum rating in order to have the minimal error all over the range. So, if you calibrate any device with a low value, the error is much bigger than calibrating from a bigger value.

Ideally, for precision , we should calibrate at a few points e.g. 0.1W , 1W, 10W, 100W,1000W. But better advance one by one - this field is so ample, we could open another topic for that.

[rasti70]

I think that a time resultion of 10-20 seconds is an acceptable tradeoff

Your opinion. I reject this from a technical perspective though :)

I don't want a Wifi oscilloscope.. I need a nice looking wifi socket switch with energy monitoring function.

[andrethomas]

I need a nice looking wifi socket switch with energy monitoring function.

Let me know when you find one that does not require hiding its inefficiencies by averaging output data :)

[rasti70]

Let me know when you find one that does not require hiding its inefficiencies by averaging output data :)

I fear we won't see such a device at a reasonable price in the near future. For that reason I made this request .....

[andrethomas]

I think there's an important part which you are not completely grasping though.

Below is my mains input as measured by an industrial grade voltage sensor.

Over a 24 hour period an hysteria of up to 5% is observed on the AC supply voltage to the house.

Lets assume a 50W light bulb for calculation purposes, which is rated at 50W 220V so theoretically should draw around 227mA @ 220V meaning an effective reactive resistance of 968 ohms.

If we apply ohms law and bump the input voltage by 5% up to 232V and assume the internal resistance of the bulb remains 968 ohms this would translate to ~240mA - or recalculated to a wattage would mean ~55W - Aside from the additional unwanted 5W of power what is more interesting to note is the physics behind the math and what the math tells us about the physics - we bumped our voltage up by 5% but our power demand increased by 10%! Weird, huh?

In reality things are a little different though.

If you ever find yourself in posession of a variable transformer you should take the oppertunity to play with it a little and see what kind of hysteria is injected in your theoretical model of how things should work in an ideal environment.

If you were not a programmer you would get terribly confused by the following if statements (which actually mean the same thing)

if (known == unknown) { }
if (unknown == known) { }

So I'll leave you with that and, of course, that adding a software filter to fix a hardware problem has no place in the firmware itself... you may of course post process the data in your HA software by all means necessary for any particular use case.

[rasti70]

I think there's an important part which you are not completely grasping though.

I don't think so.

Below is my mains input as measured by an industrial grade voltage sensor. Over a 24 hour period an hysteria of up to 5% is observed on the AC supply voltage to the house. we bumped our voltage up by 5% but our power demand increased by 10%! Weird, huh?

This is only weired for people living in a linear world.
P=(UxU)/R and 1,05*1,05 =1,10 So not weired at all.

In reality things are a little different though.....

Sorry I don't understand what you want to tell us with all the rest so I won't comment on it.

So I'll leave you with that and, of course, that adding a software filter to fix a hardware problem has no place in the firmware itself...

Intelligent sensors do exactly perform that task : Data prepreprocessing which in many cases means data smoothing. But, useless to discuss this further more, we obviously have different opinions about that.

you may of course post process the data in your HA software by all means necessary for any particular use case.

I explained above, why this is not a good solution: Massive raw data transfer to the home automation server.

I don't understand why you're fighting so enthusiastically against my suggestion: Filtering should be an option, not default. If it's implemented in the software, you are not obliged to use it.

[andrethomas]

Massive raw data transfer to the home automation server.

Unless you're sending tens of thousands of MQTT messages per second you won't even make a measurable impact on a Raspberry Pi A+ running mosquito and node-red.

I don't understand why you're fighting so enthusiastically against my suggestion: Filtering should be an option, not default. If it's implemented in the software, you are not obliged to use it.

I'm not fighting it, just questioning the need for it. Make a PR if you feel that way inclined :)

[ascillato]

This post is interesting. Let's summary ideas so far:

• Why to add this processing for spikes in power measurement?

The devices

Gosund 111 Blitzwolf SHP6

have a really bad repeteability of power measurements for lower power loads < 100w.

The rest of power measurement capable devices and sensors supported by Tasmota have a better resolution.

• Where to add or not the processing for spikes in power measurement in HA software or in Tasmota?

In HA software: Can be added any type of processing and representation customized for each use-case. IIR, FIR, moving average filters, statistical median or others. (ACTUAL WORKAROUND)

In Tasmota: Can be added a simple processing that uses few flash and few RAM and don't block Tasmota for other tasks.

• What to add?

In the case of adding to Tasmota seems that a statistical median would smooth the readings, limiting spikes and having a close-to-real values.

Could be taking 9 samples (Round(Teleperiod/9) sampling rate?) ordering from lowest to highest and displaying in the teleperiod message the sample number 5.

• When to add?

Theo is the owner and the one deciding to add it or not. So, I think he was clear saying that it is needed that someone can prove a fast and light method to improve those devices.

So, this request is on hold up to someone would like to program, test and prove a enhancement for those devices.

As a workaround for now is to use another power measurement device or a PZEM sensor for more stable power measurements lower than 100w, or process the data in your home automation software.

Thanks everyone for sharing theirs ideas. Very interesting and very appreciated.

[rasti70]

Thank you Adrian for your summary.

So if Theo as the owner decides to add, we still need a volunteer to implement such a routine.

I think that the small simple median-like subroutine as Adrian mentioned above is already sufficient. Or any other filter, algorithms are standard and C-coded examples can be found on the www, e.g. https://atwillys.de/content/cc/digital-filters-in-c/

However, I won't be the volunteer to program that. I can't. If I could program, I would have done it already myself. I have written small pieces of c-code many many years ago when I was a student. The subroutine itself probably will not the problem, this should be a rather small piece of code, however when I look at the current tasmota code this is so overwhelming that I don't even know where I should start, which arguments to hand over from which function etc. But in any case I would volunteer for testing.

[andrethomas]

@ascillato I'll write a compact filter class for Tasmota so it can be re-used by other sensors where necessary. For now, I'll just add a class for median filtering but we can add other strategies as and when needed.

[rasti70]

@ascillato I'll write a compact filter class for Tasmota so it can be re-used by other sensors where necessary. For now, I'll just add a class for median filtering but we can add other strategies as and when needed.

Thank you Andre. Sounds good. !! Please let me know where and how this is available for testing.

[andrethomas]

I've made a PR for the SensorFilter library here https://github.com/arendst/Sonoff-Tasmota/pull/4741

Included is a change I made for the LM75AD temperature sensor which uses 5 elements whilst picking element number 3 (the midpoint) as the new value which will be used.

If and how Theo will implement this in the Gosund 111 / Blitzwolf SHP6 driver is not up to me.

[arendst]

See #4741 for intermediate comments.

Continuing on the SHP6, from what I see as current values I guess the sensor is mounted before the SHP6 logic and therefor if a load of 25W with powerfactor 1 (incandescent bulb) the current jumps up and down from 0,137A to 0,190A measuring both bulb and SHP6. Why power jumps up and down from 23.1W to 27.1W beats me as I'm quite sure the SHP6 doesn't draw 4W...

[andrethomas]

I do not have an SHP6 but I'll order one as soon as BG has them in stock.

@arendst Have you measured the current consumed by the SHP6 on the input side when it is turned on but has no load connected to it? It should be around 0.4W without being turned on and perhaps a little more if the device is allowing power through without a load connected to it.

[rasti70]

0.4 W would be wonderful. It's rather 0.8 (off) - 1.1 (on) watts

[andrethomas]

@rasti70 Is that measured between live AC and the device using a DMM?

[rasti70]

It has been measured with a simple cheap energy monitor. Ebay # 132836883619 maybe not too precise, however, the wattage reflects the experience of other fhem users.

[andrethomas]

The ESP cannot consume more than 0.4W at maximum and will be closer to 0.1W on firmware version 6.3.0.10 and higher. The 0.3W variance between on and off (60mA @ 5V for the relay) seems in line with what I get with ebay relay boards given they also have a led (~20mA) on them - They measure in at 80mA.

So there's a phantom 0.1W-0.7W floating around somewhere in the rest of the electronics.

[flashtel]

Greeting all, here is for an inspiring 2019, this is a fascinating discussion.

At the risk of going off topic and excuse me if I am posting in the wrong place.

I'm developing a couple of energy switch designs at the moment and I am wanting to contribute to the tasmota code where possible.

The first is a 30amp 1 1/2 HP motor start Songle SLA-5VDC-SL-A , HLW8012 / 7W 0.001 shunt theoretically can run to 83 amps. Current Sensing is done in the Neutral and the esp is at ground. The current enclosure is in IP68, with DS18B20 included, It's designed for pool pumps, electric storage heaters ect.

If you are interested for a sample PM me, we are due to finish the first production pilot mid January.

The second is a 3 phase unit with sensing in each phase and neutral (to be decided) for bore pumps and 3 phase heating/motor applications.

Anyway my point from this conversation, where do I begin to start understanding the tasmota frame work, it is overwhelming. Who is best to communicate with around HLW8012 code enhancements?

I would say I'm a beginner at programming but I want to learn and I want to study tasmota, my background is Industrial Electrical construction/engineering.

I echo this.

"If I could program, I would have done it already myself. I have written small pieces of c-code many many years ago when I was a student. The subroutine itself probably will not the problem, this should be a rather small piece of code, however when I look at the current tasmota code this is so overwhelming that I don't even know where I should start, which arguments to hand over from which function etc."

[flashtel]

[flashtel]

[flashtel]

I have 10 development boards available now, it's now using the ESP-WROOM-02 module to work with the pick and place machine and this module has FCC certification. I am applying for SAA certification in Australia and will be pursing US / EU certification after that. The production run is using a better case as well. I can send you a dev board/enclosure now if you are interested to assist in integrating with tasmota. The New Year is starting, it's an exciting time. Enjoy Paul

[KPK75]

Anyone noticed that DMM shows around 10V AC on SHP6 while is off ?? I was expecting 0 like on POWr2.

[rasti70]

Anyone noticed that DMM shows around 10V AC on SHP6 while is off ?? I was expecting 0 like on POWr2.

try to take the shp6 out, turn it 180° and plug in again. Do you still measure a voltage when switched off ?

[KPK75]

Normally it can't be done ( "french" connector ). But I have tested on the socket wihout ground and it shows 1.5 V AC when put upside down. Dangerous.

[rasti70]

OK. Just measured the same. Approx. 1.7V AC when the relay switches L1 and different voltages from 20-40V AC (I connected&diconnected a few times to make several measurements) if the relay switches N .

I just compared that to a "quality product" of a renowned German brand. https://www.elv.de/fs20-funk-schalt-spar-set-3-mit-1x-fs20-s8-3-3x-fs20-st-4.html Same result.

I currenly have no idea if this residual voltage when switching L1 is dangerous or not.

Switching only N instead of L1 is definitly dangerous, however, almost all switchable sockets on the market are constructed that way (to switch only 1 of the 2 conductors)

[KPK75]

I accidently touched wires while calibrating the device ( relay off ) and it wasn't nice. And Sonoff POWr2 shows 0 while off.

[rasti70]

I accidently touched wires while calibrating the device ( relay off ) and it wasn't nice.

So better don't do that again....

But : If you can insert the switch in the outlet only in one position and if then you switch N instead of L1 with the SHP6 then your electrical installation at home is faulty.

[andrethomas]

I accidently touched wires while calibrating the device ( relay off ) and it wasn't nice. And Sonoff POWr2 shows 0 while off.

The POW's can display voltage while the relay is off using SetOption21

But yes, touching anything metal on devices that are plugged in (be it off or not) is a job best left for rubberman :)

[Jason2866]

I received 2 gosund SP111 = Blitzwolf SHP-6 A 40 W bulb (old Edison type) Measured voltage and current and after calibrating (with Voltage an Power=Watt) i get this output:

[Jason2866]

So Apparent and Reactive Power is completly wrong! Power Factor has to be 1

[rasti70]

If your 40W value is stable over time, you are lucky. My shp6 is fluctuating. Apparent and Reactive Power is also wrong with other plugs, e.g. SHP2 / SP1

[Jason2866]

Second SP111 has same false Power Factor