Issue with P1 smart meter (teleinfo) real time graphic

darrepac commented 5 years ago

Hi

Platform: Raspberry Pi Stretch Domoticz Version: latest stable V4.1017

I am using Teleinfo plugin (which is P1 smart meter type) for power consumption of my house. Teleinfo is gathering many information but we will focus on real-time power consumption of the house (PAPP item in teleinfo).

I made some changes in my consumption and I was tracking down the power consumption and I felt there was some stranges values in the last 24hours graph. This graph is showing a value each 5min. If guy on this thread is correct (https://www.domoticz.com/forum/viewtopic.php?f=4&t=29270&p=223409#p223409) the value is an average value of the measures taken during last 5min. True or not, there is an issue somewhere.

I installed an influxdb database and used grafana to store and show the value of realtime consumption and when I compare the graph with domoticz graph, there is clearly a problem. (see attachement) Domoticz is showing during 10min, nearly 0W value

In the same time, Grafana (and so the Domoticz panel) is showing about 300W consumption. Grafana

That's an example but I replicated it many times. I have some difficulties to find what could cause this problem.

gizmocuz commented 5 years ago

The short log (last 24 hours) are values stored in the database every 5 minutes, these are the last values received. It is not averaged, not min/max is calculated

The values in the day/month/year graph are using the 'counter' values, so this graph is always correct

What you see here is that influxDB is logging every value, maybe even every 10 seconds , so this graph is much more accurate

But I also notice that the Grafana chart looks like 10 times higher and the chart in Domoticz 10 times lower 22 Watt seems also a very low consumption...

I don't know if there is internally a difference between Teleinfo/P1 related to the graphs, but I am going to check this as well.

When I look in Grafana and check the current consumption (in Watt), the Domoticz P1 widget is showing the correct value:

And when I wait 5 minutes, the last value is stored in the short log:

So this looks correct. It could be a telegram issue, or at the time of the snapshot for the short-log the consumption was 22 Watt ?

Internally in Domoticz there is already a method to store the min/max/avr for a sensor, this is used by some types like wind sensors and temperature sensors (trend calculation) It might be a good idea to do this for Power sensors as well, but that's another story

The P1/Chart is in Domoticz for years .... besides from the snapshot not having taking the 'max' value but the actual value, I can't think of anything wrong related to the chart and logging (I can only test with the Dutch P1 meter, but i expect no issues

If you export your database and look in the meter tables, do you see those low values as well, or do you see correct values ?

Could you create a widget like mine that shows the latest received value in Grafana, and set the dashboard refresh rate to 10 seconds Then compare the Domoticz P1 widget's value against this ?

rwaaren commented 5 years ago

@gizmocuz : Understand from your very detailed explanation that I misjudged the code in WebServer.cpp (around line 14315) by assuming that the real time (5 minute) value in the graph for ALL counter like types, iIncluding P1 meter, is an average.
Can you please help me understand if this value is an average for other counters ? Thx.

gizmocuz commented 5 years ago

You mean this line ?

float tdiff = static_cast<float>(difftime(atime, lastTime));

I am 99,999% sure that the values in the short-log is the last value received, and is not an average

The code for the graph plotting has been updated some time ago, you could be right about this as some (for example zwave) nodes reports once in a while, but the original values it uses are coming from the short-log, which are not averaged and are snaphots of the last value received. Going from 280 Watt to 22 Watt seems not a good average ...

Before going into a deeper investigation we need to know also the answers to the above questions:

Widget Grafana (last value), vs Domoticz widget
Export of the database and checking the values in the meter short-log table against grafana (+/- the same, not 200 Watt difference)

darrepac commented 5 years ago

Hi

I can already reply for this one as I did it before: Widget Grafana (last value), vs Domoticz widged Yes they are the same.

I have to try to check when the problem appear (22W is damn low but what I hope is true cause I have installed some PV panels...but 300W could be true as well) I do agree that graph based on counter are 100% correct.

Le 8 sept. 2019 à 09:32, gizmocuz notifications@github.com a écrit :

Widget Grafana (last value), vs Domoticz widget

darrepac commented 5 years ago

Right now:

instantaneous values in Grafana or domoticz widget : never below 300W
Domoticz graph last values: something below 200W
Grafana graph: ok

The look of the graph is correct but not the values... I wait during 5 min: last value of domoticz widget, 380W, value in the graph, 288W.

Le 8 sept. 2019 à 09:32, gizmocuz notifications@github.com a écrit :

You mean this line ?

float tdiff = static_cast(difftime(atime, lastTime));

I am 99,999% sure that the values in the short-log is the last value received, and is not an average

The code for the graph plotting has been updated some time ago, you could be right about this as some (for example zwave) nodes reports once in a while, but the original values it uses are coming from the short-log, which are not averaged and are snaphots of the last value received. Going from 280 Watt to 22 Watt seems not a good average ...

Before going into a deeper investigation we need to know also the answers to the above questions:

Widget Grafana (last value), vs Domoticz widget Export of the database and checking the values in the meter short-log table against grafana (+/- the same, not 200 Watt difference) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

darrepac commented 5 years ago

ok, I checked the database (see pictures below): database seems with correct values and the same than Grafana, but Domoticz graph is using other values...

PS: and yes, having averaged values during 5min period in domoticz graph would make more sense, but that's a feature request than a bug report ;)

grafana_graph domo_graph database

gizmocuz commented 5 years ago

@darrepac , thank you for the fast response ! Based on your values of the database I will try to replicate this a.s.a.p.

gizmocuz commented 5 years ago

Having looked into this (@rwaaren , code line is ~13645 when debugging, but could be on multiple places)

What happens is that the log is using the 'counter' fields of the meter, in this case it does

curUsage1 = 5801477 - 5801463 = 14

it knows we are 300 seconds between the actual and the previous log entry (5 minutes, tdiff)

Then it calculates the assumed Watt used between those two timepoints

float tdiff = static_cast<float>(difftime(atime, lastTime));
float tlaps = 3600.0f / tdiff;
curUsage1 *= int(tlaps);

tdiff = 300 tlaps = 12

result = 14 * 12 = 168

Assuming the calculation is correct, you could say that 168 is the average from the 300 seconds

The previous code just showed the values that where stored in the shortlog

darrepac commented 5 years ago

I was suspecting that the graph was using the counter and not the stored realtime consumption. It lacks somes precision but is doing « naturally » the averaging.

What I don’t understand in your email is the difference between what was done before and what you changed. Your formula sounds correct so I guess it is not how it was done?? (= it is your modification)

Le 8 sept. 2019 à 15:01, gizmocuz notifications@github.com a écrit :

Having looked into this (@rwaaren , code line is ~13645 when debugging, but could be on multiple places)

What happens is that the log is using the 'counter' fields of the meter, in this case it does

curUsage1 = 5801477 - 5801463 = 14

it knows we are 300 seconds between the actual and the previous log entry (5 minutes, tdiff)

Then it calculates the assumed Watt used between those two timepoints

float tdiff = static_cast(difftime(atime, lastTime)); float tlaps = 3600.0f / tdiff; curUsage1 *= int(tlaps); tdiff = 300 tlaps = 12

result = 14 * 12 = 168

Assuming the calculation is correct, you could say that 168 is the average from the 300 seconds

The previous code just showed the values that where stored in the shortlog

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

gizmocuz commented 5 years ago

Before it was showing the values from the database table instead of calculating it So currently it's ok?

darrepac commented 5 years ago

I think I miss something... What I understood: BEFORE your code change: Was taking the difference between 2 counter values and x12 to get a Watt (cause 5min sampling). So in the table screenshot above: 5801588-5801560 = 28 28 * 12 = 336W (= the value shown in the screenshot graph above)

If I am correct, we can consider that this value has a precision of 1 * 12 = 12W (Note that while this formula seems correct, and so the graphic, it doesn't explain the important difference between the graphic values and the widget values... much more than 12W At first I thought that the graphic was not correct, but on the other I felt it was correct and the formula above, based on counter, proves it. So I think now the problem is from the widget and as the widget is directly linked with Teleinfo, I am guessing it is a problem there. May-be directly in the electric counter that is not supporting that I consume 0W. Cause right now, by night, so without solar panel effect, the graphic and the widgets values are aligned. I need to track this down)

AFTER your code change: What did you change? as it seems it was working well.

darrepac commented 5 years ago

Forgot what I said saying that grafana and domoticz graph sounded accurate during night (=without solar panel effect). Here is what happen this night: Grafana: mostly 300W Domoticz: seems centered around 200W

(Still using latest stable version, @gizmocuz I don't know if you commit any changes) domoticz_nuit grafana_nuit

darrepac commented 5 years ago

Note: widget and grafana are showing PAPP value (apparent power) from teleinfo which is a VA value. Domoticz graph, as you discover, is using counter values which is a W value. That could explain some difference between the 2, but no idea if that could explain such difference

gizmocuz commented 5 years ago

@darrepac , the 'widget' value is the last received usage from the meter itself the graph value is calculated

before the code change (i did not change this), my code (before this code change) was showing the values from the database, the short-log, snapshot's from the last received value from the meter itself (what you see on the widget)

Grafana is using the values from the meter itself (the values you see in the Domoticz widget)

Hope this explains it

darrepac commented 5 years ago

So you mean that you change from last received value from widget/meter, to the diff onto counter index values? Did you already make a Beta of it so I can check it?

gizmocuz commented 5 years ago

@darrepac, no... no changes, so again:

the widget is using the last value received from the P1 meter
the short log (in the database) are snapshots (every 5 minutes) of the last value received from the P1 meter
the graph calculates it based on the algorithm above

darrepac commented 5 years ago

now I think I undertood! :) so it sounds good on the domoticz side... I need to understand why there are such differences between meter values and counter (graph) values, but seems the reason is not on domoticz side.

darrepac commented 5 years ago

One doubt that you can clarify: is the widget value updated each time teleinfo is receiving a value or only when the new value is different from the previous value? [Edit: forget about it, it was my setting for Grafana that was updating only on value change]

gizmocuz commented 5 years ago

Okey closing issue, feel free to re-open when needed

darrepac commented 5 years ago

I am sorry but there is still something weird. I am really tracking down since this morning and I have an issue. As soon as the widget value should be 0W it is not updated or really really less frequently updated. If value >0W, I have a value nearly each 10sec (which my max update rate setting) If value =0W (or should be equal to 0W cause sometimes the widget value is not updated while it should be at 0W), no more updates during minutes. I checked the serial port input: no change depending the value

As soon as the value is no more 0W, the frequent updates start again. Is there any value check inside the plugin, may-be considering that 0W should not exist and dropping values??

Look at picture below 0W problem

darrepac commented 5 years ago

Look: Each time the power consumption is 0W, there is a hole in value

darrepac commented 5 years ago

In plugin teleinfobase.cpp, there seems to have a system to avoid to update if 2 consecutives values are the same: line 162-164 if ((teleinfo.pAlertPAPP != teleinfo.PAPP) || (difftime(atime, teleinfo.last) >= 290)) { teleinfo.pAlertPAPP = teleinfo.PAPP; BUT if power (=PAPP) is not null, the widget is well updated even if 2 consecutive values are the same (I checked several times and I am not talking about the timeout)

darrepac commented 5 years ago

If animated gif is working in github, here is the proof that 2 consecutive values > 0W are well updated in Domoticz, so the line code above should not be blocking for 0W values

darrepac commented 5 years ago

more tests today. I have seen that when consumption is equal to 0W, the widget is updated once every 5min nothing more. So we seem to fall into the "(difftime(atime, teleinfo.last) >= 290)"

gizmocuz commented 5 years ago

Is it a TeleInfo.cpp source problem ? Or a gui update problem ? To check if it is a gui issue, could you press F5 in the browser, does after this it shows 0 ?

darrepac commented 5 years ago

What is the problem, I don't know, but for sure it is not a GUI problem as it is the same in Grafana (screenshots above)

gizmocuz commented 5 years ago

Okey, then please debug the c++ code, maybe there is something like

if (usage != 0) then send the values

darrepac commented 5 years ago

In

    if ((teleinfo.pAlertPAPP != teleinfo.PAPP) || (difftime(atime, teleinfo.last) >= 290))
    {
        teleinfo.pAlertPAPP = teleinfo.PAPP;

        //Send data at mamximum rate specified in settings, and at least every 5mn (minus 10s as a grace period for the watchdog)
        if ((difftime(atime, teleinfo.last) >= m_iRateLimit) || (difftime(atime, teleinfo.last) >= 290))
        {

I would remove

(teleinfo.pAlertPAPP != teleinfo.PAPP) ||

The original idea is not bad: do not update if new value is the same than old value. But I see 2 issues with that (as far as I can see):

as it discards the update, it means that we do not update the widget and so the lastupdate timing field, which could cause problem
there could be an issue due to the teleinfo.pAlertPAPP = teleinfo.PAPP; between the two "if" statement. Use case: new_value != old_value, so we pass the 1st "if" we update "old_value" imagine we are below the rate threshold => we do not passe the next "if"

So we do not display the new value (due to rate) but we have updated the old value. So then if the new new value is the same than the new old value, we will not display it (but we never display the new old value due to rate) so we miss it completely. That's exactly what I am seeing in my grafana graph.

The only doubt that I have is the animated gif that I posted some comments above (but it could have been updated due to the same reason than above.

So yes, I would remove this : > (teleinfo.pAlertPAPP != teleinfo.PAPP) ||

domoticz / domoticz

Issue with P1 smart meter (teleinfo) real time graphic #3610