Incorrect aggregated (1d) results

[andrzej-r]

It is a rather elusive issue, I noticed it in the past but was having trouble reproducing it. I've now managed to capture screenshots of it today (taken at 9:01am, in case it matters).

Data averaged over a period of 1 hour show a periodic daily activity (as one would expect from a solar power generation plot). However, when I zoom out a bit more, to trigger averaging over 1 day periods, I get daily averages that do not match the former plot. Specifically, daily average on 14/4 is zero and first (partial) day on record 11/4 is significantly overestimated. Other days look broadly aligned with hourly averages and report from energy dashboard.

Hourly averages (statistic=mean stacked areas for "to grid", "to battery", "to house", statistic=max unstacked for "total max"):

The same but with daily averages:

I have a feeling this may be triggered by moving boundaries of averaging periods. For example, now at 10:05am, the "zero" day has shifted to 12/4 - see below. Is there a way of forcing the aggregation to start at midnight, regardless of the current time? While this does not explain or fix the issue, it would likely be a sufficient work around for this plot.

Solar production plot from energy dashboard for reference:

[dbuezas]

Hi Andrzej! I agree with your suspicion that the HA API is not handling boundaries correctly. Another user opened an issue in home assistant which is relevant to this: https://github.com/home-assistant/core/issues/88696#issuecomment-1444033948

I may try to work around it if this is not fixed in the upcoming months

[tomatensaus]

With the latest update I also noticed that my graphs are disappearing. Recently upgraded to Home Assistant 2023.4.6 4 days ago and the graphs for the last 4 days are missing with the incorrect values being reported. Is there any way to work around the issue? During the day the current day values are reported fine, but when the day changes the values are completely wrong

type: custom:plotly-graph
view_layout:
  grid-area: daily
entities:
  - entity: sensor.deyeinverter_summary_day_pv_energy
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Solar" + "🔆" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    filters:
      - filter: i>0
    marker:
      color: rgb(255, 155, 48)
  - entity: sensor.deyeinverter_summary_day_load
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Load" + "⚡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    filters:
      - filter: i>0
    texttemplate: '%{y}'
    marker:
      color: rgb(95, 182, 173)
  - entity: sensor.deyeinverter_summary_day_grid_import_buy
    statistic: max
    name: |
      $fn ({ ys,meta }) =>
        "Grid Import" + "💡" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    filters:
      - filter: i>0
    marker:
      color: rgb(84, 144, 194)
  - entity: sensor.deyeinverter_summary_day_battery_discharge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Discharge" + "🖱️" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    filters:
      - filter: i>0
    marker:
      color: rgb(151, 90, 182)
  - entity: sensor.deyeinverter_summary_day_battery_charge
    statistic: state
    name: |
      $fn ({ ys,meta }) =>
        "Battery Charge" + "🔋" + "(" +ys[ys.length - 1]+"kWh)"
    period: day
    type: bar
    texttemplate: '%{y}'
    filters:
      - filter: i>0
    marker:
      color: yellow
hours_to_show: 12d
title: null
refresh_interval: 120
defaults:
  yaxes:
    fixedrange: true
layout:
  legend:
    bgcolor: rgba(0,0,0,0)
    itemsizing: constant
    font:
      size: 11
  height: 410
config:
  displayModeBar: false
  scrollZoom: false

I can also mention that setting to hourly shows the data so looking up the data for a day seems the the broken part

Another stacked graph with the same issues for last 4 days (while today shows correct values, by tomorrow it will be wrong too)

[dbuezas]

If you reload the page and the data is still missing, then you lost data from your database. Have you restored a backup?

Do the standard HA plot cards have this information?

[tomatensaus]

Refresh generates the same graph, even checked it from my phone. same result

No the data is all there, selecting hourly I can confirm that the values shown is not 0. I have not restored any DB.

Initially I noticed strange values for yesterday but today looks good. After midnight (next day) the values that was good is suddenly now zero and today is good again. This pattern repeated itself for a couple of days and in the end it can only be a breaking change somewhere in Home Assistant/HA OS. Data before that still shows correctly Added some console logs and I can see 0 values coming through

Here is the output for hourly.... the hours during the night have a 0 number till solar production starts. I have checked these numbers and they correspond to what I expect to see

200: 10.7
201: 10.7
202: 0
203: 0
204: 0
205: 0
206: 0
207: 0
208: 0
209: 0
210: 0.1
211: 0.5
212: 0.9
213: 2.1
214: 5
215: 7.9
216: 9.4
217: 10.5
218: 10.8
219: 11
220: 11
221: 11
222: 11
223: 11
224: 11
225: 11
226: 0
227: 0
228:0
229:0
230:0
231: 0
232: 0
233: 0
234: 0
235: 0.3
236: 0.8
237: 2.2
238: 5.2
239: 8.5
240: 10
241: 11
242: 11.2
243: 11.4
244: 11.4
245: 11.4
246: 11.4
247: 11.4
248: 11.4
249: 11.4
250: 0
251: 0
252: 0
253: 0
254: 0
255: 0
256: 0
257: 0
258: 0
259: 0.2
260: 0.4
261: 0.9
262: 1.6
263: 2.2
264: 3.2
265: 4
266: 4.5
267: 4.6
268: 4.7
269: 4.7
270: 4.7
271: 4.7
272: 4.7
273: 4.7
274: 0
275: 0
276: 0
277: 0
278: 0
279: 0
280: 0
281: 0
282: 0.3
283: 0.8
284: 2.1
285: 5.1
286: 7.9

Also standard component historic values

[dbuezas]

Same if you remove your filters?

[tomatensaus]

Same if you remove your filters?

Same output... no filters

period: day simply returns 0 for all days since I upgraded... before that it still works. Today will show till I reach midnight and then also show as broken with 0 values

[dbuezas]

Are you sure that the statistics card in HA shows different data? Note that "history" and "statistics" are two different tables

[tomatensaus]

Are you sure that the statistics card in HA shows different data? Note that "history" and "statistics" are two different tables

No I am not aware of this (quite new to home assistant, sorry). I showed the history in my last post and need to read up where I can find the statistics to view the values then. Sorry if I wasted your time

[dbuezas]

Oh, no problem :). Here's a standard Home Assistant card to see the statistics: https://www.home-assistant.io/dashboards/statistics-graph/

Let me know how it went!

[tomatensaus]

Day

Hour

Generated by the statistics graph built-in card So clearly there is something broken in home assistant statistics and not in ploty-graph. The data returned is not correct.

Is there a way to work around the issue so that my graphs (ploty-graph) show correctly?

[dbuezas]

Yes, the long term statistics had a hiccup then. You could add another trace with the same entity but no statistics. You could also make a custom fn filter that only shows it if the statistics are missing. If you are not a programmer it will be a bit of a project in itself

[tomatensaus]

Long term statistics consistently have hiccups, from the graph below the data for 2May also shows zero today (Yesterdaty it was displaying the correct values). At the end of each day the daily values in statistics are set to zero. Compare with the graphs from yesterday where 2 May had values.

[dbuezas]

You may want to add an issue in HA core for this. It looks like an issue in the long term statistics component

[tomatensaus]

I logged this bug, hopefully it will make sense https://github.com/home-assistant/core/issues/92391

[tomatensaus]

I did notice that selecting "change" as the statistic option shows the correct value on my graph in the standard components. Screenshot below

I looked through https://github.com/dbuezas/lovelace-plotly-graph-card/blob/master/src/recorder-types.ts and could not find any equivalent definition. Is it not implemented? Is it new? or did I perhaps miss it?

At this stage I am thinking that using "statistic: state" does not work as my last_reset is not populated on the entity, and it is a daily total so being reset each day which might cause the statistics to record the daily movement from 0 to 0.

[dbuezas]

That must be new. Could you try adding an "fn" filter and see if "change" is inside the statistics parameter?

[tomatensaus]

statistics

I think I figured out how to log the statistics object, and no I do not see the "Change" as one of the returned values above.

[dbuezas]

That looks correct. I don't know what this "changed" statistic is.

[dbuezas]

We're passing the data that HA sends back unmodified to that object: https://github.com/dbuezas/lovelace-plotly-graph-card/blob/master/src/cache/fetch-statistics.ts#L29

[tomatensaus]

I see it is listed in the HA source code

https://github.com/home-assistant/frontend/blob/dev/src/data/recorder.ts

export interface Statistic {
  max: number | null;
  mean: number | null;
  min: number | null;
  change: number | null;
}

[dbuezas]

High five people! https://github.com/home-assistant/core/pull/92848/files#diff-39951366dd56691c4e51988078d3295e0ba6fc8a6089f719ac4f054886435608R1659

The wizzard emontnemery from the Home Assistant team made the HA api align data to multiples of periods, I'll close this issue and the fix will land in the next monthly HA.