Broken statistics period in 1.5.0

[FrnchFrgg]

First, since installing 1.5.0 plotly-graph behaves as if period: auto was set even if I set period: hour in defaults:/entity:.

Second, the transition between hour and day occurs roughly when there are 10 days visible, which does not mean 500 data points are visible (only 10, maybe 15 at most). Even if I considered that it sums over all six displayed entities, that's not 500 points by far. I can supply a GIF screen cast if you cannot reproduce.

[netweaver1970]

First, since installing 1.5.0 plotly-graph behaves as if period: auto was set even if I set period: hour in defaults:/entity:.

Second, the transition between hour and day occurs roughly when there are 10 days visible, which does not mean 500 data points are visible (only 10, maybe 15 at most). Even if I considered that it sums over all six displayed entities, that's not 500 points by far. I can supply a GIF screen cast if you cannot reproduce.

Same here, the switching to a higher aggregation level happens too fast or too late. Or not. Sometimes I get this: an hourly period but a with a time frame of 12 Months: (too detailed)

While most times I get for the same graph/time frame, a month period (too aggregated):

So basically the auto period determination is all over the place and is clearly depending on GUI state/mouse hover or not.

Also auto refresh switches back to the non-statistics source. I think this auto-refresh is the cause of most of the switch-backs to the non-statistics source.

Thanks for trying to get this fully working. We've come a long way over the last weeks!

[dbuezas]

Oh darn, I think I pushed the wrong branch. Shouldn't release half asleep, will fix today

[dbuezas]

ups...

[dbuezas]

Fixed in v1.5.1

[netweaver1970]

mmm, not there yet. I have a default period of 24 hr and buttons for 3 days, 7 days, 14 days, 1 month, 3 months, 1 year and 3 years. The below screen2gif capture shows the cycling through them. It looks like the 500 data points (defining the period value used) is too much for me, It stays too long at a 5 min period. Also, it looks like for the bigger time frames, 3, 12 and 36 months I'm also getting 5min periods, which is killing the performance of the graph and totally is against the idea of auto periods :) Is it possible to make that 500 adjustable either in YAML or at runtime (or pref. both)? And fix the error with higher time periods? Then people can tune the graph to what they're after?

[dbuezas]

I see. I'll make the default 100 for now. I'll implement custom resolution once these bugs are fixed. Can you open the dev console of the browser and see if you get any errors?

[dbuezas]

I think the issue with 5 minutes is that the card is demanding too much from HA and it starts responding with trash and the card kind of crashes. I can't reproduce it myself because though (I only have around 1 year of data, I'm kind of new in HA :) )

[dbuezas]

Please try v1.5.2, waiting for the download between button clicks.

[netweaver1970]

Just when opening my graph I already have the below erro. Might be unrelated as I have more graphs planned in this window, and it might be complaining about something related to that:

When flipping through the bigger time periods, I'm not getting any extra errors in my Dev tools. Also the auto graph refresh seems to flip the period to 5min, even when the display is - at that time - showing correct but not aggregated enoough data, eg. 12 months data with a 1hr period.

When deciding on the period to use in auto-mode, the code should 'round up to' the more aggregated period. Now I get for a 12 months time range: 365 days * 24 data points. That's far more than the 500 you aim for. A 12 months time range should most likely result in 365 data points, so a day period is better. Maybe your code currently 'rounds down' to the less aggregated period, blowing up the number of data points along the way.

Some clever code should look at the time window and intelligently decide what LTS period to use, based on the 'aimed for' number of data points, being the 'resolution' user wants. In your code it's still hardcoded to 500.

[dbuezas]

500 is the minimum resolution of the period. I'm not sure what you mean with "round up". try 1.5.2 please, I think 100 is a reasonable value

[dbuezas]

please click on Object so the error is visible. The second error is not this card

[netweaver1970]

I just tried 1.5.3, I still had to disable the 10s auto refresh because that (seemed to) force(d) the grouping period back to 5s every 10s, independently of my selected time range. Anyhow, for these types of graphs, auto refresh is not needed.

Generally, the data is returned is fast and stable but the grouping period auto determined compared to the time range requested is too short. Some screenshots below. Also, in the day grouping(averaging) charts (such as 12 and 36 months), I would expect to have rather flat MEAN values on consecutive days, while I'm still seeing rather spiky graphs. I guess that's mainly due to highly zoomed in nature of the Y-axis. Maybe that can be visually straightened out, using splines, as I was doing before LTS values :)

24 hr

3 days

7 days

14 days

1 month

3 months

12 months

36 months

[dbuezas]

I guess what you'd rather have is the ability to explicitly set the period depending on arbitrary date ranges, right? Something like

statistic: mean
period:
  3d: 5second
  7d: hour
  14d: day
  3M: week
  1Y: month

Regarding this

expect to have rather flat MEAN values on consecutive days

Maybe you can use the statistics card to confirm if the data just doesn't follow your expectations, or if there is something else going wrong.

[netweaver1970]

I guess what you'd rather have is the ability to explicitly set the period depending on arbitrary date ranges, right? Something like

statistic: mean
period:
  3d: 5second
  7d: hour
  14d: day
  3M: week
  1Y: month

yes, that would be a nice stop-gap solution for more individual customizing. As an alternative to the -perhaps for most people good enough- 'auto' setting. This proposed mapping is not possible in Grafana ... nice advantage :D

Regarding this

expect to have rather flat MEAN values on consecutive days

Maybe you can use the statistics card to confirm if the data just doesn't follow your expectations, or if there is something else going wrong.

Overall, figures look to have the right order of magnitude and delta direction, perhaps indeed this is reality, 'human feeling' averaging a full day temperatures might be a bad quality sensor, for comparison to a 'proper' sensor :D Maybe I should compare it to a government or commercially supplied daily average temperature graph, to see if I see the same (direction) variations.

Thanks for all the work so far !

[netweaver1970]

I guess what you'd rather have is the ability to explicitly set the period depending on arbitrary date ranges, right? Something like

statistic: mean
period:
  3d: 5second
  7d: hour
  14d: day
  3M: week
  1Y: month

Just thinking a bit further on this, the hardcoded mapping would be fine for static graph usages, but how would the manual scrolling/zooming around influence the period then? That's half of the fun in plotly. One could have all kinds of non-fixed time ranges, so a kind of auto/intelligent function is still needed in that case.

[dbuezas]

This proposed mapping is not possible in Grafana

😎 Believe it or no, I struggled a bit with the API of the custom resolution. It felt weird and somewhat obscure. This step-wise option is not harder and feels less like tweaking some internal value.

averaging a full day temperatures

Nitpick here: the median is not the average.

how would the manual scrolling/zooming around influence the period then?

It will switch as soon as you cross the barrier through zooming. Scrolling doesn't change the visible range and will therefore not change the period.

That's half of the fun in plotly

If not 99% of it! 😆

One could have all kinds of non-fixed time ranges, so a kind of auto/intelligent function is still needed in that case.

That I don't get. Can you expand?

[netweaver1970]

This proposed mapping is not possible in Grafana

😎 Believe it or no, I struggled a bit with the API of the custom resolution. It felt weird and somewhat obscure. This step-wise option is not harder and feels less like tweaking some internal value.

averaging a full day temperatures

Nitpick here: the median is not the average.

how would the manual scrolling/zooming around influence the period then?

It will switch as soon as you cross the barrier through zooming. Scrolling doesn't change the visible range and will therefore not change the period.

That's half of the fun in plotly

If not 99% of it! 😆

One could have all kinds of non-fixed time ranges, so a kind of auto/intelligent function is still needed in that case.

That I don't get. Can you expand?

ah, if the current time range is constantly been taking into account and compared to the fixed time range/period mapping, then it could work. In both directions. There will always be an edge point where it flips between periods, but that's inevitable. Comes with the flexibility of zooming around, I guess.

Another bug (?) I just saw, When doing lots of manipulations, selecting/unselecting, zooming etc, more often than not the non-selected sensors disappear from the legend and can only be retrieved again using a full graph reset

then a full 'reset':

[dbuezas]

https://github.com/dbuezas/lovelace-plotly-graph-card/tree/feat/step-function-auto-period feedback please! :)

type: custom:plotly-graph-dev
entities:
  - entity: sensor.garden_temperature
    period:
      0s: 5minute
      1d: hour
      1M: day
      1y: month

[netweaver1970]

https://github.com/dbuezas/lovelace-plotly-graph-card/tree/feat/step-function-auto-period feedback please! :)

type: custom:plotly-graph-dev
entities:
  - entity: sensor.garden_temperature
    period:
      0s: 5minute
      1d: hour
      1M: day
      1y: month

Why is this on entity level? Shouldn't the period grouping be on the graph level? Because it makes little sense to do it on entity level, it's rather confusing to allow this on this granular level. On the X-axis only 1 period level can be shown anyway. Or if you really want to allow it on entity level, allow it to make a it default setting per graph.

[netweaver1970]

Also, when specifying the discrete mapping in YAML, the period grouping is not shown anymore. It would be nice if it was there, to validate the correct behaviour. If you don't want to put it there by default (eg. for space savings on the graph), make it at least selectable with an option.

Right now I can't really tell if the discrete grouping is indeed working, But by looking at the graphs I'm expecting it doesn't ...

24hr time range, data point labels per hour

1 month time range, data points still per hour:

1 year time frame, data points label per day but multiple data points shown for the SAME date, hence the zigzag. So not a single mean period grouping has been used:

I'm on HA 2022.11 Beta so I do have LTS week period available. my YAML:

entities:
  - entity: sensor.th_livingroom_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 1
    name: Livingroom
    yaxes: y1
    line:
      color: red
      shape: spline
  - entity: sensor.th_kitchen_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 2
    name: Kitchen
    yaxes: y1
    line:
      color: orange
      shape: spline
  - entity: sensor.th_garage_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 3
    name: Garage
    yaxes: y1
    line:
      color: green
      shape: spline
  - entity: sensor.buienradar_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 4
    visible: legendonly
    name: Outside
    yaxes: y2
    line:
      color: gray
      shape: spline
  - entity: sensor.th_office2_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 5
    name: Office
    yaxes: y1
    line:
      dash: dot
      shape: spline
  - entity: sensor.th_guest_temperature
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 6
    name: Guest
    yaxes: y1
    line:
      dash: dot
      shape: spline
      color: red
  - entity: sensor.th_master_br_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 7
    name: Master
    yaxes: y1
    line:
      dash: dot
      shape: spline
      color: orange
  - entity: sensor.th_alastair_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 8
    name: Alastair
    yaxes: y1
    line:
      dash: dot
      shape: spline
  - entity: sensor.th_bathroom_temperature
    statistic: mean
    period:
      24h: 5minute
      3d: 5minute
      7d: hour
      14d: hour
      1m: day
      3m: day
      12m: week
      36m: week
    legendgroup: 9
    name: Bathroom
    yaxes: y1
    line:
      dash: dot
      shape: spline
hours_to_show: 24
no_default_layout: true
layout:
  dragmode: pan
  margin:
    t: 30
    l: 50
    r: 30
    b: 50
  showlegend: true
  height: 650
  legend:
    x: 0
    'y': 1
    bgcolor: rgba(0,0,0,.4)
    orientation: h
    itemsizing: fixed
    font:
      size: 10
  grid:
    rows: 3
    columns: 1
    roworder: top to bottom
  yaxis:
    title:
      text: Temperature
  yaxis2:
    title:
      text: Humidity
  yaxis3:
    title:
      text: Abs. Humidity
  plot_bgcolor: black
  y1:
    gridcolor: whitesmoke
    zeroline: false
    fixedrange: true
  xaxis:
    rangeselector:
      'y': 1.02
      buttons:
        - count: 3
          step: day
        - count: 7
          step: day
        - count: 14
          step: day
        - count: 1
          step: month
        - count: 3
          step: month
        - count: 12
          step: month
        - count: 36
          step: month
defaults:
  entity:
    line: null
  width: 1
title: Climate

[netweaver1970]

Btw, I'm at the version 1.6.0 as I expect the offset coding shouldn't impact the LTS stats handling. Also, when zooming in/out, the grouping doesn't change at all. So it looks that the underlaying dataset used is always hour. Which would be consistent with the graph shapes shown.

[dbuezas]

Let's move the discussion out of this issue, this one is long resolved :)

Don't use 1.6 it has a lot issues. Weekly is not active yet in the card (it doesn't matter which HA you have. Defaults go to defaults/entity