Gaps in data when scrolling

[mwolter805]

Discussed in https://github.com/dbuezas/lovelace-plotly-graph-card/discussions/314

Using the following versions: lovelace-plotly-graph-card: 3.3.3 Home Assistant: 2023.9.3

^{Originally posted by **mwolter805** September 29, 2023} Hello, Great integration, thanks for your work on this! I've used plotly in the past and was excited to see it available for home assistant. I'm attempting to create a bar chart for energy usage of several circuits in my house. I think I've cobbled it together well enough that it displays the info, however, there are several gaps in the data scrolling. I've searched through the discussions and haven't found anything related to this. It appears the first period for the chart is lost and when scrolling it is never obtained. I've seen reference in other discussions regarding HA not data for the first period. Is there a way to fill the gaps filled? Thanks again for all your work on this. In the animation below, I scroll to an earlier time period, then click the 1 day button and the chart is missing bars for 7am, 4am and 1am  However, if I were to click the button for 1d directly, this data for 4am and 1am appears. Still missing 7am though.  ```yaml type: custom:plotly-graph hours_to_show: 6h entities: - entity: sensor.emporia_vue_2_1_circuit_1_energy visible: legendonly - entity: sensor.emporia_vue_2_1_circuit_2_energy - entity: sensor.emporia_vue_2_1_circuit_3_energy - entity: sensor.emporia_vue_2_1_circuit_4_energy - entity: sensor.emporia_vue_2_1_circuit_5_energy - entity: sensor.emporia_vue_2_1_circuit_6_energy - entity: sensor.emporia_vue_2_1_circuit_7_energy - entity: sensor.emporia_vue_2_1_circuit_8_energy - entity: sensor.emporia_vue_2_1_circuit_9_energy - entity: sensor.emporia_vue_2_1_circuit_10_energy - entity: sensor.emporia_vue_2_1_circuit_11_energy visible: legendonly - entity: sensor.emporia_vue_2_1_circuit_12_energy - entity: sensor.emporia_vue_2_1_circuit_13_energy - entity: sensor.emporia_vue_2_1_circuit_14_energy visible: legendonly fn: | $fn({getFromConfig, vars})=> { const range = getFromConfig("visible_range"); const width = range[1] - range[0]; vars.scroll = (label, p) => ({ args: [ { layout: { "xaxis.range": [range[0] + width*p, range[1] + width*p], } }, { transition: { duration: 150, } } ], label, method: "animate", }) vars.zoom = (label, h) => ({ args: [ { layout: { "xaxis.range": [Date.now()-1000*60*60*h, Date.now()], } } ], label, method: "animate", }) } layout: height: 400 barmode: group xaxis: autorange: true updatemenus: - buttons: - $fn({vars}) => vars.scroll( '<', -.5) - $fn({vars}) => vars.scroll( '>', .5) direction: right active: -1 pad: r: 10 t: 10 type: buttons x: 1 xanchor: right 'y': 1 yanchor: top - buttons: - $fn({vars}) => vars.zoom( '1y', 24*365) - $fn({vars}) => vars.zoom( '1m', 24*30) - $fn({vars}) => vars.zoom( '1w', 24*7) - $fn({vars}) => vars.zoom( '1d', 24) - $fn({vars}) => vars.zoom( '1h', 1) direction: right active: -1 pad: r: 10 t: 10 type: buttons x: 0 xanchor: left 'y': 1 yanchor: top defaults: entity: statistic: state filters: - fn: |- (params) => { const ys = []; ys.push(0); for (let i = 1; i < params.statistics.length; i++){ if (params.statistics[i-1].state > params.statistics[i].state) { ys.push(params.statistics[i].state); } else { ys.push(params.statistics[i].state-params.statistics[i-1].state); } }; return { ys }; } - filter: i>0 type: bar width: $fn () => 60*60*60*1 period: 6h: hour 7d: day 6M: month yaxes: fixedrange: true autorange_after_scroll: true config: scrollZoom: false doubleClickDelay: 600 ```

[CendaL]

I see the same error (HA 2023.10.5).

It looks to me like 1-off bug because when I have a graph displaying last 18 hours, in debugger I see only 17 values were loaded (which may be by design - I don't know).

Looking at websocket communication:

{"type":"recorder/statistics_during_period","start_time":"2023-10-27T22:24:30.823Z","end_time":"2023-10-28T16:24:30.824Z","statistic_ids":["sensor.solary_tracker_on"],"period":"hour","id":35}

returns 17 records:

{"id":35,"type":"result","success":true,"result":{"sensor.solary_tracker_on":[{"start":1698447600000,"end":1698451200000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698451200000,"end":1698454800000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698454800000,"end":1698458400000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698458400000,"end":1698462000000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698462000000,"end":1698465600000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698465600000,"end":1698469200000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698469200000,"end":1698472800000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698472800000,"end":1698476400000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698476400000,"end":1698480000000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698480000000,"end":1698483600000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698483600000,"end":1698487200000,"last_reset":null,"state":10.25,"sum":2164.25,"change":10.25},{"start":1698487200000,"end":1698490800000,"last_reset":null,"state":48.5,"sum":2212.75,"change":48.5},{"start":1698490800000,"end":1698494400000,"last_reset":null,"state":22.0,"sum":2234.75,"change":22.0},{"start":1698494400000,"end":1698498000000,"last_reset":null,"state":0.0,"sum":2234.75,"change":0.0},{"start":1698498000000,"end":1698501600000,"last_reset":null,"state":9.24,"sum":2243.99,"change":9.239999999999782},{"start":1698501600000,"end":1698505200000,"last_reset":null,"state":0.0,"sum":2243.99,"change":0.0},{"start":1698505200000,"end":1698508800000,"last_reset":null,"state":0.0,"sum":2243.99,"change":0.0}]}}

Moving the graph a little:

{"type":"recorder/statistics_during_period","start_time":"2023-10-27T20:22:54.151Z","end_time":"2023-10-27T22:21:06.576Z","statistic_ids":["sensor.solary_tracker_on"],"period":"hour","id":36}

results:

{"id":36,"type":"result","success":true,"result":{"sensor.solary_tracker_on":[{"start":1698440400000,"end":1698444000000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0},{"start":1698444000000,"end":1698447600000,"last_reset":null,"state":0.0,"sum":2154.0,"change":0.0}]}}

The graph does not show 1:00 my local time but the data is there (1698447600000 epoch time; this is after move)

[dbuezas]

Thanks for the in depth analysis. So this means HA is sending the data but this card isn't considering it, correct?

[dbuezas]

Home assistant probably stopped sending a fake first datapoint, so this line is causing issues: https://github.com/dbuezas/lovelace-plotly-graph-card/blob/ef1834ac403a30e0d0495e3dfa552315c71c2dd5/src/cache/Cache.ts#L123

[CendaL]

I would guess this is the change in HA: https://github.com/home-assistant/core/pull/92823

[CendaL]

I've removed fake boundary handling in my local build it looks good to me. I will try to issue PR with proper code change later.

[dbuezas]

Oh cool! One thing you can do to confirm that this works correctly:

refresh_interval: 1s
entities:
  - entity: sensor.some_sensor_that_updates_frequently
     line:
       shape: linear

The result should be a plot that doesn't go in steps, but that connects the dots with direct lines. If it goes in steps, it means that the algorithm is adding faux datapoints.

[CendaL]

It does not work correctly for me with refresh_interval: 1s - the graph does not auto-scroll anymore if I use it. Without it, it scrolls fine. In both cases, lines look fine to me:

type: custom:plotly-graph
entities:
  - entity: sensor.main_power_channel_a_power
    line:
      shape: linear
  - entity: sensor.main_power_channel_b_power
    line:
      shape: linear
  - entity: sensor.main_power_channel_c_power
    line:
      shape: linear
hours_to_show: 0.1

[dbuezas]

This means that there is a problem with fetching. Removing update interval is equivalent to setting it to auto, which grabs datapoints as they come (and skips fetching). If you scroll you are now probably losing datapoints

[CendaL]

It works - the bug was in specifying refresh interval - it must be just refresh_interval: 1 - not refresh_interval: 1s :)

[dbuezas]

Ohhhh you are right!

If you:

• get no lost datapoints,
• and no "invented" datapoints in-between
• And scrolling works fine

then you fixed it 🎉

[CendaL]

Well, maybe but I am not sure - I've just removed all fake data point handling. I want to check what data is fetched - whether we still need to fetch a little bit more or not (as it is commented in Cache.ts), then I will issue a PR.

[dbuezas]

Here's an idea to test it:

• Create a number sensor that changes on every second (either a template sensor or a helper + automation)
• Create a plot using:

entities:
  - entity: number.your_template_sensor
       line:
         shape: linear
       mode: lines+markers
       filters:
         - fn: console.log # see https://github.com/dbuezas/lovelace-plotly-graph-card#debugging

refresh_interval: 0.1
hours_to_show: 30s

Then:

• Let it run for a couple of minutes
• Open the plot and check that there is exactly a marker per second and that scrolling loads data correctly without gaps or duplications.
• And look at the data outputted in the console to check that there are no duplicates

[CendaL]

It does not work well so far:

More debugging:

api/history/period/2023-11-14T07:55:56.642Z?filter_entity_id=sensor.seconds_time_tracker&significant_changes_only=0&no_attributes&&minimal_response&&end_time=2023-11-14T07:56:01.643Z

[
    [
        {
            "entity_id": "sensor.seconds_time_tracker",
            "state": "1699948501.1",
            "attributes": {},
            "last_changed": "2023-11-14T07:55:56.642000+00:00",
            "last_updated": "2023-11-14T07:55:56.642000+00:00"
        }
    ]
]

api/history/period/2023-11-14T07:55:57.133Z?filter_entity_id=sensor.seconds_time_tracker&significant_changes_only=0&no_attributes&&minimal_response&&end_time=2023-11-14T07:56:02.134Z

[
    [
        {
            "entity_id": "sensor.seconds_time_tracker",
            "state": "1699948501.1",
            "attributes": {},
            "last_changed": "2023-11-14T07:55:57.133000+00:00",
            "last_updated": "2023-11-14T07:55:57.133000+00:00"
        }
    ]
]

Which shows last_changed and last_updated is changing.

For the first returned record only which is actually the oldest record for the two...:

api/history/period/2023-11-14T07:55:57.581Z?filter_entity_id=sensor.seconds_time_tracker&significant_changes_only=0&no_attributes&&minimal_response&&end_time=2023-11-14T07:56:02.582Z

[
    [
        {
            "entity_id": "sensor.seconds_time_tracker",
            "state": "1699948501.1",
            "attributes": {},
            "last_changed": "2023-11-14T07:55:57.581000+00:00",
            "last_updated": "2023-11-14T07:55:57.581000+00:00"
        },
        {
            "state": "1699948561.1",
            "last_changed": "2023-11-14T07:56:01.141916+00:00"
        }
    ]
]

api/history/period/2023-11-14T07:55:58.478Z?filter_entity_id=sensor.seconds_time_tracker&significant_changes_only=0&no_attributes&&minimal_response&&end_time=2023-11-14T07:56:03.479Z

[
    [
        {
            "entity_id": "sensor.seconds_time_tracker",
            "state": "1699948501.1",
            "attributes": {},
            "last_changed": "2023-11-14T07:55:58.478000+00:00",
            "last_updated": "2023-11-14T07:55:58.478000+00:00"
        },
        {
            "state": "1699948561.1",
            "last_changed": "2023-11-14T07:56:01.141916+00:00"
        }
    ]
]

[CendaL]

It looks to me that the issue is only for statistic data so I've create PR #325 to fix it.

[mwolter805]

Hi @dbuezas, I see this issue is completed. Appears this fix is not in the latest release. Is there a way I can test it?

[dbuezas]

I'll make a release now. Give it a couple of minutes and update the card from hacs