Incorrect daily max for 'today' sensors when using long term statistics

[jata1]

I use long term statistics (max) to chart daily rainfall from the daily rain sensor over an extended period (e.g. last 30 days). If the max rainfall for the previous day is greater than the current day, then the statistics returns the max from the previous day. So I think the statistic is resetting just after midnight and this creates a 'erroneous statistic' for today's max.

create a chart using apex charts with 2 series based on the daily rain sensor. for one series, use group_by day max for the other use statistics by day max look for days/pattern where the there is a different max for a day

here is a my chart yaml and screenshot to make it easier to reproduce

type: custom:apexcharts-card
apex_config:
  xaxis:
    labels:
      format: ddd dd/MM
      trim: false
      offsetY: -5
      style:
        fontWeight: 400
        fontSize: 10
  legend:
    show: true
    labels:
      style:
        fontWeight: 400
        fontSize: 10
all_series_config:
  stroke_width: 1
  opacity: 0.1
  show:
    offset_in_name: false
    legend_value: false
    extremas: false
    name_in_header: false
    datalabels: false
graph_span: 20d
span:
  end: day
header:
  show: true
  title: Rainfall History
  show_states: false
  colorize_states: true
yaxis:
  - id: rain
    min: 0
    max: ~120
    decimals: 0
    apex_config:
      tickAmount: 6
      title:
        rotate: -90
        text: Rain (mm)
        style:
          fontWeight: 400
          fontSize: 9
    show: true
    opposite: false
series:
  - entity: sensor.undercliffe_sensors_precipitation_today
    name: Rain (group_by)
    color: blue
    opacity: 0.4
    yaxis_id: rain
    type: column
    group_by:
      func: max
      duration: 1d
  - entity: sensor.undercliffe_sensors_precipitation_today
    name: Rain (stats)
    color: grey
    opacity: 0.4
    yaxis_id: rain
    type: column
    statistics:
      type: max
      period: day
      align: start

This is only an issue when you want to plot this data over an extended period - where LTS are needed.

If you didn't notice from my chart, there has been a 'rain event' here in sydney over the last couple of days!

[briis]

I am not sure how I can help. Long Term statistics are handled by Home Assistant, and this integration just reports the raw data to Home Assistant. The rain data coming from WeatherFlow are reset by WeatherFlow at midnight, so it is not anything I do.

So could you elaborate on what you would like this integration to provide, that it doesn't do now?

[jata1]

Understood. I just wasn't sure how the integration worked in terms of LTS. I agree it is an underlying issue with HA as I have had a similar issue with a sensor I created using the history_stats platform.

I have a work-a-round for this issue using a utility meter helper to create a rainfall daily total myself. The utility meter works fine. Not sure if there is something to explore there in terms of how your integration handles daily totals as at the moment I have two sensors doing the same thing but one works (utility meter) properly for what I need.

Thought I would report it here in case other folks have a similar issue.

[briis]

I might look at changing the STATE CLASS from measurement to total_increasing. Here is the description of the total_increasing:

Similar to total, with the restriction that the state represents a monotonically increasing positive total which periodically restarts counting from 0, e.g. a daily amount of consumed gas, weekly water consumption or lifetime energy consumption. Statistics of the accumulated growth of the sensor's value since it was first added is updated every 5 minutes. A decreasing value is interpreted as the start of a new meter cycle or the replacement of the meter.

It is not available for all device classes, so I will need to test it.

[briis]

I have now made the change for Precipitation last hour and Precipitation today. I get no errors from Home Assistant, but as it is not raining for the next few days, it is hard to test for me. If I released this as a beta, would you be willing to test it?

[jata1]

Great stuff. Thanks. There is some rain forecast over the next few days here and I am happy to test the beta.

[briis]

I have now released V1.0.9-test. You need to show beta releases in HACS to get it. Try and install it, and let it run for few days, and see how this affects your charting.

[jata1]

Thanks. I will try setup the beta and watch for a few days. I have a number of test sensors running trying different ideas so I hope to find a good solution. I will report back soon.

[jata1]

I think something is going wrong with the test build changes. charting when using group_by show data correctly. In all my testing, I can't seem to get any data to display using LTS (statistics). I will need to do more testing but something is not right.

I am also having issues using a utility meter to calculate daily use on the core weatherflow integration so the issue might be related to the total_increasing state class. I will continue to test and provide updates over the next few days.

[jata1]

I have made progress. It seems with the change of the state class to total_increasing impacts on how you use LTS in apex-charts. I'm not sure if this is just apex-charts or more generally with LTS.

So if you configure statistics in the chart to use the 'state' rather than 'max' - it works correctly - see chart below. Note that there are no erroneous rain data the day after it rained. This is a good indicator that this change has resolved the issue.

The main impact/consideration of this change is that updating to V1.0.9-test changes the state class and historic LTS data from earlier than the update is not displayed using statistics (either max or state). So this will impact users (breaking change I suppose) of the integration that do charting on LTS

Note: The chart below has the following sensors/config. Some were setup my me while it was raining so that is why they don't all report the same total for the day value. When I get more rain I will let you know the results.

WFF (group_by) = WeatherFlow Forecast Precipitation using group by max WFF (stats) = WeatherFlow Forecast Precipitation using statistics on 'state' WFF-UM (stats) = Utility Meter on WeatherFlow Forecast Precipitation using statistics on 'state' (now a duplicate of WFF stats) WF-UM (group_by) = Utility Meter on core WeatherFlow Precipitation using group by max WF-UM (stats) = Utility Meter on core WeatherFlow Precipitation using statistics on 'state'

[jata1]

I can confirm the change of the state class impacts on LTS generally. I tested by using the core/stock statistics-graph card. The statistics available are sum, state and change (mean, min and max are not available)

[briis]

So it changes things, but are the changes working for you? Or should I reverse it?

[jata1]

I think it is working correctly with the change. I need more time to test as haven’t had rain for a while. Rain forecast later this week so I will update you in a few days

[briis]

No worries. Let me know when you have a final conclusion.

[jata1]

Hey @briis - I have had rain and this is now working correctly. Thanks for helping with this.

Are you planning to make the change into prod/release soon or do I keep using the test/beta?

[briis]

Very good. I will release this as production later today, so just upgrade when you see it in HACS.