filter_out sensor filter is filtering out too many values on certain conditions

[malteger]

The problem

The filter_out filter for the sensor component is filtering out values unexpectedly on certain occasions. The filter uses the accuracy_decimals property of the sensor and uses this to compare the values (see source). This causes problems if this filter is used before an e.g. multiply filter, as filter_out: 0.0 will filter out all values below 0.5 if accuracy_decimals is 0 as the filter multiplies the incoming values by 10^accuracy_decimals and then rounds to the nearest integer for comparison (Which is also not documented).

The documentation for accuracy_decimals says that the setting will only affect the reported values, not the raw values processed by the filters.

Manually set the number of decimals to use when reporting values. This does not impact the actual value reported to Home Assistant, it just sets the number of decimals to use when displaying it.

So using

accuracy_decimals: 0
filters:
  - filter_out: 0.0
  - multiply: 100.0

for any sensors which report non integer values will result in unexpected behaviour.

I encountered this for a water sensor, where I measure the distance from a reference point above the water to the surface using the ultrasonic component which reports a value in meters, filter out too short distances and then convert the value to the actual volume in the container. (See snippet below, but the issue only became clear while looking at the source)

Which version of ESPHome has the issue?

2022.8.3

What type of installation are you using?

pip

Which version of Home Assistant has the issue?

No response

What platform are you using?

ESP8266

Board

No response

Component causing the issue

sensor

Example YAML snippet

sensor:
  - platform: ultrasonic
    trigger_pin: D6
    echo_pin: D5
    name: capacity
    unit_of_measurement: l
    accuracy_decimals: 0
    filters:
      - filter_out: 0.00
      - multiply: -1.0
      - offset: 1.36
      - multiply: 2702.7

Anything in the logs that might be useful for us?

No response

Additional information

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[malteger]

As there has been no change in the relevant code, this issue still persist

[ssieb]

You didn't provide any logs. The accuracy decimals only affects the displayed value on HA, it doesn't affect the filtered value. Enable verbose logs for more details.

[malteger]

The accuracy decimals only affects the displayed value on HA, it doesn't affect the filtered value.

@ssieb This is not correct (MWE and logs below), as the linked code is clearly using accuracy_decimals as also explained in the original issue content:

    int8_t accuracy = this->parent_->get_accuracy_decimals();
    float accuracy_mult = powf(10.0f, accuracy);
    float rounded_filter_out = roundf(accuracy_mult * this->value_to_filter_out_);
    float rounded_value = roundf(accuracy_mult * value);
    if (rounded_filter_out == rounded_value) {
      return {};
    } else {
      return value;
    }

MWE yaml:

esphome:
  platform: ESP32
  board: nodemcu-32s
  name: test-bug

logger:
  level: VERY_VERBOSE

sensor:
  - platform: template
    name: 'Test Sensor'
    lambda: return 0.04;
    update_interval: 10s
    accuracy_decimals: 0
    filters:
      - filter_out: 0.00
      - multiply: 100.0

results in the following log:

INFO Successfully uploaded program.
INFO Starting log output from /dev/ttyUSB0 with baud rate 115200
[14:46:36][     6][D][esp32-hal-cpu.c:244] setCpuFrequencyMhz(): PLL: 480 / 2 = 240 Mhz, APB: 80000000 Hz
[14:46:36][I][logger:258]: Log initialized
[14:46:36][VV][sensor:093]: Sensor(0x3ffb9478)::add_filter(0x3ffb956c)
[14:46:36][VV][sensor.filter:029]: Filter(0x3ffb956c)::initialize(parent=0x3ffb9478 next=0x0)
[14:46:36][VV][sensor:093]: Sensor(0x3ffb9478)::add_filter(0x3ffb958c)
[14:46:36][VV][sensor.filter:029]: Filter(0x3ffb956c)::initialize(parent=0x3ffb9478 next=0x3ffb958c)
[14:46:36][VV][sensor.filter:029]: Filter(0x3ffb958c)::initialize(parent=0x3ffb9478 next=0x0)
[14:46:36][I][app:029]: Running through setup()...
[14:46:36][V][app:030]: Sorting components by setup priority...
[14:46:36][VV][scheduler:057]: set_interval(name='', interval=60000, offset=1907)
[14:46:36][VV][scheduler:057]: set_interval(name='update', interval=10000, offset=4087)
[14:46:36][I][app:062]: setup() finished successfully!
[14:46:36][VV][scheduler:196]: Running interval 'update' with interval=10000 last_execution=4294953304 (now=105)
[14:46:36][V][sensor:076]: 'Test Sensor': Received new state 0.040000
[14:46:36][VV][sensor.filter:014]: Filter(0x3ffb956c)::input(0.040000)
[14:46:36][VV][scheduler:196]: Running interval '' with interval=60000 last_execution=4294905473 (now=105)
[14:46:36][I][app:102]: ESPHome version 2022.12.3 compiled on Jan 23 2023, 14:46:24
[14:46:36][C][logger:293]: Logger:
[14:46:36][C][logger:294]:   Level: VERY_VERBOSE
[14:46:36][C][logger:295]:   Log Baud Rate: 115200
[14:46:36][C][logger:296]:   Hardware UART: UART0
[14:46:36][C][template.sensor:023]: Template Sensor 'Test Sensor'
[14:46:36][C][template.sensor:023]:   State Class: ''
[14:46:36][C][template.sensor:023]:   Unit of Measurement: ''
[14:46:36][C][template.sensor:023]:   Accuracy Decimals: 0
[14:46:36][C][template.sensor:024]:   Update Interval: 10.0s
[14:46:42][VV][scheduler:196]: Running interval 'update' with interval=10000 last_execution=4294963304 (now=6011)
[14:46:42][V][sensor:076]: 'Test Sensor': Received new state 0.040000
[14:46:42][VV][sensor.filter:014]: Filter(0x3ffb956c)::input(0.040000)
[14:46:52][VV][scheduler:196]: Running interval 'update' with interval=10000 last_execution=6008 (now=16008)
[14:46:52][V][sensor:076]: 'Test Sensor': Received new state 0.040000
[14:46:52][VV][sensor.filter:014]: Filter(0x3ffb956c)::input(0.040000)

as you can see the value reaches the filter and that's it. If accuracy_decimals is changed to 2 the expected value 4 is reported.

[ssieb]

Oh, I see. I checked the code, but I didn't look at that specific filter. It is using that setting... https://github.com/esphome/issues/issues/741 Use a lambda filter to filter out the specific value the way you want.

[malteger]

Yes, the lambda filter is the fallback I am using.

The issue here is twofold: The documentation for the accuracy decimals is currently not correct, as it is used somewhere. It might also be a good idea to change the behaviour of the filter and remove the usage of accuracy decimals.