Rest sensor not updating (non-numeric value: 'None')

[flo-wer]

The problem

The rest sensor is still raising the non-numeric value: 'None' (<class 'str'>) error despite using the availability config to filter any non-numeric values (see the is_number condition). I included example API responses for both cases. In the not available case, the availability condition evaluates to false and the value_template equals null/None. This produces the below exception.

What version of Home Assistant Core has the issue?

core-2024.10

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

rest

Link to integration documentation on our website

https://www.home-assistant.io/integrations/sensor.rest/

Diagnostics information

No response

Example YAML snippet

rest:
  - resource: http://192.168.1.110:3001/api/status-page/heartbeat/default
    scan_interval: 30
    sensor:
      - unique_id: rest_uptime_ping1
        name: uptime_ping1
        value_template: '{% set v = value_json.heartbeatList["1"][-1] %}{{ v.ping }}'
        availability: '{% set v = value_json.heartbeatList["1"][-1] %}{{ v.status == 1 and is_number(v.ping) }}'
        unit_of_measurement: 'ms'
        state_class: measurement

Anything in the logs that might be useful for us?

2023-09-08 18:27:10.124 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/components/sensor/__init__.py", line 591, in state
    numerical_value = float(value)  # type:ignore[arg-type]
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: 'None'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 235, in _handle_refresh_interval
    await self._async_refresh(log_failures=True, scheduled=True)
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 392, in _async_refresh
    self.async_update_listeners()
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 173, in async_update_listeners
    update_callback()
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/components/rest/entity.py", line 51, in _handle_coordinator_update
    self._update_from_rest_data()
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/components/rest/sensor.py", line 180, in _update_from_rest_data
    self._process_manual_data(raw_value)
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/trigger_template_entity.py", line 242, in _process_manual_data
    self.async_write_ha_state()
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 730, in async_write_ha_state
    self._async_write_ha_state()
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 830, in _async_write_ha_state
    state, attr = self._async_generate_attributes()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 771, in _async_generate_attributes
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/entity.py", line 736, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/components/sensor/__init__.py", line 593, in state
    raise ValueError(
ValueError: Sensor sensor.uptime_ping1 has device class 'None', state class 'measurement' unit 'ms' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'None' (<class 'str'>)

Additional information

REST API response available:

{
  "heartbeatList": {
    "1": [
      {
        "status": 1,
        "ping": 21.4
      }
    ]
  }
}

not available:

{
  "heartbeatList": {
    "1": [
      {
        "status": 0,
        "ping": null
      }
    ]
  }
}

[home-assistant[bot]]

rest documentation rest source

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[flo-wer]

This is still an issue for 2023.12

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[flo-wer]

This is still an issue for 2024.3

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[timr49]

I tested this with 2024.6.0 and it is still there. Based on some debug tracing and code inspection, I believe the cause is as follows.

In class ManualTriggerEntity the method _process_manual_data calls async_write_ha_state and later calls method_render_templates in superclass TriggerBaseEntity, which populates _rendered. But the ValueError exception is raised during _stringify_state as called indirectly from async_write_ha_state so the call to _render_templates never happens. The value of _rendered[CONF_AVAILABILITY] is initially None and it stays that way.

The available property of class ManualTriggerEntity treats None as True:

@property
def available(self) -> bool:
      """Return availability of the entity."""
      return (
          self._rendered is not self._static_rendered
          and
          # Check against False so `None` is ok
          self._rendered.get(CONF_AVAILABILITY) is not False
      )

So, the availability template never gets rendered and the availability continues to be judged on the basis of the original None, which means True and the exception keeps getting raised each time there is new data.

Someone with more knowledge of this or other integrations may be able to comment on the preferred way of ensuring that availability is rendered before it is tested as a precondition to populating the state (or even on whether basing availability templates on the value data is supposed to be supported).

[symposion]

I'm having this exact problem and it is significantly impacting the usefulness of HA for me. I use a large array of REST sensors that report the values of devices that aren't always fully reliable. I used to be able to return "null" or some other error value for numeric sensors for which a value wasn't available. Now, AFAICT, it is basically impossible to handle any kind of error with a numeric sensor. Worse, it silently maintains the original value in the UI and in the graphing of the value. This is just wrong. If there's no value, there's no value, and I want that reflected in the UI and in the history of the sensor. Even worse, the way the exception is raised means that if any single sensor in a rest integration fails, none of the values get updated for any of the sensors.

So far, I haven't even found a credible way to work around this. Are there any HA devs watching these issues? It seems like this issue is just being ignored, which is quite upsetting as it has a pretty serious impact for me.

[1mfaasj]

Same issue here: https://community.home-assistant.io/t/rest-sensor-ignores-the-availability-line-in-the-yaml/753237/4

The rest sensor is not respecting its availability ...

[timr49]

So far, I haven't even found a credible way to work around this. Are there any HA devs watching these issues? It seems like this issue is just being ignored, which is quite upsetting as it has a pretty serious impact for me.

Hi @symposion,

One reason it may not be getting attention is because it is not just about RESTful sensors: evaluating the state before evaluating the availability template is common to other integrations (see my comment above).

Not sure what counts as "credible", but one workaround would be to have a template sensor for each of your numeric sensors so that the RESTful sensors are always strings and the template sensors would assess availability and avoid invalid numeric values.

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[flo-wer]

This is still an issue for 2024.10