different output rest sensors using identical resource

[Mariusthvdb]

The problem

having a rest: sensor on my HA resource and components attribute, results in 49 components listed. The exact same resource as sensor platform: rest configured results in over 227.... all other attributes (unit_system and config_dir) are identical.

Given the fact no exceptions or other differences are documented, this can only be an issue?

What version of Home Assistant Core has the issue?

2023.5.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

rest

Link to integration documentation on our website

https://www.home-assistant.io/integrations/sensor.rest/ https://www.home-assistant.io/integrations/rest/

Diagnostics information

No response

Example YAML snippet

rest:

  - resource: !secret config_resource
    scan_interval: 86400
    verify_ssl: false
    headers:
      Content-Type: application/json
      Authorization: !secret api_bearer_token

    sensor:
      - unique_id: ha_main_config_rest
        name: Ha Main config
        value_template: >
          {{value_json.version}}
        json_attributes:
          - components
          - unit_system
          - config_dir
sensor:

  - platform: rest
    unique_id: ha_main_config_sensor_platform_rest
    name: Ha Main config
    resource: !secret config_resource
    value_template: >
      {{value_json.version}}
    verify_ssl: false
    json_attributes:
      - components
      - unit_system
      - config_dir
    headers:
      Content-Type: application/json
      Authorization: !secret api_bearer_token
    scan_interval: 86400

Anything in the logs that might be useful for us?

No response

Additional information

[home-assistant[bot]]

Hey there @epenet, mind taking a look at this issue as it has been labeled with an integration (rest) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `rest` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign rest` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

rest documentation rest source _{^{(message by IssueLinks)}}

[epenet]

Did you enable debug to see if there are differences in the requests or in the responses?

[Mariusthvdb]

no I havent (yet).

should I set debug on homeassistant.components.rest: debug only for both?

btw, it seems we can not hot reload these configs? I always need to restart for changes in the yaml to take effect (like now, I just added icons and it will only kick in on restart)

[Mariusthvdb]

debug output:

023-05-24 11:39:09.473 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched from resource: {"latitude":lat,"longitude":long,"elevation":5,"unit_system":{"length":"km","accumulated_precipitation":"mm","mass":"g","pressure":"Pa","temperature":"°C","volume":"L","wind_speed":"m/s"},"location_name":"Home","time_zone":"Europe/Amsterdam","components":["hue","scene.homeassistant","auth","sensor.sun2","sensor.nmbs","lock","input_number","system_log","stookwijzer","energy","diagnostics","sensor.integration","sensor.energy","season","sensor.uptime","notify.mobile_app","input_button","my","lovelace","zha","switch.wake_on_lan","homeassistant_alerts","browser_mod","tradfri","weather","group","frontend","wake_on_lan","plex","logger","usb","trace","sensor.statistics","generic","panasonic_viera","cover.group","xbox","device_tracker","analytics","sensor.time_date","spook","gdacs","filesize","moon","climate","network","sensor.rest","template","sun","panel_custom","utility_meter","binary_sensor.sun2","solaredge","tts.cloud","remote","mill","light.switch_as_x","binary_sensor.template","esphome","dlna_dms","airvisual_pro","recorder","workday","cover.template","water_heater","buienradar","ping","fan.mqtt","sensor.zodiac","sensor.filesize","websocket_api","notify.group","stookalert","select.mqtt","update.mqtt","spotify","notify.tts","humidifier","persistent_notification","custom_icons","siren","application_credentials","person","easyenergy","samsungtv","sensor.utility_meter","sensor.sun","homeassistant","local_ip","energyzero","watchman","luftdaten","integration","feedreader","version","update","timer","ssdp","camera","upnp","airvisual","image_upload","forecast_solar","synology_dsm","binary_sensor.mqtt","switch.group","overkiz","camera.generic","shell_command","light.mqtt","blueprint","input_select","sensor.min_max","stt","scene","alarm_control_panel.mqtt","here_travel_time","light","cloud","threshold","switch","fan.switch_as_x","media_source","number","media_player","sensor.stookwijzer","powercalc","conversation","counter","binary_sensor","history","sensor.mqtt","github","input_text","assist_pipeline","binary_sensor.threshold","switch_as_x","alarm_control_panel","input_boolean","sensor.group","search","camera.mqtt","input_datetime","philips_js","vacuum","api","dlna_dmr","mqtt","config","alarm_control_panel.manual","stream","automation","min_max","switch.mqtt","unifiprotect","hassio","binary_sensor.hassio","co2signal","cover.mqtt","dhcp","switch.template","tag","binary_sensor.cloud","http","repairs","cpuspeed","cover","binary_sensor.group","fan","androidtv_remote","zeroconf","sensor.moon","rest","speedtestdotnet","plugwise","sensor.file","hacs","sensor.hassio","alert","openweathermap","rest_command","media_player.group","shelly","geo_location","sensor.command_line","unifi","button","python_script","sensor","apple_tv","sensor.systemmonitor","webhook","tts","entsoe","binary_sensor.meteoalarm","knmi","openuv","device_automation","sensor.afvalwijzer","notify.google_assistant_sdk","tts.google_translate","select","sensor.template","google_assistant_sdk","sensor.nederlandse_spoorwegen","profiler","ipp","light.group","zwave_js","zodiac","notify.file","mobile_app","tomorrowio","update.hassio","number.mqtt","sensor.mold_indicator","logbook","bluetooth","onboarding","sensor.season","sensor.local_ip","zone","hardware","uptime","text","file_upload","system_health","select.template","notify"],"config_dir":"/config","whitelist_external_dirs":["/config","/config/www","/media"],"allowlist_external_dirs":["/config","/config/www","/media"],"allowlist_external_urls":[],"version":"2023.5.4","config_source":"yaml","safe_mode":false,"state":"NOT_RUNNING","external_url":null,"internal_url":null,"currency":"EUR","country":"NL","language":"nl"}

2023-05-24 11:39:09.495 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched from resource: {"latitude":lat,"longitude":long,"elevation":5,"unit_system":{"length":"km","accumulated_precipitation":"mm","mass":"g","pressure":"Pa","temperature":"°C","volume":"L","wind_speed":"m/s"},"location_name":"Home","time_zone":"Europe/Amsterdam","components":["zeroconf","auth","sensor.sun2","recorder","stt","system_log","websocket_api","http","light","cloud","diagnostics","switch","persistent_notification","lovelace","sensor","person","webhook","tts","group","binary_sensor.meteoalarm","frontend","device_automation","binary_sensor","logger","usb","homeassistant","tts.google_translate","select","analytics","search","sensor.time_date","update","api","network","config","update.hassio","template","ssdp","onboarding","utility_meter","binary_sensor.sun2","hassio","binary_sensor.hassio","tts.cloud","image_upload","dhcp","file_upload","binary_sensor.cloud","repairs","cover"],"config_dir":"/config","whitelist_external_dirs":["/config","/config/www","/media"],"allowlist_external_dirs":["/config","/config/www","/media"],"allowlist_external_urls":[],"version":"2023.5.4","config_source":"yaml","safe_mode":false,"state":"NOT_RUNNING","external_url":null,"internal_url":null,"currency":"EUR","country":"NL","language":"nl"}

[epenet]

So the issue is not in the parsing of the data.

Now you need to analyse the details of the requests, you may need to enable httpx logging:

homeassistant.components.rest: debug
httpx: debug
httpcore: debug

For example, it could be that some headers are not going through correctly.

[Mariusthvdb]

will do. do note that besides the much longer list of components in the top logging, the bottom logging also shows several extra attributes. That is, on the log, it does Not show these in the arttributes of the entity in dev tools/state

`"whitelist_external_dirs":["/config","/config/www","/media"],"allowlist_external_dirs":["/config","/config/www","/media"],"allowlist_external_urls":[],"config_source":"yaml","safe_mode":false,"state":"NOT_RUNNING","external_url":null,"internal_url":null,`

[Mariusthvdb]

seeing things like this:

2023-05-24 11:54:59.949 INFO (MainThread) [httpx] HTTP Request: GET http://homeassistant:<redactedportnumber>/api/config "HTTP/1.1 200 OK"
2023-05-24 11:54:59.949 DEBUG (MainThread) [httpcore] http11.receive_response_body.started request=<Request [b'GET']>
2023-05-24 11:54:59.950 DEBUG (MainThread) [httpcore] http11.receive_response_body.complete
2023-05-24 11:54:59.950 DEBUG (MainThread) [httpcore] http11.response_closed.started

and

2023-05-24 11:54:41.973 DEBUG (MainThread) [homeassistant.components.rest.data] Updating from http://homeassistant:<redactedportnumber>/api/config
2023-05-24 11:54:41.974 DEBUG (MainThread) [httpx] load_ssl_context verify=<ssl.SSLContext object at 0x7f2382517d40> cert=None trust_env=True http2=False

but other than that, dont see a lot indicating issues for these rest sensors

[epenet]

Please try to send a fuller log. The above doesn't identify the httpx request, and they are not in the correct order.

I am trying to understand if the issue is from the rest integration or from the api/config endpoint.

[Mariusthvdb]

log snippet.txt

hope this helps. Ive edited the private details, and cut many of the startup log entries that dont relate to this. if you need more, Id be happy to share, but not going public ;-)

[epenet]

OK... I think I have an explaination. I think simply the initial REST request is sent too soon, whilst the server is still starting up. So the first attempt has a smaller response, but subsequent attempts have the full response.

You can try to change the scan_interval to 60 seconds, then you should find that the original sensor gets the full response after 1 minute.

[Mariusthvdb]

ok, let me try that.

so reloading the rest yaml config has no impact on that ( I mean, I did try that, and it does not seem to change anything). Or that might be explained by the fact reloading does not work at all, as posted above.

also, why would there be a difference for those 2 formats? Seems unexpected.

edit

this is even more surprising...

ive taken out the scan_interval on the rest: sensor, and now it reports far more components: 421...

the platform: rest sensor still holds 235 components, so maybe that too needs the scan_interval to be lowered?

Thing is, I made it to update once a day, so it would not bother the system. But with that set, it will never be correct. at least from startup.

setting it to once a minute (which is the default?) makes it work will be using resources unnecessarily ..

[epenet]

The difference is probably simply the response delay. The core continues to load during the delay between first and second sensor.

[Mariusthvdb]

I've just tested that, and that is not completely true. Taking out the scan_interval on the platform: rest sensor now also loads those extra components on second update. ( so after 1 minute, the default scan_interval)

so we definitely need to lower that, for the sensors to be reliable. Wonder what this means for all of my other rest sensors...

can we use the homeassistant.update_entity service for this?

so set a lower scan_interval (like before), and then auto update once then system has started, and next leave as is?

this seems to not work:

      - service: homeassistant.update_entity
        target:
          entity_id:
            - sensor.ha_main_config
            - sensor.ha_main_config_rest

[epenet]

Since the problem has been identified, this is no longer a rest issue, but a general HA issue and I don't have an easy solution.

update_entity service does not work with a large scan interval because there is a cooldown period in the coordinator (=scan interval) update_entity service is really meant to be used with config entries that have updates disabled

Right now, I only see shorter scan interval as a solution - but maybe there is another opinion out there.

[Mariusthvdb]

yeah, I understand that. (although I now see the entity Is updated and receives all of its components. It has a certain lag, but with scan_interval back to 86400, and the service in a post startup script, all is populated)

any thoughts on the failure to reload the yaml and the sensors not updating their config (the icon in this case)?

[epenet]

any thoughts on the failure to reload the yaml and the sensors not updating their config (the icon in this case)?

That should be a separate issue... I've never looked at how the reload service works.

It might be that it only picks up new entities, but doesn't alter existing entities?

[Mariusthvdb]

made that separate issue, but that too is a bit complex, in the sense a singel issue (reload) reveals several consequences. Hope it is clear

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.