search

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.
MIT License
296 stars 16 forks source link

Using resource_template at startup of home assistant there are not important errors in log #103

Closed Metus88 closed 2 years ago

Metus88 commented 2 years ago

Version of the custom_component

v5.7.0

System Health

version core-2022.3.8
installation_type Home Assistant Container
dev false
hassio false
docker true
user root
virtualenv false
python_version 3.9.9
os_name Linux
os_version 5.10.103-v7l+
arch armv7l
timezone Europe/Rome
Home Assistant Community Store GitHub API | ok -- | -- GitHub Content | ok GitHub Web | ok GitHub API Calls Remaining | 3957 Installed Version | 1.24.1 Stage | running Available Repositories | 1006 Downloaded Repositories | 33
Home Assistant Cloud logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok
Lovelace dashboards | 1 -- | -- resources | 19 views | 24 mode | storage

Configuration

sensor:
  - platform: file
    file_path: /config/filestatisensori/linkautoletturagas.txt
    name: link gas da file

multiscrape:
  - resource_template: "{{ states('sensor.link_gas_da_file')}}"
    scan_interval: 3600
    name: letture date gas
    sensor:
      - unique_id: fai_la_lettura_del_gas_dal
        name:  Fai la lettura del gas dal
        select: 'script[type="text\/plain"]'
        value_template: "{{ (value | regex_findall_index('data_da..............', index=0)).split('\"')[-2].strip()  }}"
      - unique_id: fai_la_lettura_del_gas_al
        name:  Fai la lettura del gas al
        select: 'script[type="text\/plain"]'
        value_template: "{{ (value | regex_findall_index('data_a..............', index=0)).split('\"')[-2].strip()  }}"

Describe the bug

During the startup of home assistant there are Errors in log that tell that the "resource_template" is not a link like http or https. How you can see from my configuration.yaml I use a sensor file where inside the txt file is wrote the url link. I don't know if it's possible to add a "delay" (standard or editable from the user) during the startup for the component/sensor OR a waiting template.
Is not a big problem. Maybe there is already a solution that I don't know. However also with this log info/error it works great... so thanks a lot for your fantastic component!

Debug log


2022-04-01 22:21:19 ERROR (MainThread) [custom_components.multiscrape.scraper] Error fetching data: unknown failed with Request URL is missing an 'http://' or 'https://' protocol.
2022-04-01 22:21:19 ERROR (MainThread) [custom_components.multiscrape.scraper] Unable to parse response.
2022-04-01 22:21:19 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: Request URL is missing an 'http://' or 'https://' protocol.; Retrying in background in 30 seconds
danieldotnl commented 2 years ago

Could you please check if this behavior still exists in version pre-v6.1.0?

Metus88 commented 2 years ago

I just upgrade to beta version "pre-v6.1.0" from hacs. I restarted homeassistant and there are changes:

2022-04-13 21:11:05 WARNING (SyncWorker_6) [homeassistant.loader] We found a custom integration multiscrape which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2022-04-13 21:11:12 ERROR (MainThread) [custom_components.multiscrape.http] letture date gas # Error executing get request to url: unknown.
 Error message:
 UnsupportedProtocol("Request URL is missing an 'http://' or 'https://' protocol.")
2022-04-13 21:11:12 ERROR (MainThread) [custom_components.multiscrape.coordinator] letture date gas # Updating failed with exception: Request URL is missing an 'http://' or 'https://' protocol.
2022-04-13 21:11:12 ERROR (MainThread) [custom_components.multiscrape.coordinator] Error fetching multiscrape data: Unable to update data from resource
2022-04-13 21:11:22 WARNING (MainThread) [homeassistant.setup] Setup of multiscrape is taking over 10 seconds.
2022-04-13 21:11:25 ERROR (MainThread) [custom_components.multiscrape.http] letture date gas # Error executing get request to url: unknown.
 Error message:
 UnsupportedProtocol("Request URL is missing an 'http://' or 'https://' protocol.")
2022-04-13 21:11:25 ERROR (MainThread) [custom_components.multiscrape.coordinator] letture date gas # Updating failed with exception: Request URL is missing an 'http://' or 'https://' protocol.
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:25 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:27 WARNING (MainThread) [homeassistant.components.sensor] Platform multiscrape not ready yet: None; Retrying in background in 30 seconds
2022-04-13 21:11:55 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform multiscrape is taking over 10 seconds.
2022-04-13 21:11:57 ERROR (MainThread) [custom_components.multiscrape.http] letture date gas # Error executing get request to url: https://************CENSORED************** (I edit it to privacy =) ).
 Error message:
 ConnectTimeout('')
2022-04-13 21:11:57 ERROR (MainThread) [custom_components.multiscrape.coordinator] letture date gas # Updating failed with exception:

However after a few minutes all the sensors works again (like the preview version). So i don't know if I can give you more details. I'm noob =/ I think that my problem is that I want use a link that is inside a txt file that I read with sensor

sensor:
  - platform: file
    file_path: /config/filestatisensori/linkautoletturagas.txt
    name: link gas da file

And during the startup of home assistant this sensor is maybe empty... so multiscrape have not the link (resource_template)

multiscrape:
  - resource_template: "{{ states('sensor.link_gas_da_file')}}"
    scan_interval: 3600
    name: letture date gas
    ....

So it is not your problem... but mine. I try to set resource_template directly the link and I will prove that is not you problem but mine.

EDIT: YES! I can confirm it. If I set directly the link on the resource_template all work great.... no warning or error during startup of home assistant.

multiscrape:
  - resource_template: https://*****CENSORED*******
    scan_interval: 3600
    name: letture date gas

I have to google if there is a way to add a delay to some sensor in home assistant or dependency...

danieldotnl commented 2 years ago

Ok, makes sense, multiscrape parses the resource_template before the file sensor has run. I'll close the issue.