search

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.
MIT License
293 stars 16 forks source link

[feature request] Is login successful? Why does scrape fail? #73

Closed astranberg closed 2 years ago

astranberg commented 2 years ago

Version of the custom_component

v5.5.0

Configuration

multiscrape:
  #https://thepagewiththedatathatyouwant.com
  - resource: "https://customer.xfinity.com/#/services/internet#usage"
    scan_interval: 3600
    form_submit:
      submit_once: False
      #https://thesitewiththeform.com
      resource: "https://login.xfinity.com/login"
      select: "#right > div > form"
      input:
        user: !secret xfinity_username
        passwd: !secret xfinity_password
    sensor:
      - select: "#usage > div > div:nth-child(2) > div > div > div > p > span > b:nth-child(1)"
        name: scraped-value-after-form-submit

Describe the bug

Failed to extract data, but why? Is select not available on the page? Did login occur successfully? I would like more debug info to troubleshoot the issue.

Debug log

Logger: custom_components.multiscrape.sensor
Source: custom_components/multiscrape/sensor.py:139
Integration: Multiscrape scraping component (documentation, issues)
First occurred: 6:36:38 PM (1 occurrences)
Last logged: 6:36:38 PM

Sensor scraped-value-after-form-submit was unable to extract data from HTML
danieldotnl commented 2 years ago

I dedicated a full release to troubleshooting. Could you please try pre-v6.0.0 and let me know your thoughts?