search

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.
MIT License
275 stars 14 forks source link

Form not found, shadowroot problem? #250

Closed marcinbauer85 closed 1 year ago

marcinbauer85 commented 1 year ago

Version of the custom_component

6.7.3

Configuration

multiscrape:
  - resource: "https://app.usertesting.com/my_dashboard/available_tests_v3"
    scan_interval: 20
    timeout: 30
    log_response: true
    form_submit:
      submit_once: true
      resubmit_on_error: true
      resource: "https://app.usertesting.com/participants/sign_in"
      select: ".ng-untouched .ng-pristine .ng-invalid"
      input:
        email: x
        password: x

    sensor:
      - unique_id: ut_tests
        name: Number of tests
        select: "head > title"
        value_template: "{{ (value) }}"

Describe the bug

Im trying to setup a script to see how many usertesting.com tests are available for me, but Im getting an error when trying to submit the form in order to login to my dashboard. Maybe its because the form on the login site is in the shadowroot?

Debug log

This error originated from a custom integration.

Logger: custom_components.multiscrape.coordinator
Source: custom_components/multiscrape/coordinator.py:75
Integration: Multiscrape scraping component (documentation, issues)
First occurred: 11:57:10 AM (9 occurrences)
Last logged: 12:00:00 PM

Scraper_noname_0 # Exception in form-submit feature. Will continue trying to scrape target page. Could not find form
marcinbauer85 commented 1 year ago

Or maybe its because the form on this example didn't load yet, since the page looks like its loading parts of it asynchronously ?