Multiscrape

Need help with Multiscrape?

Personal (paid) support option

I very often get asked for help, for example with finding the right CSS selectors or with a login. Actually more often than I can handle, so I'm running an experiment with a paid support option!

Sponsor me here, and I'll try to assist you with your multiscrape configuration within 1-2 days. The support funds will go towards family time, making up for the hours I spend on Home Assistant ☺️.

Note: Scraping isn't always possible. I'd love to offer a "no cure, no pay" service, but GitHub Sponsoring doesn't support that. If you're concerned about sponsoring without guarentee, please reach out by email before sponsoring!

Other options

If you don't manage to scrape the value you are looking for, please enable debug logging and log_response. This will provide you with a lot of information for continued investigation. log_response will write all responses to files. If the value you want to scrape is not in the files with the output from BeautifulSoup (*-soup.txt), Multiscrape will not be able to scrape it. Most likely it is retrieved in the background by javascript. Your best chance in this case, is to investigate the network traffic in de developer tools of your browser, and try to find a json response containing the value you are looking for.

If all of this doesn't help, use the home assistant forum. I cannot give everyone personal assistance and please don't create github issues unless you are sure there is a bug. Check the wiki for a scraping guide and other details on the functionality of this component.

Important note: be a good citizen and be aware of your responsibility

You and you alone, are accountable for your scraping activities. Be a good (web) citizen. Set reasonable scan_interval timings, seek explicit permission before scraping, and adhere to local and international laws. Respect website policies, handle data ethically, mind resource usage, and regularly monitor your actions. Uphold these principles to ensure ethical and sustainable scraping practices.

Introduction

This Home Assistant custom component can scrape multiple fields (using CSS selectors) from a single HTTP request (the existing scrape sensor can scrape a single field only). The scraped data becomes available in separate sensors.

It is based on both the existing Rest sensor and the Scrape sensor. Most properties of the Rest and Scrape sensor apply.

Installation

\ Install via HACS (default store) or install manually by copying the files in a new 'custom_components/multiscrape' directory.

Example configuration (YAML)

multiscrape:
  - name: HA scraper
    resource: https://www.home-assistant.io
    scan_interval: 3600
    sensor:
      - unique_id: ha_latest_version
        name: Latest version
        select: ".release-date"
        value_template: "{{ value | trim }}"
      - unique_id: ha_release_date
        icon: >-
          {% if is_state('binary_sensor.ha_version_check', 'on') %}
            mdi:alarm-light
          {% else %}
            mdi:bat
          {% endif %}
        name: Release date
        select: ".release-date"
        attribute: "title"
        value_template: "{{ (value.split('released')[1]) }}"
    binary_sensor:
      - unique_id: ha_version_check
        name: Latest version == 2021.7.0
        select: ".release-date"
        value_template: '{{ value | trim == "2021.7.0" }}'
        attributes:
          - name: Release notes link
            select: ".release-date"
            attribute: href

Options

Based on latest (pre) release.

Sensor/Binary Sensor

Configure the sensors that will scrape the data.

Refresh button

Configure a refresh button to manually trigger scraping.

Sensor attributes

Configure the attributes on the sensor that can be set with additional scraping values.

Form-submit

Configure the form-submit functionality which enables you to submit a (login) form before scraping a site. More details on how this works can be found on the wiki.

Form Variables

Configure the variables that will be scraped from the form_submit response. These variables can be used in the value_template of the main configuration of the current integration: a selector in sensors/attributes or in a header. A common use case is to populate the X-Login-Token header which is the result of the login.

Example:

multiscrape:
  - resource: "https://somesiteyouwanttoscrape.com"
    form_submit:
      submit_once: True
      resource: "https://authforsomesiteyouwanttoscrape.com"
      input:
        email: "<email>"
        password: "<password>"
      variables:
        - name: token
          value_template: "{{ ... }}"
    headers:
      X-Login-Token: "{{ token }}"
    sensor: ...

Selector

Used to configure scraping options.

On-error

Configure what should happen in case of a scraping error (the css selector does not return a value).

Services

For each multiscrape instance, a service will be created to trigger a scrape run through an automation. (For manual triggering, the button entity can now be configured.) The services are named multiscrape.trigger_{name of integration}.

Multiscrape also offers a get_content and a scrape service. get_content retrieves the content of the website you want to scrape. It shows the same data for which you now need to enable log_response and open the page_soup.txt file.\ scrape does what it says. It scrapes a website and provides the sensors and attributes.

Both services accept the same configuration as what you would provide in your configuration yaml (what is described above), with a small but important caveat: if the service input contains templates, those are automatically parsed by home assistant when the service is being called. That is fine for templates like resource and select, but templates that need to be applied on the scraped data itself (like value_template), cannot be parsed when the service is called. Therefore you need to slightly alter the syntax and add a ! in the middle. E.g. {{ becomes {!{ and %} becomes %!}. Multiscrape will then understand that this string needs to handled as a template after the service has been called.\ If someone has a better solution, please let me know!

To call one of those services, go to 'Developer tools' in Home Assistant and then to 'services'. Find the multiscrape.get_content or multiscrape.scrape services and go to yaml mode. There you enter your configuration. Example:

service: multiscrape.scrape
data:
  name: HA scraper
  resource: https://www.home-assistant.io
  sensor:
    - unique_id: ha_latest_version
      name: Latest version
      select: ".release-date"
      value_template: "{!{ value | trim }!}"
    - unique_id: ha_release_date
      name: Release date
      select: ".release-date"
      attribute: "title"
      value_template: "{!{ (value.split('released')[1]) }!}"

Debug logging

Debug logging can be enabled as follows:

logger:
  default: info
  logs:
    custom_components.multiscrape: debug

Depending on your issue, also consider enabling log_response.

Contributions are welcome!

If you want to contribute to this please read the Contribution guidelines

Credits

This project was generated from @oncleben31's Home Assistant Custom Component Cookiecutter template.

Code template was mainly taken from @Ludeeus's integration_blueprint template