search

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.
MIT License
275 stars 14 forks source link

Unable to scrape #48

Closed hasamba closed 3 years ago

hasamba commented 3 years ago

im trying to scrape my socialblade page: i followed the wiki https://github.com/danieldotnl/ha-multiscrape/wiki/Scraping-guide and this is what i tried

 - resource: https://socialblade.com/youtube/channel/UCb9Iz9w_jEq3iXIVrQqwdsQ
    scan_interval: 600
    sensor:
      - unique_id: youtube_channel_subs
        name: Youtube Channel Subscribers
        select: "#YouTubeUserTopInfoBlock > div:nth-child(3) > span:nth-child(3)"
      - unique_id: youtube_channel_views
        name: Youtube Channel Views
        select: "#YouTubeUserTopInfoBlock > div:nth-child(4) > span:nth-child(3)"

i also tried:

select: "#youtube-stats-header-subs" (which works in regular scrape platform
select: ".youtube-stats-header-subs"
select: "youtube-stats-header-subs"

none of them works

i thought maybe the site blocks me so i tried adding:

    headers:
      User-Agent: Mozilla/5.0

didnt worked also

what am i missing?

hasamba commented 3 years ago

2 days i played with no success, after posting the issue it started working :)

BrendanxP commented 3 years ago

Everyone struggling to find out what is going wrong with the scarping should try it out at Jsoup.