search

pnbt / youtube-explore

Finding YouTube's top recommendations from any query
211 stars 54 forks source link

Script doesn't find any video #9

Closed dhsdshdhk closed 5 years ago

dhsdshdhk commented 5 years ago

I am trying to run this script every day, and today I ran into this problem:


INFO We will NOT be printing keys - False
Running, will save the resulting json to:results/youtube-scrape-2019-08-13.json
Location = None Language = None
Searching for Bolsonaro
Sorting search restuls by relevance
Searching URL: https://www.youtube.com/results?sp=EgIQAQ%253D%253D&q=Bolsonaro
Search results []

Search term = Bolsonaro

counts: {}

It just won't find anything to continue crawling, despite the fact that we see a bunch of videos in that URL. Did youtube change something to break this script?

dhsdshdhk commented 5 years ago

Update: it seems to work sometimes, and finds results for some queries and not for others.

dhsdshdhk commented 5 years ago

Found a solution. If you run into this problem or a division by zero problem, it's lxml's fault and you should use Python’s html.parser, BeautifulSoup(html, "html.parser").

brunoanuncia commented 4 years ago

I'm having the same trouble. Python's html.parser solution doesn't work for me. Do you have any other suggestion @dhsdshdhk? Thank you in advance.