search

hartleybrody / public-amazon-crawler

676 stars 227 forks source link

RuntimeError: maximum recursion depth exceeded #5

Closed pratgen closed 7 years ago

pratgen commented 7 years ago

I tried to test this script. I found following issues a)I first got Encoding error which I resolved by changing line 48 in helpers.py with page_text = r.text.encode('utf-8').decode('ascii', 'ignore') return BeautifulSoup(page_text), page_text b)The script did not find any sub category and probably that results in recursion depth error. python error.txt log2.txt

hartleybrody commented 7 years ago

As you can see from the long string of error message log lines, the issue is that you're running the scraper w no URLs in the queue. As mentioned in the project's readme, it's up to you to start by populating the queue with category (and subcategory) URLs.

I provide some sample code written years ago for populating the queue, but I can't know ahead of time what categories you care about for your scrape and how the markup on those category pages differ from the ones I saw.