search

scrapehero-code / amazon-review-scraper

A basic python 3 based web scraper for extracting reviews from Amazon. Built using Selectorlib and requests
https://www.scrapehero.com/how-to-scrape-amazon-product-reviews/
54 stars 49 forks source link

TypeError: 'NoneType' object is not iterable #2

Open wolleakira opened 4 years ago

wolleakira commented 4 years ago

I installed all the requirements and ran the code for testing without changing the urls or anything and got the following error:

Downloading https://www.amazon.com/Nike-Womens-Reax-Running-Shoes/product-reviews/B07ZPL752N/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews Traceback (most recent call last): File "C:/Users/amazon-review-scraper-master/reviews.py", line 46, in for r in data['reviews']: TypeError: 'NoneType' object is not iterable

Could you tell me where the code might need to be changed or what I should do?

christiefung commented 4 years ago

I have the same issue! It used to work fine (a few weeks ago) but I see the same error right now. Would you great if anybody can help fixing this?

jabney27 commented 4 years ago

Looks like Amazon has begun blocking these requests. The Robots.txt file shows the customer reviews are Disallowed. They appear to be pushing folks to the Product Advertising API which requires you to be a seller or have a e-commerce site to pull the data. Hopefully I'm wrong, but based on the information that came back on the query it appears that is what is going on.

Enter the characters you see below

Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.

scrapehero-code commented 4 years ago

Definitely getting blocked here. You may want to change the User-Agent

rishab05 commented 3 years ago

In reviews.py if we add one more condition in if then It will work fine because if verified purchase is blank(None) then it will show error. If we add a condition in if block to check that, the error will be solve. changes:

    if 'verified' in r and r['verified'] is not None:
        if 'Verified Purchase' in r['verified']:
            r['verified'] = 'Yes'
       else:
            r['verified'] = 'Yes'
   else:
       r['verified'] = 'No'

You can see I add one more condition in first If and also add a else for outer If.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.