search

scrapehero-code / amazon-review-api

A very basic API to scrape product reviews from Amazon and get data in real time as JSON with all fields, that amazon product adverstising api does not provide.
https://www.scrapehero.com/free-amazon-product-reviews-api-build-you-own-using-python/
40 stars 16 forks source link

Internal server error #1

Open ghost opened 3 years ago

ghost commented 3 years ago

I get an internal server error on running this

cberrioa commented 3 years ago

I got the same server error, it seems that Amazon detects that you are trying to scrape their website. I oppened it with my browser, checked the cookies in the request, and then uses those cookies in the request that is been done in app.py, something like this: cookies= {"aws_lang": "YOUR_AWS_LANG", "i18n-prefs" : "YOUR_i18N_PREFS", "regStatus" : "YOUR_REGSTATUS", "session-id" : "YOUR_SESSION_ID", "session-id-time" : "YOUR_SESSION_ID_TIME", "session-token": "YOUR_SESSION_TOKEN", "skin" : "YOUR_SKIN", "sp-cdn": "YOUR_SP_CDN", "ubid-main": "YOUR_UBID"} ... r = requests.get(url, headers=headers, cookies=cookies) ... (and I also had to change this line:) data['number_of_reviews'] = int(data['number_of_reviews'].split(' global rating')[0].replace(',', '') )

schabertrobbinger commented 3 years ago

@cberrioa thank you so much for your helpful comments! By following your directions, I was finally able to get something other than "internal server error" out. I made a big mistake assuming that this API would work out of the box...would you by any chance have any idea how to automate the extraction of the cookies in flask? Also, did the API just return the first page of review results for you? That's all that I have been able to get out so far, though I suppose the cookies are the real show-stopper here.

cberrioa commented 3 years ago

@cberrioa thank you so much for your helpful comments! By following your directions, I was finally able to get something other than "internal server error" out. I made a big mistake assuming that this API would work out of the box...would you by any chance have any idea how to automate the extraction of the cookies in flask? Also, did the API just return the first page of review results for you? That's all that I have been able to get out so far, though I suppose the cookies are the real show-stopper here.

@schabertrobbinger sorry, I haven't tried to extract these cookies automatically, and right now I don't have much idea how to do that. I have the same issue that the API just returns the first page of review results to me, it seems that the other reviews are obtained dynamically.

Zezzz830 commented 2 years ago

It says {"error":"URL to scrape is not provided"} every time I run the flask application, and I wonder why this happens.