search

maria-antoniak / goodreads-scraper

A Python scraper for Goodreads books and reviews.
GNU General Public License v3.0
274 stars 83 forks source link

Will only scrape reviews of 1 title #9

Closed JosedeKruif closed 3 years ago

JosedeKruif commented 3 years ago

Thank you for this very useful notebook. The code for fetching book titles works. The code for fetching reviews starts up, but opens only the reviews for 1 title. After that, reviews of the next title are not fetched. No idea what goes wrong since there is no feedback on errors. Also, it might be nice to include pandas into the requirements.tx.t

maria-antoniak commented 3 years ago

Hi José! Thanks for your feedback. Could you provide some example book IDs so that I can test the problem?

JosedeKruif commented 3 years ago

Hi Maria, I am testing the stuff using the example files. Scraping started out with "Pride and Predjudice" and then stopped. It seems I now found the culprit. Safety measures of my organization prevented "automated" writing in directory by Jupyter. In the mean time. I was able to perform some further testing. Good news is: the python script works fine when not in Jupyter environment. I will report some further observations in another issue.

JosedeKruif commented 3 years ago

Forgot to mention: I plan to test the script in R, using reticulate. Will keep you informed if you wish.

maria-antoniak commented 3 years ago

Glad to hear you figured out the problem! I wouldn't recommend running this program inside a Jupyter notebook. I'm closing this issue now, but let us know if you have other problems.