search

kaushikthedeveloper / Goodreads-Giveaway-BOT

An automated Bot that enters the user onto Giveaway on Goodreads
MIT License
6 stars 2 forks source link

scraper error? #5

Open tschrist opened 3 years ago

tschrist commented 3 years ago

2021-05-24 09:32:27 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.goodreads.com/giveaway?sort=recently_listed&tab=recently_listed> (referer: https://www.goodreads.com/) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/usr/lib/python3/dist-packages/scrapy/core/spidermw.py", line 84, in evaluate_iterable for r in iterable: File "/usr/lib/python3/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output for x in result: File "/usr/lib/python3/dist-packages/scrapy/core/spidermw.py", line 84, in evaluate_iterable for r in iterable: File "/usr/lib/python3/dist-packages/scrapy/spidermiddlewares/referer.py", line 339, in return (_set_referer(r) for r in result or ()) File "/usr/lib/python3/dist-packages/scrapy/core/spidermw.py", line 84, in evaluate_iterable for r in iterable: File "/usr/lib/python3/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in return (r for r in result or () if _filter(r)) File "/usr/lib/python3/dist-packages/scrapy/core/spidermw.py", line 84, in evaluate_iterable for r in iterable: File "/usr/lib/python3/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in return (r for r in result or () if _filter(r)) File "/root/Goodreads-Giveaway-BOT/goodreads/spiders/giveaway.py", line 106, in giveaway_pages pages_list.pop() IndexError: pop from empty list

DanielSmith1239 commented 3 years ago

I'm having this issue as well.

DanielSmith1239 commented 3 years ago

I believe it's caused by goodreads making some modifications to their website so the bot can't parse the pages.

DanielSmith1239 commented 3 years ago

@kaushikthedeveloper just forked and made some changes that fixes the issue: https://github.com/DanielSmith1239/GoodreadsMaster

kaushikthedeveloper commented 2 years ago

Hey folks. Am happy to see this project is still running. While I created this project for personal usage and for anyone else to try it out, since Goodreads stopped Giveaways across India (basically anywhere outside USA), I stopped looking into this.

Will check it out and give it a try once more to keep it up to date.

@DanielSmith1239 , awesome fork buddy. Will take a look.

danielkadosh10 commented 2 years ago

@DanielSmith1239 I know this isnt your fork but I couldn't find a way to comment on your fork, but the bot is not working currently could you maybe fix it?

DanielSmith1239 commented 2 years ago

@DanielSmith1239 I know this isnt your fork but I couldn't find a way to comment on your fork, but the bot is not working currently could you maybe fix it?

Sure I’ll take a look, thanks for letting me know. I have the bot automated so I didn’t realize it wasn’t working. Do you have any info that might help me pinpoint the issue?

danielkadosh10 commented 2 years ago

@DanielSmith1239 yes I'm not that good at network but from what I could gather their original sign in page was was https://goodreads.com/user/sign_in that's where you made the form request but now you need to press log in with email on that page and it takes you to another sign in page

DanielSmith1239 commented 2 years ago

@DanielSmith1239 yes I'm not that good at network but from what I could gather their original sign in page was was https://goodreads.com/user/sign_in that's where you made the form request but now you need to press log in with email on that page and it takes you to another sign in page

Yup it should be all fixed now! That was the main issue, and they changed how the authentication worked behind the scenes (I think) so I had to add some cookie stuff. I learned a lot more about scrapy today lol

danielkadosh10 commented 2 years ago

@DanielSmith1239 cool thank you, if you dont mind me asking but where did you learn how to really use scrapy

DanielSmith1239 commented 2 years ago

Uhhh Google pretty much lol, I wouldn’t really say I know how to use it. I just kept googling “how to do x with scrapy”, which is how I learn most of my coding stuff