Out of range list's index

docyx / pc-part-dataset

A dataset of PC parts scraped from PCPartPicker

MIT License

172 stars 41 forks source link

Out of range list's index #3

Closed carabistouflette closed 1 year ago

carabistouflette commented 1 year ago

Everything is installed, maybe it's my fault but i don't know where i'm wrong. INPUT:

python -m pipenv run python scripts/scrape.py

OUTPUT:

Traceback (most recent call last):
  File "scripts/scrape.py", line 137, in <module>
    scrape(endpoint, session=session)
  File "scripts/scrape.py", line 70, in scrape
    max_pages = int(h.find(".pagination > li")[-1].text)
IndexError: list index out of range

docyx commented 1 year ago

Hey, sorry for the late reply -- PCPartPicker has implemented a captcha that makes scraping really difficult, and kind of breaks this whole script, which is why the data hasn't been updated in nearly 2 years.

I haven't had time to work on a new scraper but it'll come at some point. Sorry!

docyx commented 1 year ago

data has been updated as of #5, everything should be in working order now :)