thefakequake / pypartpicker

A Python package that can be used to fetch information from PCPartPicker on products and parts lists.
MIT License
18 stars 8 forks source link

NoneType object has no attribute get_text apparently #15

Closed not-NEKO closed 1 year ago

not-NEKO commented 1 year ago

I tried to reinstall with pip but nothing changed

Input: from pypartpicker import Scraper Scraper().part_search("i5") Output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/username/.local/lib/python3.10/site-packages/pypartpicker/scraper.py", line 232, in part_search
    soup = self.__make_soup(f"{search_link}&page={i + 1}")
  File "/home/username/.local/lib/python3.10/site-packages/pypartpicker/scraper.py", line 94, in __make_soup
    if "Verification" in soup.find(class_="pageTitle").get_text():
AttributeError: 'NoneType' object has no attribute 'get_text'
not-NEKO commented 1 year ago

Information about my computer maybe:

OS: Arch linux Python: 3.10.9

everything was working fine at the beggining but it stopped working at a moment i don't know why. Maybe because pcpartpicker black listed me? Other scrapers seem to work so i don't think.

not-NEKO commented 1 year ago

nevermind it's my fault

TerminalSwagDisorder commented 1 year ago

What was your problem? I don't know if I'm stupid, but I'm getting the same error.

thefakequake commented 1 year ago

Essentially, it means that your request to PCPartPicker got blocked. All pages on PCPP as far as I can tell have this element: image

If the object is None, then your request didn't go through.

thefakequake commented 1 year ago

To fix this, use custom request headers with your request so that you can act more like a browser, pass the headers argument when you instantiate Scraper().

TerminalSwagDisorder commented 1 year ago

Thanks. For anyone looking into this, I had to specify both cookie and user-agent for it to work. You will find these values by right clicking on pcpartpicker.com and inspecting -> network tab -> possibly reloading the page -> choose name pcpartpicker.com -> scroll down to Request Headers -> copy both values

pcpp = Scraper(headers={ "cookie": "your cookie", "user-agent": "your user-agent" })