search

leonkansh / shopping-web-scraper

2 stars 1 forks source link

loads out Amazon page but `NoneType' object has no attribute 'find_all'` #4

Open leonkansh opened 3 years ago

leonkansh commented 3 years ago

Gathering Amazon listings... Traceback (most recent call last): File "main.py", line 47, in <module> amazon_price, URL = searchAmazon(search, requests, BeautifulSoup, driver) File "/Users/kanyuanliang/Documents/GitHub/shopping-web-scraper/amazon.py", line 10, in searchAmazon listings = results.find_all('div', class_='s-include-content-margin s-border-bottom s-latency-cf-section') AttributeError: 'NoneType' object has no attribute 'find_all'

leonkansh commented 3 years ago

It might be caused by Amazon changed the element names in HTML

leonkansh commented 3 years ago

Code runs... /* Enter the item you want to search for: bluetooth speaker

Searching for bluetooth speaker Gathering eBay listings... Back Bay- Newbury Wireless Bluetooth Speaker. Water-Resistant, 24-Hour Battery $39.99

https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313.TR12.TRC2.A0.H0.XApple+iPhone.TRS0&_nkw=bluetooth+speaker Gathering Amazon listings... OontZ Angle 3 (3rd Gen) - Bluetooth Portable Speaker, Louder Volume, Crystal Clear Stereo Sound, Rich Bass, 100 Ft Wireless Range, Microphone, IPX5, Bluetooth Speakers by Cambridge Sound Works (Black) $25.99

$25.99

price is returned https://www.amazon.com/s?k=bluetooth+speaker Comparing the three prices... Here's the link to the lowest price. Have fun! https://www.amazon.com/s?k=bluetooth+speaker $25.99 */

leonkansh commented 3 years ago

After I uncommented this block in amazon.py:

runs = 0
results = soup.findAll('span', attrs={'class': 'a-offscreen'})
for listing in results:
    if runs >= 1: break
    element = soup.select_one('span.a-offscreen')
    if None in element:
        continue
    print(element.get_text())
    print()
    print()
    runs += 1