search

alirezamika / autoscraper

A Smart, Automatic, Fast and Lightweight Web Scraper for Python
MIT License
6.24k stars 654 forks source link

Need help to get price from the web site. #33

Closed somniapotato closed 4 years ago

somniapotato commented 4 years ago

Hi, I can't get price from this url, even though I try to add request header to it.

url: 

url = "https://www.lazada.com.my/products/pensonic-pb-7511-multifunctional-hand-blender-white-i15717885-s19365971.html"
wanted_list = ["105.24"] #the price is 105.24
s = requests.session()
scraper = AutoScraper()
s.get(url, headers=scraper.request_headers)
result = scraper.build(url, wanted_list, request_args={'cookies': s.cookies.get_dict()})

and the output is always None How can I get the price?

alirezamika commented 4 years ago

Hi,

This website uses javascript to populate the data. You can get the js rendered website html content (using tools like puppeteer or Requests-HTML) and pass the html content to the scraper via html parameter.

somniapotato commented 4 years ago

Hi,

This website uses javascript to populate the data. You can get the js rendered website html content (using tools like puppeteer or Requests-HTML) and pass the html content to the scraper via html parameter.

thx~