search

scrapinghub / splash

Lightweight, scriptable browser as a service with an HTTP API
BSD 3-Clause "New" or "Revised" License
4.08k stars 514 forks source link

Splash does not render elements generated by javascript #1149

Open nabilchaabane opened 2 years ago

nabilchaabane commented 2 years ago

I tried to scrape this webpage for reviews: https://www.yelp.com/biz/brendas-french-soul-food-san-francisco-5 but Splash is not rendering them since they are created by javascript. I tried to increase the wait, used different browsers and disable private mode but it did not work. Any help would be much appreciated.

danilo4pm commented 2 years ago

Same problem here. Need to click a div to open a modal gallery from this site https://www.daft.ie/ one sample page is any announced real state property from this page https://www.daft.ie/property-for-sale/ireland I'm pasting the root page as the properties may have disappeared the time of reading this. But one sample is https://www.daft.ie/for-sale/detached-house-7-grange-park-grangend-dunshaughlin-co-meath/3807786 The div that should be rendered is this:

Most of the dynamic content of the page is loaded. Just this one is not.

wangdong1992 commented 1 year ago

Same problem here.I want to get price from https://i-item.jd.com/100045525084.html. I tried to increase the wait, used different browsers and disable private mode but it did not work. Any help would be much appreciated.

nabilchaabane commented 1 year ago

You should avoid using splash. You can get the information you need by monitoring the network tab in the dev tools of your browser and figure out the API call. You can then retrieve information using any programming language you want. I recommend using Python