search

scrapinghub / splash

Lightweight, scriptable browser as a service with an HTTP API
BSD 3-Clause "New" or "Revised" License
4.09k stars 513 forks source link

How to debug why javascript isn't loading #722

Closed cornelyus closed 4 years ago

cornelyus commented 6 years ago

I am using splash "browser" to scrape with scrapy a heavy javascript page.. all was working fine, until yesterday that the page only renders part of it all.

I tried a lot of solutions shown in other issues, longer wait, pushing the limit of timeout, running in private mode.. all with no result. The screenshot I am getting seems that it's not a matter of time, but appears to be "blocking" from rendering.

I am printing the HTML and HAR but don't know how to look for the exact problem .. how to "debug" essentially..

Any ideas?

kmike commented 6 years ago

We have common reasons in FAQ (http://splash.readthedocs.io/en/stable/faq.html#website-is-not-rendered-correctly), but it seems you've tried most of them already.

I can also suggest starting Splash with verbosity level 2:

docker run -it --rm -p8050:8050 scrapinghub/splash:3.1

In this case you should be able to check JS console messages in Splash log output; if there are JS errors they may give some ideas on what's going on.

cornelyus commented 6 years ago

Thank you @kmike i'll try to check those out!