niklasb / dryscrape

[not actively maintained] A lightweight Python library that uses Webkit to enable easy scraping of dynamic, Javascript-heavy web pages
http://dryscrape.readthedocs.io/
MIT License
533 stars 67 forks source link

google search results right panel is not in session source file #49

Closed deter3 closed 8 years ago

deter3 commented 8 years ago

running the code below :

search_term = 'tokyo skytree' sess = dryscrape.Session(base_url = 'https://www.google.co.jp') sess.set_attribute('auto_load_images', True) sess.visit('/') q = sess.at_xpath('//*[@name="q"]') q.set(search_term) q.form().submit()

print sess.source()

There is a google right knowledge panel in the rendered png file (see below ) , but can not find in the session source file .

How can I fix it ? Thanks a lot

Richard

google

niklasb commented 8 years ago

Compare Session#source and Session#body. The letter is probably what you want.

deter3 commented 8 years ago

Thanks a lot for the prompt reply . You are right , session#body is the one I am looking for .

The strangest thing is webkit_server and PhantomJS might has different xpath class from firefox and chrome for same elements .