add to web data advanced

[hannesdatta]

Web data advanced currently shows how to scroll through the entire page.

Yet, it may be super useful for students to learn how to only scroll "once", or "twice", or a little bit.

Please add a little section to the tutorial where this is done (inspiration can be find in students' project submissions).

[RalphGit21]

From Stackoverflow, should work for twitter as well:

SCROLL_PAUSE_TIME = 0.5

- Get scroll height

last_height = driver.execute_script("return document.body.scrollHeight")

while True:

- Scroll down to bottom

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

#- Wait to load page
time.sleep(SCROLL_PAUSE_TIME)

#- Calculate new scroll height and compare with last scroll height
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
    break
last_height = new_height

[hannesdatta]

Try out & add to tutorial?

[hannesdatta]

Try out on a different website and then add to tutorial.

[RalphGit21]

Tested in the tutorial and works - Not sure how to implement this in google colab (may be due to access rights)

[hannesdatta]

please still add to the tutorial & commit