Closed fashionprivate closed 11 months ago
One, Twitter already provides an API that you can use to scrape it: https://developer.twitter.com/en/docs/twitter-api.
Two, SeleniumBase methods have automatic-waiting. You should never be using the external implicitly_wait
, WebDriverWait
, or EC.presence_of_element_located
anywhere in your code. Use the built-in methods instead. For the raw driver
formats, see examples such as SeleniumBase/examples/raw_login_driver.py, SeleniumBase/examples/raw_driver_manager.py, and SeleniumBase/examples/offline_examples/test_extended_driver.py. For the pytest
formats, see any example test that starts with test_
or ends with _test
in the SeleniumBase/examples folder.
Hello everybody,
I'm writing a bot to continuously scrape a Twitter account to retrieve the most recent tweets. I don't know why but the SeleniumBase library gives an error after about 1 minute of scraping without errors. Below is the code and the error found after about 1 minute of correct scraping:
The error is the following:
What is wrong? The strange thing is that it works for about a minute, then the error is always selenium.common.exceptions.TimeoutException: Message: and I can't handle it. Is it possible to reinitialize the driver? Are there other solutions?