shaikhsajid1111 / twitter-scraper-selenium

Python's package to scrap Twitter's front-end easily
https://pypi.org/project/twitter-scraper-selenium
MIT License
322 stars 50 forks source link

Update dependencies to support Chrome version 115 and above #87

Closed IwasakiYuuki closed 2 months ago

IwasakiYuuki commented 2 months ago

On Chrome version 114 and older, we need to request the URL https://chromedriver.storage.googleapis.com/LATEST_RELEASE_ to determine the appropriate ChromeDriver version. However, starting with version 115 the ChromeDriver release process has been integrated with Chrome itself. More details can be found here: https://developer.chrome.com/docs/chromedriver/downloads/version-selection

Due to the above change, the following error occurred.

[WDM] - Current google-chrome version is 128.0.6613
[WDM] - Get LATEST driver version for 128.0.6613
Traceback (most recent call last):
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/profile.py", line 117, in scrap
    self.__start_driver()
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/profile.py", line 39, in __start_driver
    self.browser, self.headless, self.proxy).init()
                                             ^^^^^^
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/driver_initialization.py", line 94, in init
    driver = self.set_driver_for_browser(self.browser_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/driver_initialization.py", line 72, in set_driver_for_browser
    return webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()), options=self.set_properties(browser_option))
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/.local/lib/python3.12/site-packages/webdriver_manager/chrome.py", line 32, in install
    driver_path = self._get_driver_path(self.driver)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/.local/lib/python3.12/site-packages/webdriver_manager/manager.py", line 22, in _get_driver_path
    driver_version = driver.get_version()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/.local/lib/python3.12/site-packages/webdriver_manager/driver.py", line 40, in get_version
    return self.get_latest_release_version()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/.local/lib/python3.12/site-packages/webdriver_manager/driver.py", line 64, in get_latest_release_version
    validate_response(resp)
  File "/home/yuuki/.local/lib/python3.12/site-packages/webdriver_manager/utils.py", line 79, in validate_response
    raise ValueError("There is no such driver by url {}".format(resp.url))
ValueError: There is no such driver by url http://chromedriver.storage.googleapis.com/LATEST_RELEASE_128.0.6613

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/test.py", line 4, in <module>
    scrape_profile(
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/profile.py", line 195, in scrape_profile
    data = profile_bot.scrap()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/profile.py", line 127, in scrap
    self.__close_driver()
  File "/home/yuuki/ghq/github.com/IwasakiYuuki/twitter-scraper-selenium/twitter_scraper_selenium/profile.py", line 42, in __close_driver
    self.__driver.close()
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'close'

To resolve this error, updating webdriver-manager to version 4.0.1 and above is required. This issue was specifically addressed in PR #570. Additionally, issues caused by blinker and setuptools were also observed, so their versions need to be adjusted accordingly. Plese refer to the following URLs for more details:

After making these updates, the program ran successfully in my environment.

from twitter_scraper_selenium import scrape_profile

scrape_profile(
    twitter_username="microsoft",
    output_format="csv",
    browser="chrome",
    tweets_count=10,
    filename="microsoft",
    directory="./data"
)
shaikhsajid1111 commented 2 months ago

Understood, Thanks @IwasakiYuuki