omkarcloud / botasaurus

The All in One Framework to build Awesome Scrapers.
https://www.omkar.cloud/botasaurus/
MIT License
1.36k stars 124 forks source link

Selenium webdriver chrome 115 stopped working #10

Closed shubhamgrg04 closed 1 year ago

shubhamgrg04 commented 1 year ago

Description

The framework is failing for a new installation as it is not able to find ChromeDriver as it only goes up to version 114 due to driver restructuring by the Chromium Team for the new Chrome-for-Testing.

Steps to Reproduce

Additional context

Traceback


Traceback (most recent call last):
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/shubhamgarg/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/Users/shubhamgarg/src/google-maps-scraper/main.py", line 18, in <module>
    launch_tasks(*tasks_to_be_run)
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/launch_tasks.py", line 54, in launch_tasks
    current_output = task.begin_task(current_data, task_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/base_task.py", line 214, in begin_task
    final = run_task(False, 0)
            ^^^^^^^^^^^^^^^^^^
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/base_task.py", line 155, in run_task
    create_directories(self.task_path)
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/base_task.py", line 99, in create_directories
    _download_driver()
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/base_task.py", line 34, in _download_driver
    download_driver()
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/download_driver.py", line 50, in download_driver
    move_driver()
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/download_driver.py", line 39, in move_driver
    move_chromedriver()
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/envs/gmaps-scraper/lib/python3.11/site-packages/bose/download_driver.py", line 38, in move_chromedriver
    shutil.move(src_path, dest_path)
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/lib/python3.11/shutil.py", line 845, in move
    copy_function(src, real_dst)
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/lib/python3.11/shutil.py", line 436, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/Users/shubhamgarg/.pyenv/versions/3.11.2/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'build/115/chromedriver'
shubhamgrg04 commented 1 year ago

There is a stackoverflow thread related to this issue - https://stackoverflow.com/questions/76727774/selenium-webdriver-chrome-115-stopped-working

Chetan11-dev commented 1 year ago

This is a Temporary Bug due which occurs in Chrome 115 due to chromedriver-autoinstaller dependency. The fix is to:

  1. Go to https://googlechromelabs.github.io/chrome-for-testing/#stable and download "chrome driver" based on your chrome version. (Most likely 115)

  2. Extract the zip file

  3. Place the "chromedriver" file (or "chromedriver.exe" if you are using Windows) directly into the build/ folder. Make sure you're not placing the extracted folder but only the chrome driver file itself.

    Screenshot (520).png

  4. Run python main.py and program should run successfully.

Chetan11-dev commented 1 year ago

Fixed in latest version