Applicable driver not found; attempting to install with Selenium Manager (Beta)

[Habeeb556]

I got this INFO in celery log:

[WARNING/ForkPoolWorker-3] Applicable driver not found; attempting to install with Selenium Manager (Beta)

Originally posted by @Habeeb556 in https://github.com/apache/superset/issues/29401#issuecomment-2228313179

[dosubot[bot]]

To resolve the warning "Applicable driver not found; attempting to install with Selenium Manager (Beta)" in your Celery log while using Selenium, you need to ensure that the appropriate web driver (e.g., ChromeDriver for Chrome or GeckoDriver for Firefox) is installed and properly configured.

Follow these steps:

1. Install the Web Driver: If you are using Chrome, you need to install ChromeDriver. You can add the following commands to your Dockerfile to install ChromeDriver:

   RUN export CHROMEDRIVER_VERSION=$(curl --silent https://chromedriver.storage.googleapis.com/LATEST_RELEASE_102) && \
      wget -q https://chromedriver.storage.googleapis.com/${CHROMEDRIVER_VERSION}/chromedriver_linux64.zip && \
      unzip chromedriver_linux64.zip -d /usr/bin && \
      chmod 755 /usr/bin/chromedriver && \
      rm -f chromedriver_linux64.zip

2. Configure the Web Driver in superset_config.py: Ensure that you have the correct configuration for the web driver in your superset_config.py file. For Chrome, it should look like this:

   WEBDRIVER_TYPE = "chrome"
   WEBDRIVER_OPTION_ARGS = [
      "--force-device-scale-factor=2.0",
      "--high-dpi-support=2.0",
      "--headless",
      "--disable-gpu",
      "--disable-dev-shm-usage",
      "--no-sandbox",
      "--disable-setuid-sandbox",
      "--disable-extensions",
   ]

3. Verify the Installation: After making these changes, you can verify the installation by running the following commands inside your Superset worker container:

   firefox --headless
   geckodriver

   If you are using Chrome, you should verify with:

   chromedriver --version

By ensuring that the appropriate web driver is installed and correctly configured, you should be able to resolve the warning in your Celery log ^[1].

To continue talking to Dosu, mention @dosu.

[Habeeb556]

@dosu I am not using a docker

[dosubot[bot]]

To resolve the warning "Applicable driver not found; attempting to install with Selenium Manager (Beta)" in your Celery log in a non-Docker setup, follow these steps:

1. Install ChromeDriver: Ensure that ChromeDriver is installed and accessible in your system's PATH. You can download the appropriate version of ChromeDriver from here and place it in a directory that is included in your system's PATH.

2. Verify ChromeDriver Installation: Run the following command in your terminal to verify that ChromeDriver is correctly installed:

   chromedriver --version

   This should return the version of the installed ChromeDriver.

3. Configure the WebDriver in superset_config.py: Add the following configuration to your superset_config.py file to set the web driver type and options:

   WEBDRIVER_TYPE = "chrome"
   WEBDRIVER_OPTION_ARGS = [
      "--force-device-scale-factor=2.0",
      "--high-dpi-support=2.0",
      "--headless",
      "--disable-gpu",
      "--disable-dev-shm-usage",
      "--no-sandbox",
      "--disable-setuid-sandbox",
      "--disable-extensions",
   ]

4. Test WebDriver Accessibility: Create a simple Python script to test if the Celery worker can access the ChromeDriver and take a screenshot. Save the following script and run it in the same environment where your Celery worker operates:

   from selenium import webdriver
   from selenium.webdriver.chrome.service import Service
   from webdriver_manager.chrome import ChromeDriverManager
   
   options = webdriver.ChromeOptions()
   options.add_argument("--headless")
   options.add_argument("--disable-gpu")
   options.add_argument("--no-sandbox")
   options.add_argument("--disable-dev-shm-usage")
   
   driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
   driver.get("http://www.google.com")
   screenshot = driver.save_screenshot("test_screenshot.png")
   driver.quit()
   
   if screenshot:
      print("ChromeDriver is working and accessible by the Celery worker.")
   else:
      print("Failed to take a screenshot. Check ChromeDriver installation and configuration.")

Running this script should create a file named test_screenshot.png in the current directory if the ChromeDriver is correctly installed and accessible. If the screenshot is successfully taken, it confirms that the ChromeDriver is working properly with the Celery worker ^[1].

To continue talking to Dosu, mention @dosu.

[Habeeb556]

@dosu find the below now:

[WARNING/ForkPoolWorker-1] Failed at generating thumbnail Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line
Traceback (most recent call last):
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/screenshots.py", line 170, in compute_and_cache
   payload = self.get_screenshot(user=user, window_size=window_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/screenshots.py", line 93, in get_screenshot
   self.screenshot = driver.get_screenshot(self.url, self.element, user)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 354, in get_screenshot
   driver = self.auth(user)
            ^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 276, in auth
   driver = self.create()
            ^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 273, in create
   return driver_class(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/firefox/webdriver.py", line 201, in __init__
   super().__init__(command_executor=executor, options=options, keep_alive=True)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
   self.start_session(capabilities, browser_profile)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
   response = self.execute(Command.NEW_SESSION, parameters)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
   self.error_handler.check_response(response)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
   raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line

[dosubot[bot]]

To resolve the error related to the browser binary location for generating thumbnails using Selenium in a non-Docker setup, you need to ensure that the path to the browser binary is correctly specified. This can be done by setting the WEBDRIVER_TYPE configuration in your application.

Here is an example of how you can set the path to the Firefox binary in your configuration:

from selenium import webdriver

options = webdriver.FirefoxOptions()
options.binary_location = '/path/to/your/firefox/binary'

driver = webdriver.Firefox(options=options)

In the context of the provided code, you would need to modify the WebDriverSelenium class to include the path to the browser binary. Here is an example of how you can do this:

class WebDriverSelenium(WebDriver):
    def __init__(self, driver_type: str, window_size: WindowSize):
        self.driver_type = driver_type
        self.window_size = window_size
        self.driver = self._init_driver()

    def _init_driver(self):
        if self.driver_type == "firefox":
            options = webdriver.FirefoxOptions()
            options.binary_location = '/path/to/your/firefox/binary'
            driver = webdriver.Firefox(options=options)
        # Add other browser types if needed
        else:
            raise ValueError(f"Unsupported driver type: {self.driver_type}")
        return driver

Make sure to replace '/path/to/your/firefox/binary' with the actual path to your Firefox binary. This should resolve the error related to the browser binary location ^[1].

To continue talking to Dosu, mention @dosu.

[Habeeb556]

To conclude this thread error with Selenium:

All I need to do is install geckodriver manually since the $PATH automatically finds /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin.

No need to configure it in superset_config.py

Then, install Firefox using:

sudo yum install firefox

This should resolve the following issue:

selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line

as suggested here.

However, I've encountered another issue now:

selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=connectionFailure&u=http%3A//0.0.0.0%3A8080/login/&c=UTF-8&d=Firefox%20can%E2%80%99t%20establish%20a%20connection%20to%20the%20server%20at%200.0.0.0%3A8080.
Stacktrace:
RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:187:5
UnknownError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:736:5
checkReadyState@chrome://remote/content/marionette/navigate.sys.mjs:61:24
onNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39
emit@resource://gre/modules/EventEmitter.sys.mjs:154:20
receiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:35:25

@dosu

[dosubot[bot]]

To resolve the new Selenium error related to the connection failure when trying to generate thumbnails, you can switch from using Selenium to Playwright for generating thumbnails. This can be done by enabling the PLAYWRIGHT_REPORTS_AND_THUMBNAILS feature flag.

Here is the relevant part of the code that handles the selection between Selenium and Playwright:

def driver(self, window_size: WindowSize | None = None) -> WebDriver:
    window_size = window_size or self.window_size
    if feature_flag_manager.is_feature_enabled("PLAYWRIGHT_REPORTS_AND_THUMBNAILS"):
        return WebDriverPlaywright(self.driver_type, window_size)
    return WebDriverSelenium(self.driver_type, window_size)

To enable the PLAYWRIGHT_REPORTS_AND_THUMBNAILS feature flag, you need to set it in your configuration. This can typically be done in your application's configuration file (e.g., config.py or similar).

Example configuration:

FEATURE_FLAGS = {
    "PLAYWRIGHT_REPORTS_AND_THUMBNAILS": True,
}

By enabling this feature flag, the system will use Playwright instead of Selenium for generating thumbnails, which should resolve the connection failure issue ^[1][2].

Additionally, ensure that you have the necessary dependencies installed for Playwright. You need to install the "playwright" pip package and other system dependencies as mentioned in the Dockerfile:

pip install playwright
playwright install-deps
playwright install chromium

System dependencies include libnss3, libdbus-glib-1-2, libgtk-3-0, libx11-xcb1, libasound2, and libxtst6 ^[3][4][5].

To continue talking to Dosu, mention @dosu.

[Habeeb556]

Resolved with #9135

change WEBDRIVER_BASEURL = "http://0.0.0.0:8080/" to WEBDRIVER_BASEURL = "http://0.0.0.0:8088/" and add this to superset_config.py

Then, faced another error:

[2024-07-30 16:27:50,021: ERROR/ForkPoolWorker-4] Selenium timed out waiting for chart containers to draw at url http://0.0.0.0:8088/superset/dashboard/24/?standalone=3

And I think it is related to #29730 and resolved with this https://github.com/apache/superset/issues/29401#issuecomment-2228250274

SCREENSHOT_LOCATE_WAIT = 200  # Increase the wait time for locating elements
SCREENSHOT_LOAD_WAIT = 1200   # Increase the wait time for loading the page