[🐛 Bug]: No more UnhandledAlertException with Firefox 129 and problem with BiDi handling of confirmation dialog with Chrome

What happened?

Hi,

we are facing some problems since recent versions of Firefox and Selenium for dealing with confirmation dialog that are displayed using beforeunload events. So, first of all we have in our test base 2 different tests that are expecting such confirmation dialog to be opened: on the first test (test 1) we rely on a click on a button, and we are waiting for the confirmation dialog to be displayed, we have something like this:

driver.findElementById('#someButton').click();
// An alert should appear to ask the user if he wants to leave the page.
driver.waitUntilCondition(ExpectedConditions.alertIsPresent());
// We dismiss it so we can stay on the page and check the UI.
driver.switchTo().alert().dismiss();

In the other test (test 2), we rely on moving to another page, and we are expecting an UnhandledAlertException to be thrown, we have something like this:

try {
  // should open a confirmation modal for leaving since we didn't save
  driver.get("some url"));
  fail("A confirm alert should be triggered");
} catch (UnhandledAlertException e) {
  Alert alert = testUtils.getDriver().switchTo().alert();
  alert.dismiss(); // remain on the page
}

Note that our test setup uses capability.setCapability(CapabilityType.UNHANDLED_PROMPT_BEHAVIOUR, UnexpectedAlertBehaviour.IGNORE); and we also have options.addPreference("dom.disable_beforeunload", false); for Firefox. We are running our tests using TestContainer on Firefox and Chrome. On Chrome those tests are still working perfectly fine. On Firefox, we started to see a regression when moving from Firefox 123.0 to Firefox 124.0: the confirmation dialog stopped to appear leading to test failure. Back then I opened a first ticket on selenium-docker and then another ticket on geckodriver. I followed the comments I received on my geckodriver tickets, and I enabled BiDi for both browsers in our test framework. For Chrome it didn't changed anything at first: tests are still passing. For Firefox we waited the release of Firefox 129, and I realized it didn't changed anything.

So I digged a bit further, and I discovered that for BiDi there was another recommended way of dealing with confirmation dialog documented in https://www.selenium.dev/documentation/webdriver/bidi/w3c/browsing_context/. So I changed my test 2, to now rely on:

// this is needed because we use a RemoteWebDriver
WebDriver driver = new Augmenter().augment(driver);
try (BrowsingContextInspector inspector = new BrowsingContextInspector(driver)) {
    BrowsingContext context = new BrowsingContext(driver, driver.getWindowHandle());
    inspector.onUserPromptOpened(userPromptOpened -> {
        assertEquals(context.getId(), userPromptOpened.getBrowsingContextId());
        context.handleUserPrompt(false);
    });
    driver.get(testReference);
}

This piece of code is working, but there's a big flaw in this new design for my test: it's passing even if the confirmation dialog is not displayed. And in my case I want to ensure it's displayed.

But that's not the only issue: I then tried to refactor test 1 to use the same construct too so basically I wrote:

// this is needed because we use a RemoteWebDriver
WebDriver driver = new Augmenter().augment(driver);
try (BrowsingContextInspector inspector = new BrowsingContextInspector(driver)) {
    BrowsingContext context = new BrowsingContext(driver, driver.getWindowHandle());
    inspector.onUserPromptOpened(userPromptOpened -> {
        assertEquals(context.getId(), userPromptOpened.getBrowsingContextId());
        context.handleUserPrompt(false);
    });
    driver.findElementById('#someButton').click();
}

and this time it doesn't work anymore with Chrome: when I use this test with Chrome I'm getting a UnhandledAlertException being thrown. And this sounds very close to the problem already reported in https://github.com/mozilla/geckodriver/issues/2182#issuecomment-2263607428.

So for me there's now 3 things to clarify:

why don't we get at all any UnhandledAlertException with Firefox anymore?
why Chrome is triggering an UnhandledAlertException when a confirmation dialog is opened while there's a onUserPromptOpened listener added?
what's the recommended way nowadays to deal with the confirmation dialog in Selenium?

I've performed all my tests using Selenium 4.23.1 (note that I just noticed Selenium 4.24.0 release, I gave it a quick try for the new tests without any difference).

How can we reproduce the issue?

import org.junit.Test;
import org.openqa.selenium.Alert;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.time.Duration;

public class FirefoxTest {
    @Test
    public void validate() throws InterruptedException {
        FirefoxOptions firefoxOptions = new FirefoxOptions();
        firefoxOptions.enableBiDi();
        WebDriver webDriver = new FirefoxDriver(firefoxOptions);
        webDriver.get("https://www.selenium.dev/blog/2024/selenium-4-21-released/");
        ((JavascriptExecutor) webDriver).executeScript("window.onbeforeunload = function () { return false; }");
        webDriver.get("https://google.com");
        WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(5));
        Alert alert = wait.until(ExpectedConditions.alertIsPresent());
        System.err.println("alert text: " + alert.getText());
        alert.accept();
        webDriver.quit();
    }
}

Relevant log output

N/A

Operating System

Ubuntu

Selenium version

Java 4.23.1

What are the browser(s) and version(s) where you see this issue?

Firefox 129.0

What are the browser driver(s) and version(s) where you see this issue?

GeckoDriver 0.34.0

Are you using Selenium Grid?

No response

SeleniumHQ / selenium