search

bonigarcia / webdrivermanager

Automated driver management and other helper features for Selenium WebDriver in Java
https://bonigarcia.dev/webdrivermanager/
Apache License 2.0
2.56k stars 674 forks source link

Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver #1154

Closed kazurayam closed 11 months ago

kazurayam commented 11 months ago

I have mac M1 chip, 8GB memory, macOS 12.9, JDK 17, Gradle 8.4, Firefox 119.0

Description of the problem:

I have cloned the https://github.com/bonigarcia/selenium-webdriver-java master branch.

I ran https://github.com/kazurayam/selenium-webdriver-java/blob/master/selenium-webdriver-junit4/src/test/java/io/github/bonigarcia/webdriver/junit4/ch02/helloworld/HelloWorldFirefoxJUnit4Test.java

As you see in the build.gradle, Selenium-java version is 4.13.0 (I intentionally downgraded from 4.14.1), WebDriverManager version is 5.5.3.

I got the following error:

io.github.bonigarcia.wdm.config.WebDriverManagerException: io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver" type="io.github.bonigarcia.wdm.config.WebDriverManagerException">io.github.bonigarcia.wdm.config.WebDriverManagerException: io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
    at app//io.github.bonigarcia.wdm.WebDriverManager.handleException(WebDriverManager.java:1409)
    at app//io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:1183)
    at app//io.github.bonigarcia.wdm.WebDriverManager.fallback(WebDriverManager.java:1425)
    at app//io.github.bonigarcia.wdm.WebDriverManager.handleException(WebDriverManager.java:1402)
    at app//io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:1183)
    at app//io.github.bonigarcia.wdm.WebDriverManager.setup(WebDriverManager.java:425)
    at app//io.github.bonigarcia.webdriver.junit4.ch02.helloworld.HelloWorldFirefoxJUnit4Test.setupClass(HelloWorldFirefoxJUnit4Test.java:41)
    at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base@17.0.8/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@17.0.8/java.lang.reflect.Method.invoke(Method.java:568)
    at app//org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at app//org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at app//org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at app//org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
    at app//org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at app//org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at app//org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40)
    at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
    at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@17.0.8/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base@17.0.8/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@17.0.8/java.lang.reflect.Method.invoke(Method.java:568)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at jdk.proxy1/jdk.proxy1.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
    at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
    at app//io.github.bonigarcia.wdm.online.HttpClient.execute(HttpClient.java:177)
    at app//io.github.bonigarcia.wdm.WebDriverManager.openGitHubConnection(WebDriverManager.java:1592)
    at app//io.github.bonigarcia.wdm.WebDriverManager.getDriversFromGitHub(WebDriverManager.java:1606)
    at app//io.github.bonigarcia.wdm.managers.FirefoxDriverManager.getDriverUrls(FirefoxDriverManager.java:102)
    at app//io.github.bonigarcia.wdm.WebDriverManager.createUrlHandler(WebDriverManager.java:1430)
    at app//io.github.bonigarcia.wdm.WebDriverManager.download(WebDriverManager.java:1258)
    at app//io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:1177)
    ... 39 more

I tried to open the https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver using Firefox browser. I successfull see the following:

[
  {
    "tag_name": "v0.33.0",
    "assets": [
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux-aarch64.tar.gz"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux32.tar.gz"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux32.tar.gz.asc"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux64.tar.gz"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-linux64.tar.gz.asc"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-macos-aarch64.tar.gz"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-macos.tar.gz"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-win-aarch64.zip"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-win32.zip"
      },
      {
        "browser_download_url": "https://github.com/mozilla/geckodriver/releases/download/v0.33.0/geckodriver-v0.33.0-win64.zip"
      }
    ]
  },
  {
    "tag_name": "v0.32.2",
    "assets": [
....

Also using Chrome browser with DevTool I tried to get https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver

Screenshot 2023-10-30 at 7 47 28

I got no problem.

QUESTION: Why do I get the HTTP 404 error from WebDriverManager?

In the commandline, I tried this:

$ curl https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver

this responded a JSON just the same as above. Therefore I believe I don't have any network issue.

kazurayam commented 11 months ago

I read the instruction by @bonigarcia in the https://github.com/bonigarcia/webdrivermanager/issues/891

so I modified the HelloWorldFirefoxJUnit4Test.java slightly

    @BeforeClass
    public static void setupClass() {
        //WebDriverManager.firefoxdriver().setup();
        WebDriverManager.firefoxdriver().useMirror().setup();
    }

https://github.com/kazurayam/selenium-webdriver-java/blob/kazurayam14/selenium-webdriver-junit4/src/test/java/io/github/bonigarcia/webdriver/junit4/ch02/helloworld/HelloWorldFirefoxJUnit4Test.java

Then the test ran successfully without "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver".

I do not yet see what's going on with the useMirror() part.

kazurayam commented 11 months ago

I experienced this "Error HTTP 404" on my MacBook Air with Chip Apple M1 , mac OS 12.7. Firefox v119.0

I have one more MacBook Air (Retina, 130inch, 2018) with Chip X86-64, mac OS 12.6.8, Firefox v118.0.2. I tried to execute the same Test : https://github.com/kazurayam/selenium-webdriver-java/blob/kazurayam14/selenium-webdriver-junit4/src/test/java/io/github/bonigarcia/webdriver/junit4/ch02/helloworld/HelloWorldFirefoxJUnit4Test.java

It silently passed without "Error HTTP 404" error on another MacBook Air X86-64 with FF v118.0.2!

kazurayam commented 11 months ago

On my MacBook Air with Apple M1 chip, macOS 12.6.8 I updated Firefox from v118.0.2 to v119.0, then tried the same test : https://github.com/kazurayam/selenium-webdriver-java/blob/kazurayam14/selenium-webdriver-junit4/src/test/java/io/github/bonigarcia/webdriver/junit4/ch02/helloworld/HelloWorldFirefoxJUnit4Test.java

The result: worked OK

> Task :selenium-webdriver-junit4:test
2023-10-30 10:26:49 [Test worker] DEBUG i.g.bonigarcia.wdm.WebDriverManager.<init>(225) - Using WebDriverManager 5.5.3
2023-10-30 10:26:50 [Test worker] DEBUG i.g.b.wdm.cache.ResolutionCache.checkKeyInResolutionCache(186) - Resolution firefox=118 in cache (valid until 10:59:08 30/10/2023 JST)
2023-10-30 10:26:50 [Test worker] DEBUG i.g.b.wdm.cache.ResolutionCache.checkKeyInResolutionCache(186) - Resolution firefox118=0.33.0 in cache (valid until 09:59:08 31/10/2023 JST)
2023-10-30 10:26:50 [Test worker] INFO  i.g.bonigarcia.wdm.WebDriverManager.resolveDriverVersion(1217) - Using geckodriver 0.33.0 (resolved driver for Firefox 118)
2023-10-30 10:26:50 [Test worker] DEBUG i.g.bonigarcia.wdm.WebDriverManager.manage(1172) - Driver geckodriver 0.33.0 found in cache
2023-10-30 10:26:50 [Test worker] INFO  i.g.bonigarcia.wdm.WebDriverManager.exportDriver(1270) - Exporting webdriver.gecko.driver as /Users/kazuakiurayama/.cache/selenium/geckodriver/mac64/0.33.0/geckodriver
2023-10-30 10:27:00 [Test worker] DEBUG i.g.b.w.j.c.h.HelloWorldFirefoxJUnit4Test.test(60) - The title of https://bonigarcia.dev/selenium-webdriver-java/ is Hands-On Selenium WebDriver with Java
io.github.bonigarcia.webdriver.junit4.ch02.helloworld.HelloWorldFirefoxJUnit4Test > test PASSED
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD SUCCESSFUL in 1m 51s
5 actionable tasks: 5 executed
10:27:03: Execution finished ':selenium-webdriver-junit4:test --tests "io.github.bonigarcia.webdriver.junit4.ch02.helloworld.HelloWorldFirefoxJUnit4Test"'.

This experiment implies 2 factors as a possible cause of the "Error HTTP 404" error:

  1. Apple X86-64 Chip vs M1 Chip
  2. macOS 12.68 vs 12.70

... umn, highly unlikely

kazurayam commented 11 months ago

I forked the original webdrivermanager repository into my private GitHub account. I cloned the repository onto my MacBook Air with x86-64Arm M1 Chip, 8GB memory, JDK17, Gradle8.4, Maven 3.9.5, Firefox v119. I checked out the master branch. I tried to execute the tests, and got a lot of errors. 

[ERROR] Tests run: 179, Failures: 7, Errors: 23, Skipped: 17

Among others, the "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver" error occured repeatedly. 

$ cd webdrivermanager
$ mvn test
...
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   CliResolveDriverTest.testCli:53 
Expecting value to be true but was false
[ERROR]   CliResolveDriverTest.testCli:53 
Expecting value to be true but was false
[ERROR]   DockerChromeRecordingTest.test:65 
Expecting actual not to be null
[ERROR]   DockerChromeVncTest.test:56 
Expecting actual not to be null
[ERROR]   ServerResolverTest.testServerResolver:85 
expected: 200
 but was: 500
[ERROR]   ServerResolverTest.testServerResolver:85 
expected: 200
 but was: 500
[ERROR]   ServerResolverTest.testServerResolver:85 
expected: 200
 but was: 500
[ERROR] Errors: 
[ERROR]   CacheTest.testCache:61 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   DockerChromeVersionsTest.test:38 » WebDriverManager There was an error creating WebDriver object for Chrome
[ERROR]   DockerCustomImageTest.test:71 NullPointer Cannot invoke "java.net.URL.openConnection()" because "dockerSessionUrl" is null
[ERROR]   DockerExecCommandInContainerTest.test:55 » NullPointer Cannot invoke "com.github.dockerjava.api.DockerClient.execCreateCmd(String)" because "this.dockerClient" is null
[ERROR]   DockerFirefoxDevTest.setupTest:37 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   DockerSeveralBrowsersDifferentVersionsTest.test:60->exercise:80 NullPointer Cannot invoke "java.net.URL.openConnection()" because "dockerSessionUrl" is null
[ERROR]   DockerSeveralBrowsersTest.test:61->exercise:81 NullPointer Cannot invoke "java.net.URL.openConnection()" because "dockerSessionUrl" is null
[ERROR]   Firefox90Test.test:38 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   Firefox90Test.test:38 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   Firefox90Test.test:38 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   FirefoxArmTest.testFirefoxArm:44->checkArm:52 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   ForceDownloadTest.testForceDownload:46 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   ForceError403Test.test403:47 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   GenericTest.test:44 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   IExplorerArchTest.testIExplorer32:44 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/selenium
[ERROR]   IExplorerArchTest.testIExplorer64:50 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/selenium
[ERROR]   IExplorerVersionTest.testIExplorerLatest:44 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/selenium
[ERROR]   IExplorerVersionTest.testIExplorerVersion:50 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/selenium
[ERROR]   WebDriverTest.testWebDriver:43 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   BrowserVersionTest.testFirefoxVersion:50 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   IgnoredVersionTest.ignoredVersionsFirefox:64 » WebDriverManager io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   VersionsTest.testChromeDriverVersions:49 » WebDriverManager Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
[ERROR]   VersionsTest.testChromeDriverVersions:49 » WebDriverManager Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/operadriver
[INFO] 
[ERROR] Tests run: 179, Failures: 7, Errors: 23, Skipped: 17
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21:12 min
[INFO] Finished at: 2023-10-30T21:38:05+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.1:test (default-test) on project webdrivermanager: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/kazuakiurayama/github/webdrivermanager/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
:~/github/webdrivermanager (master)
$

On the other hand, I did the same on Windows 10 PC with JDK17, Gradle 8.4, Maven 3.9.5, Firefox v119. I did NOT see the "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver" message at all.

It seems that HttpClient class of the WebDriverManager works wrong on Mac as you see 

Caused by: io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver
    at app//io.github.bonigarcia.wdm.online.HttpClient.execute(HttpClient.java:177)
    at app//io.github.bonigarcia.wdm.WebDriverManager.openGitHubConnection(WebDriverManager.java:1592)
    at app//io.github.bonigarcia.wdm.WebDriverManager.getDriversFromGitHub(WebDriverManager.java:1606)
    at app//io.github.bonigarcia.wdm.managers.FirefoxDriverManager.getDriverUrls(FirefoxDriverManager.java:102)
    at app//io.github.bonigarcia.wdm.WebDriverManager.createUrlHandler(WebDriverManager.java:1430)
    at app//io.github.bonigarcia.wdm.WebDriverManager.download(WebDriverManager.java:1258)
    at app//io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:1177)

but the same code works fine on Windows.

Why?

kazurayam commented 11 months ago

I wanted to see all the HTTP request and response exchanged between the test class and the target URL. I would use BrowserMobProxy.

...

I created https://github.com/kazurayam/selenium-webdriver-java/blob/kazurayam14/selenium-webdriver-junit4/src/test/java/io/github/bonigarcia/webdriver/junit4/ch02/helloworld/HelloWorldFirefoxWithHARTest.java

It produced a HAR file, which contained the following messages:

Screenshot 2023-10-30 at 22 40 27

The information contained in this HAR file is not very useful. I wanted to see the HTTP request from the WebDriverManager class to the URL https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver. But the message was not recorded in the HAR.

This issue is a difficult one.

kazurayam commented 11 months ago

I found this:

https://bonigarcia.dev/webdrivermanager/#known-issues

  1. Known Issues 7.1. HTTP response code 403 Some of the drivers (e.g., geckodriver or operadriver) are hosted on GitHub. When external clients (like WebDriverManager) make many consecutive requests to GitHub, and due to its traffic rate limit, it eventually responds with an HTTP 403 error (forbidden), as follows:
io.github.bonigarcia.wdm.config.WebDriverManagerException: Error HTTP 403 executing https://api.github.com/repos/mozilla/geckodriver/releases
at io.github.bonigarcia.wdm.online.HttpClient.execute(HttpClient.java:172)
at io.github.bonigarcia.wdm.WebDriverManager.openGitHubConnection(WebDriverManager.java:1266)
at io.github.bonigarcia.wdm.WebDriverManager.getDriversFromGitHub(WebDriverManager.java:1280)
at io.github.bonigarcia.wdm.managers.FirefoxDriverManager.getDriverUrls(FirefoxDriverManager.java:95)
at io.github.bonigarcia.wdm.WebDriverManager.createUrlHandler(WebDriverManager.java:1111)
at io.github.bonigarcia.wdm.WebDriverManager.download(WebDriverManager.java:959)
at io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:877)
at io.github.bonigarcia.wdm.WebDriverManager.fallback(WebDriverManager.java:1106)
at io.github.bonigarcia.wdm.WebDriverManager.handleException(WebDriverManager.java:1083)
at io.github.bonigarcia.wdm.WebDriverManager.manage(WebDriverManager.java:883)
at io.github.bonigarcia.wdm.WebDriverManager.setup(WebDriverManager.java:328)

To avoid this problem, WebDriverManager can make authenticated requests using a personal access token. See the advanced configuration section to discover how to set up this token in WebDriverManager.

As of WebDriverManager 5.3.0, this issue should not happen anymore (even without a GitHub token). To avoid the 403 error, an automated job in GitHub Actions mirrors the GitHub API responses twice a day. This mirror is used internally by WebDriverManager instead of querying the GitHub API.

kazurayam commented 11 months ago

Now I have realized that I should study more on my side. I would close this issue.

kazurayam commented 11 months ago

I found that my MacBook has the environment variable GITHUB_TOKEN. However the value of the envVar was invalid. Possibly I created a valid GitHub Personal Access Token months ago and gave the PAT as the $GITHUB_TOKEN value. And later the PAT had expired, or maybe I removed it from my GitHub account, but the envVar stayed as was created.

When the envVar GITHUB_TOKEN is defined but the value is invalid, then WebDriverManager throws an error "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver". Such insident is likely to occur, and it is not documented.

kazurayam commented 11 months ago

When I searched Google with keyword "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver", I got nothing relevant to it. So I got at a loss what to do.

I would propose here, the WebDriverManager's document https://bonigarcia.dev/webdrivermanager/#known-issues should explicitly mention the case of "7.x HTTP response code 404". The document should show the message "Error HTTP 404 executing https://raw.githubusercontent.com/bonigarcia/webdrivermanager/master/docs/mirror/geckodriver" so that the page is looked up by the search engines.

When a user encouned the HTTP 404 error, she/he already has a environment variable GITHUB_TOKEN but its value is invalid. She/he should do either of the following two:

  1. create a new valid GitHub Personal Access Token, and specify it to the envVar GITHUB_TOKEN
  2. remove the envVar GITHUB_TOKEN with invalid value; then WebDriverManager will look at the Mirror URL
kazurayam commented 11 months ago

I wrote:

It seems that HttpClient class of the WebDriverManager works wrong on Mac as you see ... but the same code works fine on Windows.

On my 2 Mac machines I had the envVar GITHUB_TOKEN with invalid value.

On my Windows machine, I had no envVar GITHUB_TOKEN defined, therefore WebDriverManager accessed the mirror URL and succeeded.

The envVar just explains what I saw.

bonigarcia commented 11 months ago

As of version 5.3.0, you don't need to use GITHUB_TOKEN anymore, so you can remove that env var (especially if it has a wrong value).