bonigarcia / selenium-jupiter

JUnit 5 extension for Selenium WebDriver
https://bonigarcia.dev/selenium-jupiter/
Apache License 2.0
235 stars 53 forks source link

Cannot get recordings on failure only to work with maven failsafe. #79

Closed MichielBugherJelli closed 3 years ago

MichielBugherJelli commented 4 years ago

Description of the problem: According to the documentation here: https://bonigarcia.github.io/selenium-jupiter/#configuration there is a property for "sel.jup.recording.when.failure"

I cannot get this to work properly. I have tried only setting sel.jup.recording.when.failure and not setting sel.jup.recording. I tried setting sel.jup.recording to false and sel.jup.recording.when.failure to true.

No matter what combination I have tried, I always have recordings in my output folder when the tests complete.

Browser and version: Latest version of Chrome in Docker.

Operating system: MacOS Catalina

Selenium-Jupiter version: 3.3.4

Selenium-Jupiter use:

sel.jup.vnc=false
sel.jup.vnc.screen.resolution=1920x1080x24
sel.jup.vnc.create.redirect.html.page=false
sel.jup.vnc.export=vnc.session.url
sel.jup.recording.when.failure=true
sel.jup.recording.video.screen.size=1920x1080
sel.jup.recording.video.frame.rate=12
sel.jup.recording.image=selenoid/video-recorder:latest-release
sel.jup.output.folder=target/failsafe-reports
sel.jup.screenshot.at.the.end.of.tests=whenfailure
sel.jup.screenshot.format=png
sel.jup.exception.when.no.driver=true
sel.jup.browser.template.json.file=classpath:browsers.json
sel.jup.default.browser=chrome-in-docker
sel.jup.default.version=latest
sel.jup.default.browser.fallback=chrome,firefox,safari,edge,phantomjs
sel.jup.default.browser.fallback.version=latest,latest,latest,latest,latest
sel.jup.remote.webdriver.wait.timeout.sec=20
sel.jup.remote.webdriver.poll.time.sec=2
sel.jup.ttl.sec=86400
sel.jup.wdm.use.preferences=true
sel.jup.browser.list.from.docker.hub=true
sel.jup.browser.session.timeout.duration=1m0s
sel.jup.browser.list.in.parallel=true
sel.jup.selenoid.image=aerokube/selenoid:1.9.3
sel.jup.selenoid.port=4444
sel.jup.selenoid.vnc.password=selenoid
sel.jup.selenoid.tmpfs.size=128m
sel.jup.novnc.image=psharkey/novnc:3.3-t6
sel.jup.novnc.port=8080
sel.jup.chrome.image.format=selenoid/vnc:chrome_%s
sel.jup.chrome.first.version=48.0
sel.jup.chrome.latest.version=78.0
sel.jup.chrome.path=/
sel.jup.chrome.beta.image=elastestbrowsers/chrome:beta
sel.jup.chrome.beta.path=/wd/hub
sel.jup.chrome.unstable.image=elastestbrowsers/chrome:unstable
sel.jup.chrome.unstable.path=/wd/hub
sel.jup.firefox.image.format=selenoid/vnc:firefox_%s
sel.jup.firefox.first.version=3.6
sel.jup.firefox.latest.version=70.0
sel.jup.firefox.path=/wd/hub
sel.jup.firefox.beta.image=elastestbrowsers/firefox:beta
sel.jup.firefox.beta.path=/wd/hub
sel.jup.firefox.unstable.image=elastestbrowsers/firefox:nightly
sel.jup.firefox.unstable.path=/wd/hub
sel.jup.opera.image.format=selenoid/vnc:opera_%s
sel.jup.opera.first.version=33.0
sel.jup.opera.latest.version=64.0
sel.jup.opera.path=/
sel.jup.opera.binary.path.linux=/usr/bin/opera
sel.jup.opera.binary.path.win=C:\\Program Files\\Opera\\launcher.exe
sel.jup.opera.binary.path.mac=/Applications/Opera.app/Contents/MacOS/Opera
sel.jup.edge.image=windows/edge:%s
sel.jup.edge.path=/
sel.jup.edge.latest.version=18
sel.jup.iexplorer.image=windows/ie:%s
sel.jup.iexplorer.path=/
sel.jup.iexplorer.latest.version=11
sel.jup.docker.wait.timeout.sec=20
sel.jup.docker.poll.time.ms=200
sel.jup.docker.default.socket=/var/run/docker.sock
sel.jup.docker.hub.url=https://hub.docker.com/
sel.jup.docker.stop.timeout.sec=5
sel.jup.docker.api.version=1.35
sel.jup.docker.network=bridge
sel.jup.docker.timezone=America/Boise
sel.jup.docker.lang=en
sel.jup.docker.startup.timeout.duration=3m
sel.jup.server.port=4042
sel.jup.server.path=/wd/hub
sel.jup.server.timeout.sec=180

Selenium-Jupiter traces: I don't see any relevant traces to this issue

Error trace: None that I know of.

bonigarcia commented 3 years ago

Recordings work only when using Docker browsers. I have just executed this example and it works fine:

import static io.github.bonigarcia.seljup.BrowserType.CHROME;

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.openqa.selenium.WebDriver;

import io.github.bonigarcia.seljup.DockerBrowser;
import io.github.bonigarcia.seljup.SeleniumExtension;

public class DockerRecordingWhenFailureTest {

    @RegisterExtension
    static SeleniumExtension seleniumExtension = new SeleniumExtension();

    @BeforeAll
    static void setup() {
        seleniumExtension.getConfig().setRecordingWhenFailure(true);
    }

    @Test
    public void test(@DockerBrowser(type = CHROME) WebDriver driver) {
        driver.get("https://bonigarcia.github.io/selenium-jupiter/");
        Assertions.fail("Forced fail");
    }

}

Can you try it, please?

MichielBugherJelli commented 3 years ago

@bonigarcia I actually have the opposite problem of what your test is doing.

This is the important part:

No matter what combination I have tried, I always have recordings in my output folder when the tests complete.

When a test passes, I always have a recording. I always get recordings and screenshots regardless of the outcome. I expect to not get a recording if the test passes.

bonigarcia commented 3 years ago

Ah, ok, I misunderstood. I have just tried this one (in Windows right now):

import static io.github.bonigarcia.seljup.BrowserType.CHROME;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.RegisterExtension;
import org.openqa.selenium.WebDriver;

import io.github.bonigarcia.seljup.DockerBrowser;
import io.github.bonigarcia.seljup.SeleniumExtension;

public class DockerRecordingWhenFailureTest {

    @RegisterExtension
    static SeleniumExtension seleniumExtension = new SeleniumExtension();

    @BeforeAll
    static void setup() {
        seleniumExtension.getConfig().setRecordingWhenFailure(true);
    }

    @Test
    public void test(@DockerBrowser(type = CHROME) WebDriver driver) {
        driver.get("https://bonigarcia.github.io/selenium-jupiter/");
    }

}

Configuring the Selenium-Jupiter loginng at TRACE level, we can see what is happening:

2020-09-09 20:00:49 [main] TRACE i.g.b.seljup.SeleniumExtension.resolveParameter(176) -- Context id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest]/[method:test(org.openqa.selenium.WebDriver)]
2020-09-09 20:00:49 [main] TRACE i.g.b.seljup.SeleniumExtension.isSingleSession(419) -- Single session false
2020-09-09 20:00:49 [main] TRACE i.g.bonigarcia.seljup.config.Config.getProperty(313) -- Property key sel.jup.selenium.server.url not found, using default value
2020-09-09 20:00:49 [main] TRACE i.g.b.seljup.SeleniumExtension.initHandlerForDocker(376) -- Adding new container map (context id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest]/[method:test(org.openqa.selenium.WebDriver)]) (handler io.github.bonigarcia.seljup.handler.RemoteDriverHandler@397fbdb)
2020-09-09 20:00:49 [main] TRACE i.g.bonigarcia.seljup.config.Config.getProperty(313) -- Property key sel.jup.docker.server.url not found, using default value
2020-09-09 20:00:49 [main] TRACE i.g.bonigarcia.seljup.config.Config.getProperty(313) -- Property key sel.jup.docker.request.entity.processing not found, using default value
2020-09-09 20:00:49 [main] TRACE i.g.bonigarcia.seljup.DockerService.<init>(95) -- Using RequestEntityProcessing CHUNKED
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.SeleniumExtension.putDriverHandlerInMap(272) -- Put driver handler io.github.bonigarcia.seljup.handler.RemoteDriverHandler@397fbdb in map (context id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest]/[method:test(org.openqa.selenium.WebDriver)], new context id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest])
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.SeleniumExtension.putDriverHandlerInMap(287) -- Adding io.github.bonigarcia.seljup.handler.RemoteDriverHandler@397fbdb to handler new map (id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest]/[method:test(org.openqa.selenium.WebDriver)])
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.InternalPreferences.checkKeyInPreferences(120) -- Preference found CHROME=85.0 (valid until 2020-09-10 18:27:30)
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.SurefireReports.getOutputFolder(58) -- Output folder .
2020-09-09 20:00:50 [main] TRACE i.g.b.s.handler.DockerDriverHandler.getCapabilities(508) -- Using Capabilities {browserName: chrome, enableVideo: true, goog:chromeOptions: {args: [], extensions: []}, platform: ANY, version: , videoFrameRate: 12, videoScreenSize: 1024x768}
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.config.Config.getProperty(313) -- Property key sel.jup.selenium.server.url not found, using default value
2020-09-09 20:00:50 [main] INFO  i.g.b.s.handler.DockerDriverHandler.startDockerBrowser(720) -- Using CHROME version 85.0 (latest)
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.InternalPreferences.checkKeyInPreferences(120) -- Preference found selenoid/vnc:chrome_85.0=pulled (valid until 2020-09-10 18:28:53)
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.InternalPreferences.checkKeyInPreferences(120) -- Preference found aerokube/selenoid:1.9.3=pulled (valid until 2020-09-10 18:28:55)
2020-09-09 20:00:50 [main] TRACE i.g.b.seljup.InternalPreferences.checkKeyInPreferences(120) -- Preference found selenoid/video-recorder:latest-release=pulled (valid until 2020-09-10 18:30:03)
2020-09-09 20:00:50 [main] TRACE i.g.b.s.handler.DockerDriverHandler.getDockerPath(1147) -- The path of file . in Docker format is /c/Users/boni/dev/selenium-jupiter-examples/.
2020-09-09 20:00:50 [main] TRACE i.g.b.s.handler.DockerDriverHandler.getDockerBrowserCount(1049) -- Number of required Docker browser(s): 2
2020-09-09 20:00:50 [main] TRACE i.g.b.s.handler.DockerDriverHandler.getDockerPath(1147) -- The path of file . in Docker format is /c/Users/boni/dev/selenium-jupiter-examples/.
2020-09-09 20:00:50 [main] INFO  i.g.bonigarcia.seljup.DockerService.startContainer(118) -- Starting Docker container aerokube/selenoid:1.9.3
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(132) -- Using network: bridge
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(138) -- Using port bindings: {4444=[PortBinding{hostIp=0.0.0.0, hostPort=}]}
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(144) -- Using binds: [/var/run/docker.sock:/var/run/docker.sock, /c/Users/boni/dev/selenium-jupiter-examples/.:/opt/selenoid/video]
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(149) -- Using envs: [DOCKER_API_VERSION=1.35, TZ=Europe/Madrid, LANG=en, OVERRIDE_VIDEO_OUTPUT_DIR=/c/Users/boni/dev/selenium-jupiter-examples/.]
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(154) -- Using cmd: [sh, -c, mkdir -p /etc/selenoid/; echo '{"chrome":{"default":"85.0","versions":{"85.0":{"image":"selenoid/vnc:chrome_85.0","port":"4444","path":"/","tmpfs":{"/tmp":"size=128m"},"env":["DOCKER_API_VERSION=1.35","TZ=Europe/Madrid","LANG=en"],"volumes":[]}}}}' > /etc/selenoid/browsers.json; /usr/bin/selenoid -listen :4444 -service-startup-timeout 3m -conf /etc/selenoid/browsers.json -video-output-dir /opt/selenoid/video/ -timeout 1m0s -container-network bridge -limit 2]
2020-09-09 20:00:50 [main] TRACE i.g.bonigarcia.seljup.DockerService.startContainer(159) -- Using entryPoint: []
2020-09-09 20:00:52 [main] TRACE i.g.bonigarcia.seljup.config.Config.getProperty(313) -- Property key sel.jup.docker.host not found, using default value
2020-09-09 20:00:52 [main] TRACE i.g.bonigarcia.seljup.DockerService.getBindPort(197) -- Port list {4444/tcp=[PortBinding{hostIp=0.0.0.0, hostPort=32779}]} -- Exposed port 4444/tcp = [PortBinding{hostIp=0.0.0.0, hostPort=32779}]
2020-09-09 20:00:52 [main] TRACE i.g.b.s.handler.DockerDriverHandler.startSelenoidContainer(819) -- Selenium server URL http://localhost:32779/wd/hub
2020-09-09 20:00:52 [main] TRACE i.g.b.s.handler.DockerDriverHandler.getDriverForBrowser(284) -- Creating webdriver for CHROME  (http://localhost:32779/wd/hub)
2020-09-09 20:00:52 [main] DEBUG i.g.b.seljup.WebDriverCreator.createRemoteWebDriver(66) -- Creating WebDriver for CHROME at http://localhost:32779/wd/hub with Capabilities {browserName: chrome, enableVideo: true, goog:chromeOptions: {args: [], extensions: []}, platform: ANY, version: , videoFrameRate: 12, videoScreenSize: 1024x768}
sep 09, 2020 8:00:53 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFORMACIÓN: Detected dialect: W3C
2020-09-09 20:00:53 [main] TRACE i.g.b.seljup.WebDriverCreator.createRemoteWebDriver(83) -- Created WebDriver object (session id 80cfeed7286a2fc76e7bf8e62852a8a2)
2020-09-09 20:00:54 [main] TRACE i.g.b.seljup.SeleniumExtension.isSingleSession(419) -- Single session false
2020-09-09 20:00:54 [main] TRACE i.g.b.seljup.SeleniumExtension.teardown(430) -- After each for context id [engine:junit-jupiter]/[class:io.github.bonigarcia.seljup.test.docker.DockerRecordingWhenFailureTest]/[method:test(org.openqa.selenium.WebDriver)]
2020-09-09 20:00:54 [main] TRACE i.g.b.seljup.SeleniumExtension.quitWebDriver(469) -- Quit RemoteWebDriver: chrome on LINUX (80cfeed7286a2fc76e7bf8e62852a8a2) true
2020-09-09 20:00:54 [main] DEBUG i.g.b.s.handler.DockerDriverHandler.waitForRecording(1173) -- Waiting for recording to be available
2020-09-09 20:00:54 [main] TRACE i.g.b.s.handler.DockerDriverHandler.waitForRecording(1180) -- Recording .\80cfeed7286a2fc76e7bf8e62852a8a2.mp4 not present ... waiting 200 ms
2020-09-09 20:00:54 [main] TRACE i.g.b.s.handler.DockerDriverHandler.waitForRecording(1180) -- Recording .\80cfeed7286a2fc76e7bf8e62852a8a2.mp4 not present ... waiting 200 ms
2020-09-09 20:00:54 [main] TRACE i.g.b.s.handler.DockerDriverHandler.waitForRecording(1180) -- Recording .\80cfeed7286a2fc76e7bf8e62852a8a2.mp4 not present ... waiting 200 ms
2020-09-09 20:00:55 [main] TRACE i.g.b.s.handler.DockerDriverHandler.waitForRecording(1180) -- Recording .\80cfeed7286a2fc76e7bf8e62852a8a2.mp4 not present ... waiting 200 ms
2020-09-09 20:00:55 [main] TRACE i.g.b.s.handler.DockerDriverHandler.waitForRecording(1193) -- Renaming .\80cfeed7286a2fc76e7bf8e62852a8a2.mp4 to test_arg0_CHROME_85.0_80cfeed7286a2fc76e7bf8e62852a8a2.mp4
2020-09-09 20:00:55 [main] TRACE i.g.b.s.handler.DockerDriverHandler.cleanup(547) -- Deleting .\test_arg0_CHROME_85.0_80cfeed7286a2fc76e7bf8e62852a8a2.mp4 (recordingWhenFailure=true)
2020-09-09 20:00:55 [main] TRACE i.g.b.s.handler.DockerDriverHandler.stopContainers(573) -- There are 1 container(s): {aerokube/selenoid:1.9.3=io.github.bonigarcia.seljup.DockerContainer@6d64b553}
2020-09-09 20:00:55 [pool-2-thread-1] INFO  i.g.bonigarcia.seljup.DockerService.stopAndRemoveContainer(243) -- Stopping Docker container aerokube/selenoid:1.9.3
2020-09-09 20:00:55 [pool-2-thread-1] TRACE i.g.bonigarcia.seljup.DockerService.stopContainer(259) -- Stopping container e531cbb1dde3f9beae7fe1803c1dad8bf54dfa38858481bf85fa804085d68d5e (timeout 5 seconds)
2020-09-09 20:00:55 [pool-2-thread-1] TRACE i.g.bonigarcia.seljup.DockerService.removeContainer(267) -- Removing container e531cbb1dde3f9beae7fe1803c1dad8bf54dfa38858481bf85fa804085d68d5e

... the important line is:

2020-09-09 20:00:55 [main] TRACE i.g.b.s.handler.DockerDriverHandler.cleanup(547) -- Deleting .\test_arg0_CHROME_85.0_80cfeed7286a2fc76e7bf8e62852a8a2.mp4 (recordingWhenFailure=true)

I suppose in Mac this is not working for some reason. Could you try to get the Selenium-Jupiter traces (at TRACE level). Since I can check it in Windows and Linux, but not Mac.

MichielBugherJelli commented 3 years ago

Interesting, I do not see an instance of i.g.b.s.handler.DockerDriverHandler.cleanup being logged.

2020-09-10 11:47:25.646 [ForkJoinPool-1-worker-2] TRACE i.g.b.seljup.SeleniumExtension - After each for context id [engine:junit-jupiter]/[class:com.jelli.spearmint.test.login.LoginIT]/[test-template:iCanLoginAfterFailedAttempt()]/[test-template-invocation:#1]
2020-09-10 11:47:25.646 [ForkJoinPool-1-worker-2] TRACE i.g.b.seljup.SeleniumExtension - Quit RemoteWebDriver: chrome on LINUX (c166a25ded8ee62f240fa9791d4411f8) true
2020-09-10 11:47:25.706 [ForkJoinPool-1-worker-2] DEBUG i.g.b.s.handler.DockerDriverHandler - Waiting for recording to be available
2020-09-10 11:47:25.706 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Recording target/failsafe-reports/c166a25ded8ee62f240fa9791d4411f8.mp4 not present ... waiting 200 ms
2020-09-10 11:47:25.911 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Recording target/failsafe-reports/c166a25ded8ee62f240fa9791d4411f8.mp4 not present ... waiting 200 ms
2020-09-10 11:47:26.111 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Recording target/failsafe-reports/c166a25ded8ee62f240fa9791d4411f8.mp4 not present ... waiting 200 ms
2020-09-10 11:47:26.314 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Recording target/failsafe-reports/c166a25ded8ee62f240fa9791d4411f8.mp4 not present ... waiting 200 ms
2020-09-10 11:47:26.517 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Renaming target/failsafe-reports/c166a25ded8ee62f240fa9791d4411f8.mp4 to iCanLoginAfterFailedAttempt_arg1_CHROME_85.0_c166a25ded8ee62f240fa9791d4411f8.mp4
2020-09-10 11:47:26.522 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - Deleting iCanLoginAfterFailedAttempt_arg1_CHROME_85.0_c166a25ded8ee62f240fa9791d4411f8.mp4 (recordingWhenFailure=true)
2020-09-10 11:47:26.523 [ForkJoinPool-1-worker-2] WARN  i.g.b.s.handler.DockerDriverHandler - Exception waiting for recording iCanLoginAfterFailedAttempt_arg1_CHROME_85.0_c166a25ded8ee62f240fa9791d4411f8.mp4
2020-09-10 11:47:26.523 [ForkJoinPool-1-worker-2] TRACE i.g.b.s.handler.DockerDriverHandler - There are 1 container(s): {aerokube/selenoid:1.9.3=io.github.bonigarcia.seljup.DockerContainer@1f09e163}
2020-09-10 11:47:26.524 [pool-2-thread-1] INFO  i.g.bonigarcia.seljup.DockerService - Stopping Docker container aerokube/selenoid:1.9.3
2020-09-10 11:47:26.525 [pool-2-thread-1] TRACE i.g.bonigarcia.seljup.DockerService - Stopping container 7b6ec3a00df571348cd8b4febc25d3d0c37c6ad1aa14a0c4a7830140d74a662b (timeout 5 seconds)
2020-09-10 11:47:26.740 [pool-2-thread-1] TRACE i.g.bonigarcia.seljup.DockerService - Removing container 7b6ec3a00df571348cd8b4febc25d3d0c37c6ad1aa14a0c4a7830140d74a662b
MichielBugherJelli commented 3 years ago

I'm guessing this is the offending issue:

2020-09-10 11:47:26.523 [ForkJoinPool-1-worker-2] WARN  i.g.b.s.handler.DockerDriverHandler - Exception waiting for recording iCanLoginAfterFailedAttempt_arg1_CHROME_85.0_c166a25ded8ee62f240fa9791d4411f8.mp4

Unfortunately, I don't see any other useful information other than that. The file names are exactly the same and it definitely exists under target/failsafe-reports

MichielBugherJelli commented 3 years ago

Okay, I spent a little time debugging the issue. It looks like the issue is on line 548 of DockerDriverHandler.java. The recordingFile field has the full path in the filename. When toPath() is called, the target/failsafe-reports portion is removed.

MichielBugherJelli commented 3 years ago

I pulled down master to make the fix, but I noticed it had already been fixed in: https://github.com/bonigarcia/selenium-jupiter/commit/6c9bf4fd44d06b47de52fcafa655b8c2405577d2

Is there any eta on a release that will include this commit?

MichielBugherJelli commented 3 years ago

Going to go ahead and close this for now, since it is fixed in master.