bazelbuild / rules_webtesting

Bazel rules to allow testing against a browser with WebDriver.
Apache License 2.0
96 stars 56 forks source link

firefox times out in Github Actions #439

Open mgred opened 2 years ago

mgred commented 2 years ago

I've an Angular app closly modeled after the example in rules_nodejs that uses rules_webtesting for unit tests. Everything works fine locally, with Firefox and Chrome enabled. When I run the same tests in a Github Action Workflow (bazel test //...), the firefox-based test suites time out complaining: FirefoxHeadless has not captured in 60000 ms, killing. (full error log attached).

Does anybody know about this problem or has a clue what might go wrong here? I'm thankfull for any advice :pray:

Error Log ``` 2021/12/08 18:38:44 Downloading https://releases.bazel.build/4.1.0/release/bazel-4.1.0-linux-x86_64... Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: 7c70f3aa-e82b-42b0-a492-729a0bc9315e Loading: Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 5 packages loaded currently loading: api/metadata/src ... (13 packages) Loading: 5 packages loaded currently loading: api/metadata/src ... (13 packages) Loading: 5 packages loaded currently loading: api/metadata/src ... (13 packages) Loading: 5 packages loaded currently loading: api/metadata/src ... (13 packages) Loading: 5 packages loaded currently loading: api/metadata/src ... (13 packages) Loading: 7 packages loaded currently loading: api/metadata/src ... (11 packages) Analyzing: 78 targets (18 packages loaded, 0 targets configured) Analyzing: 78 targets (287 packages loaded, 14670 targets configured) Analyzing: 78 targets (396 packages loaded, 15854 targets configured) INFO: Analyzed 78 targets (1134 packages loaded, 53038 targets configured). INFO: Found 71 targets and 7 test targets... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt [62 / 71] Compiling TypeScript project //cdk/lib:lib [tsc -p cdk/tsconfig.json]; 8s remote-cache, linux-sandbox ... (3 actions running) [138 / 147] Compiling TypeScript project //webapp/src/app:app [tsc -p webapp/tsconfig.json]; 3s remote-cache, linux-sandbox ... (3 actions, 2 running) INFO: From Action webapp/src/index.html: html-insert-assets: Failed to find index.dev.js to stamp. Will fallback to timestamp. INFO: From Action webapp/src/_prodapp/webapp/src/index.html: html-insert-assets: Failed to find index.prod.js to stamp. Will fallback to timestamp. WARNING: /home/runner/work/data-catalog/data-catalog/webapp/src/BUILD.bazel:128:12: output 'webapp/src/pwa-config' of //webapp/src:pwa is a directory; dependency checking of directories is unsound [252 / 257] 1 / 7 tests; Splitting Javascript webapp/src/initialize_testbed.js webapp/src/app/table/table.component.spec.js [esbuild]; 1s remote-cache, linux-sandbox ... (3 actions, 2 running) [280 / 282] 2 / 7 tests; Testing //webapp/src/app:test.suite_firefox-local; 27s remote-cache, linux-sandbox ... (4 actions running) [280 / 282] 2 / 7 tests; Testing //webapp/src/app:test.suite_firefox-local; 55s remote-cache, linux-sandbox ... (4 actions running) [280 / 282] 2 / 7 tests; Testing //webapp/src/app:test.suite_firefox-local; 102s remote-cache, linux-sandbox ... (4 actions running) [280 / 282] 2 / 7 tests; Testing //webapp/src/app:test.suite_firefox-local; 156s remote-cache, linux-sandbox ... (4 actions running) FAIL: //webapp/src/app:test.suite_firefox-local (see /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.log) INFO: From Testing //webapp/src/app:test.suite_firefox-local: ==================== Test output for //webapp/src/app:test.suite_firefox-local: TEST_UNUSED_RUNFILES_LOG_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.unused_runfiles_log TEST_BINARY=webapp/src/app/test.suite_firefox-local.sh TEST_TEMPDIR=test_tempdir.71sER1 TEST_WORKSPACE=data_catalog RUNFILES_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/test.suite_firefox-local.sh.runfiles TEST_UNDECLARED_OUTPUTS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.outputs *** RUN_UNDER_RUNFILES=1 TZ=UTC TEST_TMPDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/_tmp/f0c8d0e5f3a091cd44814ad9179d3537 TMPDIR=/tmp XML_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.xml WEB_TEST_METADATA=data_catalog/webapp/src/app/test.suite_firefox-local.gen.json TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.outputs_manifest TEST_PREMATURE_EXIT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.exited_prematurely TEST_INFRASTRUCTURE_FAILURE_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.infrastructure_failure USER=runner PYTHON_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/test.suite_firefox-local.sh.runfiles TEST_SRCDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/test.suite_firefox-local.sh.runfiles SHLVL=1 _RLOCATION_ISABS_PATTERN=^/[^/].* PATH=.:/bin:/usr/bin:/usr/local/bin GTEST_TMP_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/_tmp/f0c8d0e5f3a091cd44814ad9179d3537 TEST_WARNINGS_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.warnings TEST_TARGET=//webapp/src/app:test.suite_firefox-local JAVA_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/test.suite_firefox-local.sh.runfiles TEST_SIZE=large EXPERIMENTAL_SPLIT_XML_GENERATION=1 TEST_TIMEOUT=900 TEST_LOGSPLITTER_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.raw_splitlogs/test.splitlogs OLDPWD=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/52/execroot/data_catalog BASH_FUNC_rlocation%%=() { if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): start" 1>&2; fi; if [[ "$1" =~ $_RLOCATION_ISABS_PATTERN ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): absolute path, return" 1>&2; fi; echo "$1"; else if [[ "$1" == ../* || "$1" == */.. || "$1" == ./* || "$1" == */./* || "$1" == "*/." || "$1" == *//* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): path is not normalized" 1>&2; fi; return 1; else if [[ "$1" == \\* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): absolute path without" "drive name" 1>&2; fi; return 1; else if [[ -e "${RUNFILES_DIR:-/dev/null}/$1" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found under RUNFILES_DIR ($RUNFILES_DIR), return" 1>&2; fi; echo "${RUNFILES_DIR}/$1"; else if [[ -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): looking in RUNFILES_MANIFEST_FILE ($RUNFILES_MANIFEST_FILE)" 1>&2; fi; local -r result=$(grep -m1 "^$1 " "${RUNFILES_MANIFEST_FILE}" | cut -d ' ' -f 2-); if [[ -e "${result:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found in manifest as ($result)" 1>&2; fi; echo "$result"; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): not found in manifest" 1>&2; fi; echo ""; fi; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: cannot look up runfile \"$1\" " "(RUNFILES_DIR=\"${RUNFILES_DIR:-}\"," "RUNFILES_MANIFEST_FILE=\"${RUNFILES_MANIFEST_FILE:-}\")" 1>&2; fi; return 1; fi; fi; fi; fi; fi } BASH_FUNC_runfiles_export_envvars%%=() { if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" && ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then return 1; fi; if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ -f "$RUNFILES_DIR/MANIFEST" ]]; then export RUNFILES_MANIFEST_FILE="$RUNFILES_DIR/MANIFEST"; else if [[ -f "${RUNFILES_DIR}_manifest" ]]; then export RUNFILES_MANIFEST_FILE="${RUNFILES_DIR}_manifest"; else export RUNFILES_MANIFEST_FILE=; fi; fi; else if [[ ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then if [[ "$RUNFILES_MANIFEST_FILE" == */MANIFEST && -d "${RUNFILES_MANIFEST_FILE%/MANIFEST}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%/MANIFEST}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else if [[ "$RUNFILES_MANIFEST_FILE" == *_manifest && -d "${RUNFILES_MANIFEST_FILE%_manifest}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%_manifest}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else export RUNFILES_DIR=; fi; fi; fi; fi } BASH_FUNC_is_absolute%%=() { [[ "$1" = /* ]] || [[ "$1" =~ ^[a-zA-Z]:[/\\].* ]] } _=/bin/printenv 08 12 2021 18:41:23.101:INFO [karma-server]: Karma v6.3.9 server started at http://localhost:9877/ 08 12 2021 18:41:23.103:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited 08 12 2021 18:41:23.111:INFO [launcher]: Starting browser FirefoxHeadless 08 12 2021 18:42:23.131:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:42:23.140:INFO [launcher]: Trying to start FirefoxHeadless again (1/2). 08 12 2021 18:43:23.171:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:43:23.177:INFO [launcher]: Trying to start FirefoxHeadless again (2/2). 08 12 2021 18:44:23.212:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:44:23.217:ERROR [launcher]: FirefoxHeadless failed 2 times (timeout). Giving up. ================================================================================ FAIL: //webapp/src/app/table:test.suite_firefox-local (see /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.log) INFO: From Testing //webapp/src/app/table:test.suite_firefox-local: ==================== Test output for //webapp/src/app/table:test.suite_firefox-local: TEST_UNUSED_RUNFILES_LOG_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.unused_runfiles_log TEST_BINARY=webapp/src/app/table/test.suite_firefox-local.sh TEST_TEMPDIR=test_tempdir.ELgo0L TEST_WORKSPACE=data_catalog RUNFILES_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/test.suite_firefox-local.sh.runfiles TEST_UNDECLARED_OUTPUTS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.outputs *** RUN_UNDER_RUNFILES=1 TZ=UTC TEST_TMPDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/_tmp/ec208004adbce0f871eddff7fc00a3c6 TMPDIR=/tmp XML_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.xml WEB_TEST_METADATA=data_catalog/webapp/src/app/table/test.suite_firefox-local.gen.json TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.outputs_manifest TEST_PREMATURE_EXIT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.exited_prematurely TEST_INFRASTRUCTURE_FAILURE_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.infrastructure_failure USER=runner PYTHON_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/test.suite_firefox-local.sh.runfiles TEST_SRCDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/test.suite_firefox-local.sh.runfiles SHLVL=1 _RLOCATION_ISABS_PATTERN=^/[^/].* PATH=.:/bin:/usr/bin:/usr/local/bin GTEST_TMP_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/_tmp/ec208004adbce0f871eddff7fc00a3c6 TEST_WARNINGS_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.warnings TEST_TARGET=//webapp/src/app/table:test.suite_firefox-local JAVA_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/test.suite_firefox-local.sh.runfiles TEST_SIZE=large EXPERIMENTAL_SPLIT_XML_GENERATION=1 TEST_TIMEOUT=900 TEST_LOGSPLITTER_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.raw_splitlogs/test.splitlogs OLDPWD=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/54/execroot/data_catalog BASH_FUNC_rlocation%%=() { if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): start" 1>&2; fi; if [[ "$1" =~ $_RLOCATION_ISABS_PATTERN ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): absolute path, return" 1>&2; fi; echo "$1"; else if [[ "$1" == ../* || "$1" == */.. || "$1" == ./* || "$1" == */./* || "$1" == "*/." || "$1" == *//* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): path is not normalized" 1>&2; fi; return 1; else if [[ "$1" == \\* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): absolute path without" "drive name" 1>&2; fi; return 1; else if [[ -e "${RUNFILES_DIR:-/dev/null}/$1" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found under RUNFILES_DIR ($RUNFILES_DIR), return" 1>&2; fi; echo "${RUNFILES_DIR}/$1"; else if [[ -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): looking in RUNFILES_MANIFEST_FILE ($RUNFILES_MANIFEST_FILE)" 1>&2; fi; local -r result=$(grep -m1 "^$1 " "${RUNFILES_MANIFEST_FILE}" | cut -d ' ' -f 2-); if [[ -e "${result:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found in manifest as ($result)" 1>&2; fi; echo "$result"; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): not found in manifest" 1>&2; fi; echo ""; fi; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: cannot look up runfile \"$1\" " "(RUNFILES_DIR=\"${RUNFILES_DIR:-}\"," "RUNFILES_MANIFEST_FILE=\"${RUNFILES_MANIFEST_FILE:-}\")" 1>&2; fi; return 1; fi; fi; fi; fi; fi } BASH_FUNC_runfiles_export_envvars%%=() { if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" && ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then return 1; fi; if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ -f "$RUNFILES_DIR/MANIFEST" ]]; then export RUNFILES_MANIFEST_FILE="$RUNFILES_DIR/MANIFEST"; else if [[ -f "${RUNFILES_DIR}_manifest" ]]; then export RUNFILES_MANIFEST_FILE="${RUNFILES_DIR}_manifest"; else export RUNFILES_MANIFEST_FILE=; fi; fi; else if [[ ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then if [[ "$RUNFILES_MANIFEST_FILE" == */MANIFEST && -d "${RUNFILES_MANIFEST_FILE%/MANIFEST}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%/MANIFEST}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else if [[ "$RUNFILES_MANIFEST_FILE" == *_manifest && -d "${RUNFILES_MANIFEST_FILE%_manifest}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%_manifest}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else export RUNFILES_DIR=; fi; fi; fi; fi } BASH_FUNC_is_absolute%%=() { [[ "$1" = /* ]] || [[ "$1" =~ ^[a-zA-Z]:[/\\].* ]] } _=/bin/printenv 08 12 2021 18:41:24.525:INFO [karma-server]: Karma v6.3.9 server started at http://localhost:9876/ 08 12 2021 18:41:24.527:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited 08 12 2021 18:41:24.530:INFO [launcher]: Starting browser FirefoxHeadless 08 12 2021 18:42:24.580:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:42:24.588:INFO [launcher]: Trying to start FirefoxHeadless again (1/2). 08 12 2021 18:43:24.632:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:43:24.637:INFO [launcher]: Trying to start FirefoxHeadless again (2/2). 08 12 2021 18:44:24.687:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:44:24.696:ERROR [launcher]: FirefoxHeadless failed 2 times (timeout). Giving up. ================================================================================ [352 / 353] 6 / 7 tests, 2 failed; Testing //webapp/src/app/table/filter:test.suite_firefox-local; 16s remote-cache, linux-sandbox [352 / 353] 6 / 7 tests, 2 failed; Testing //webapp/src/app/table/filter:test.suite_firefox-local; 106s remote-cache, linux-sandbox [352 / 353] 6 / 7 tests, 2 failed; Testing //webapp/src/app/table/filter:test.suite_firefox-local; 0s remote-cache, linux-sandbox FAIL: //webapp/src/app/table/filter:test.suite_firefox-local (see /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.log) INFO: From Testing //webapp/src/app/table/filter:test.suite_firefox-local: ==================== Test output for //webapp/src/app/table/filter:test.suite_firefox-local: TEST_UNUSED_RUNFILES_LOG_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.unused_runfiles_log TEST_BINARY=webapp/src/app/table/filter/test.suite_firefox-local.sh TEST_TEMPDIR=test_tempdir.UpDqdM TEST_WORKSPACE=data_catalog RUNFILES_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/filter/test.suite_firefox-local.sh.runfiles TEST_UNDECLARED_OUTPUTS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.outputs *** RUN_UNDER_RUNFILES=1 TZ=UTC TEST_TMPDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/_tmp/ebed677f15027bf98042737bbcde48e2 TMPDIR=/tmp XML_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.xml WEB_TEST_METADATA=data_catalog/webapp/src/app/table/filter/test.suite_firefox-local.gen.json TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.outputs_manifest TEST_PREMATURE_EXIT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.exited_prematurely TEST_INFRASTRUCTURE_FAILURE_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.infrastructure_failure USER=runner PYTHON_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/filter/test.suite_firefox-local.sh.runfiles TEST_SRCDIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/filter/test.suite_firefox-local.sh.runfiles SHLVL=1 _RLOCATION_ISABS_PATTERN=^/[^/].* PATH=.:/bin:/usr/bin:/usr/local/bin GTEST_TMP_DIR=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/_tmp/ebed677f15027bf98042737bbcde48e2 TEST_WARNINGS_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.warnings TEST_TARGET=//webapp/src/app/table/filter:test.suite_firefox-local JAVA_RUNFILES=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/bin/webapp/src/app/table/filter/test.suite_firefox-local.sh.runfiles TEST_SIZE=large EXPERIMENTAL_SPLIT_XML_GENERATION=1 TEST_TIMEOUT=900 TEST_LOGSPLITTER_OUTPUT_FILE=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.raw_splitlogs/test.splitlogs OLDPWD=/home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/sandbox/linux-sandbox/62/execroot/data_catalog BASH_FUNC_rlocation%%=() { if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): start" 1>&2; fi; if [[ "$1" =~ $_RLOCATION_ISABS_PATTERN ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): absolute path, return" 1>&2; fi; echo "$1"; else if [[ "$1" == ../* || "$1" == */.. || "$1" == ./* || "$1" == */./* || "$1" == "*/." || "$1" == *//* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): path is not normalized" 1>&2; fi; return 1; else if [[ "$1" == \\* ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: rlocation($1): absolute path without" "drive name" 1>&2; fi; return 1; else if [[ -e "${RUNFILES_DIR:-/dev/null}/$1" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found under RUNFILES_DIR ($RUNFILES_DIR), return" 1>&2; fi; echo "${RUNFILES_DIR}/$1"; else if [[ -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): looking in RUNFILES_MANIFEST_FILE ($RUNFILES_MANIFEST_FILE)" 1>&2; fi; local -r result=$(grep -m1 "^$1 " "${RUNFILES_MANIFEST_FILE}" | cut -d ' ' -f 2-); if [[ -e "${result:-/dev/null}" ]]; then if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): found in manifest as ($result)" 1>&2; fi; echo "$result"; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "INFO[runfiles.bash]: rlocation($1): not found in manifest" 1>&2; fi; echo ""; fi; else if [[ "${RUNFILES_LIB_DEBUG:-}" == 1 ]]; then echo "ERROR[runfiles.bash]: cannot look up runfile \"$1\" " "(RUNFILES_DIR=\"${RUNFILES_DIR:-}\"," "RUNFILES_MANIFEST_FILE=\"${RUNFILES_MANIFEST_FILE:-}\")" 1>&2; fi; return 1; fi; fi; fi; fi; fi } BASH_FUNC_runfiles_export_envvars%%=() { if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" && ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then return 1; fi; if [[ ! -f "${RUNFILES_MANIFEST_FILE:-/dev/null}" ]]; then if [[ -f "$RUNFILES_DIR/MANIFEST" ]]; then export RUNFILES_MANIFEST_FILE="$RUNFILES_DIR/MANIFEST"; else if [[ -f "${RUNFILES_DIR}_manifest" ]]; then export RUNFILES_MANIFEST_FILE="${RUNFILES_DIR}_manifest"; else export RUNFILES_MANIFEST_FILE=; fi; fi; else if [[ ! -d "${RUNFILES_DIR:-/dev/null}" ]]; then if [[ "$RUNFILES_MANIFEST_FILE" == */MANIFEST && -d "${RUNFILES_MANIFEST_FILE%/MANIFEST}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%/MANIFEST}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else if [[ "$RUNFILES_MANIFEST_FILE" == *_manifest && -d "${RUNFILES_MANIFEST_FILE%_manifest}" ]]; then export RUNFILES_DIR="${RUNFILES_MANIFEST_FILE%_manifest}"; export JAVA_RUNFILES="$RUNFILES_DIR"; else export RUNFILES_DIR=; fi; fi; fi; fi } BASH_FUNC_is_absolute%%=() { [[ "$1" = /* ]] || [[ "$1" =~ ^[a-zA-Z]:[/\\].* ]] } _=/bin/printenv 08 12 2021 18:44:30.331:INFO [karma-server]: Karma v6.3.9 server started at http://localhost:9876/ 08 12 2021 18:44:30.332:INFO [launcher]: Launching browsers FirefoxHeadless with concurrency unlimited 08 12 2021 18:44:30.336:INFO [launcher]: Starting browser FirefoxHeadless 08 12 2021 18:45:30.379:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:45:30.386:INFO [launcher]: Trying to start FirefoxHeadless again (1/2). 08 12 2021 18:46:30.434:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:46:30.439:INFO [launcher]: Trying to start FirefoxHeadless again (2/2). 08 12 2021 18:47:30.484:WARN [launcher]: FirefoxHeadless has not captured in 60000 ms, killing. 08 12 2021 18:47:30.489:ERROR [launcher]: FirefoxHeadless failed 2 times (timeout). Giving up. ================================================================================ INFO: Elapsed time: 525.903s, Critical Path: 203.05s INFO: 353 processes: 288 internal, 64 linux-sandbox, 1 worker. INFO: Build completed, 3 tests FAILED, 353 total actions //api/metadata/src:test PASSED in 4.4s //webapp/src/app:test.suite_chromium-local PASSED in 1.7s //webapp/src/app/table:test.suite_chromium-local PASSED in 1.2s //webapp/src/app/table/filter:test.suite_chromium-local PASSED in 1.7s //webapp/src/app:test.suite_firefox-local FAILED in 180.7s /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/test.suite_firefox-local/test.log //webapp/src/app/table:test.suite_firefox-local FAILED in 180.7s /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/test.suite_firefox-local/test.log //webapp/src/app/table/filter:test.suite_firefox-local FAILED in 181.2s /home/runner/.cache/bazel/_bazel_runner/c525028638995f32dc53ea162408dae4/execroot/data_catalog/bazel-out/k8-fastbuild/testlogs/webapp/src/app/table/filter/test.suite_firefox-local/test.log Executed 7 out of 7 tests: 4 tests pass and 3 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. INFO: Build completed, 3 tests FAILED, 353 total actions Error: Process completed with exit code 3. ```
devversion commented 2 years ago

Relevant info: This is likely the error undernath. I patched the firefox launcher to print the error: Error: Access was denied while trying to open files in your profile directory.

bazel test packages/concatjs/web_test/test/karma:testing_firefox-local \
  --sandbox_writable_path="$HOME/Library/Application Support"

The problem seems to be that Firefox tries to write to the default library directory in e.g. Darwin, but in Bazel with the test being sandboxed this conflicts! The flag above allows this to happen, so the tests run as expected.

It seems like Firefox does always make this request, regardless of us specifying a temporary directory in the Karma launcher. The best solution seems to be to run the tests outside of the sandbox for Firefox, unless this maybe is resolved/gone in the future.

devversion commented 2 years ago

I've put together a standalone reproduction: https://github.com/devversion/firefox-sandbox-exec-repro/tree/master

mgred commented 2 years ago

@devversion sorry for my late reply, I got sidetracked from this issue. Thanks a lot for your further investrigations

The best solution seems to be to run the tests outside of the sandbox for Firefox

What do you exactly mean with that?

devversion commented 2 years ago

I was mostly thinking about giving e.g. the web test target a tag like no-sandbox. This will cause the test to run outside of sandbox. Sandboxing is actually a key feature of Bazel though, so it's a trade-off. The bugzilla issue I filed seems to be low priority on Mozilla's side.

mgred commented 2 years ago

Adding no-sandbox to the tags of web_test_suite did it for me :star_struck: @devversion thanks for your support and explanation :pray:

devversion commented 2 years ago

@mgred no problem! can you actually keep this issue open if you don't mind? (it's a good tracking issue)