master chromium-local is breaking unless`--no-sandbox` is applied

[jmhodges]

With the latest rules_webtesting, my Go selenium tests are crashing with errors like

session not created: [Go WebDriver Client] (session not created) unable to create session: [Go WebDriver Client] (session not created) session not created

and, at one point,

unable to discover open window in chrome

I verified that I had a recent chromedriver in place (2.43, using the rules_webtesting's own packages).

The problem went away when I added a custom_browser that added

{
  "capabilities" : {
    "goog:chromeOptions" : {
      "args" : [
        "--no-sandbox"
      ]
    }
  }
}

I'm not sure what's going on.

My test suite was defined as:

go_web_test_suite(
    name = "integration_test",
    srcs = [
        "accounts_integration_test.go",
        "background_reporter_test.go",
        "fake_subs_test.go",
        "integration_test.go",
        "mailer_test.go",
        "stripe_mock_test.go",
    ],
    browser_overrides = {
        "@io_bazel_rules_webtesting//browsers:chromium-local": {
            # this tag is a hack to avoid go_default_library and this test
            # writing to the howsmyssl_billing_test database at the same
            # time. Need a more robust solution which probably involves fixes to
            # setup.sh to create integration envs.
            "tags": ["exclusive"],
        },
    },
    browsers = [
        "@io_bazel_rules_webtesting//browsers:chromium-local",
    ],
    data = [":dev_config"],
    embed = [":go_default_library"],
    deps = [
        "@com_github_golang_mock//gomock:go_default_library",
        "@com_github_google_go_cmp//cmp:go_default_library",
        "@com_github_google_go_cmp//cmp/cmpopts:go_default_library",
        "@com_github_tebeka_selenium//:go_default_library",
        "@com_github_tebeka_selenium//chrome:go_default_library",
        "@com_github_tebeka_selenium//log:go_default_library",
        "@io_bazel_rules_webtesting//go/webtest:go_default_library",
    ],
)

[jmhodges]

Oh, and I checked ./bazel-out/darwin-fastbuild/bin/external/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-mac/Chromium.app/Contents/MacOS/Chromium --version and it was Chromium 70.0.3538.0

That's a modern version and the one that rules_webtesting sets up.

[jmhodges]

I betting this is a bazel 0.18 thing? Maybe because user info gets scrubbed or something in some newer containerization as alluded to in the original blog post on headless Chrome? (search for "no-sandbox" in there)

[jmhodges]

It could also be a macOS Mojave thing? Not sure.

[jmhodges]

Ping

[DrMarcII]

We actually used to have to do this internally on Linux because of a lack of support for nested sandboxing (basically Blaze/Forge was sandboxing the action, and it was not configured to allow a nested sandbox to be created within that top-level sandbox). I suspect that something similar is happening on MacOS. However, I know nothing about how sandboxing works on MacOS, so I would suggest finding and talking to whoever has done the Bazel sandboxing for Bazel.

[jmhodges]

Ah, as I'm not a Googler, do you know who that would be?

[DrMarcII]

No. If I was trying to figure it out (since, although I am Googler, I don't work on Bazel), I would try to identify where the MacOS sandbox is configured in Bazel, and look at the commit history of that file to try to identify a user who might know more about it.

[zellyn]

Is there any reason to believe this is fixed? I'm trying to run a very simple web browser test, and it's failing with org.openqa.selenium.SessionNotCreatedException: [Go WebDriver Client] (session not created) unable to create session.

I'm on Mac OS 10.15.1 with bazel 1.1.0-homebrew.

[DrMarcII]

The actual issue is not fixed (AFAIK), but will not be fixed on this side. You can use "--no-sandbox" to work around it, and the containerization used for tests is a Bazel thing, not something that can be handled from within this project.

[zellyn]

I got it to work locally adding tags = ["no-sandbox", "native"], (the test errored out without the native tag… ¯_(ツ)_/¯) to my java_web_test_suite rule.

Would you mind explaining what the incantation would be to pass --no-sandbox down to the Chromium process? I've tried reading through some of the code, but it's getting deep :-)

[zellyn]

the containerization used for tests is a Bazel thing, not something that can be handled from within this project

I'm not sure I understand this. As I understood it, this project exists exactly to run WebDriver tests in bazel, so if something needs to be added to chromium commandlines to get that to work, this project would be exactly the right place to fix it. Or am I misunderstanding something? Thanks again for your help!

[DrMarcII]

no-sandbox in tags doesn't do anything AFAIK.

You can send: { "goog:chromeOptions": { "args": ["--no-sandbox"] } } in capabilities when starting the browser (I don't remember the exact incantation to build that up in DesiredCapabilities for Java, but it should be easy enough to figure it out by looking at Selenium Docs).

[DrMarcII]

It is also possible to include the --no-sandbox tag as part of the browser definition (it would be added to browsers/chromium-native.json file), but we don't want to disable the sandbox for Windows and Linux users. A separate browsers/chromium-native-macos.json file could be created and selected in browsers/BUILD using select and the configs defined in common/conditions. Feel free to send a PR if you want to do this.

[zellyn]

I tried changing my test thusly:

custom_browser(
    name = "no-sandbox-chromium-native",
    browser = "@io_bazel_rules_webtesting//browsers:chromium-native",
    metadata = "chromium-no-sandbox.json",
)

java_web_test_suite(
    name = "com/squareup/service/container/exemplar/BrowserTest",
    srcs = ["//service/exemplar/src/test/java:com/squareup/service/container/exemplar/BrowserTest.java"],
    browsers = [
        # "@io_bazel_rules_webtesting//browsers:chromium-local",
        ":no-sandbox-chromium-native",
    ],
   ⋮

(the json file contains the snippet at the top of this bug). But now I'm getting:

ERROR: /private/var/tmp/_bazel_zellyn/c56e2a60ec67d2dd598d789409bed348/external/io_bazel_rules_webtesting/web/web.bzl:111:12: Traceback (most recent call last):
    File "/Users/zellyn/Development/java/service/exemplar/src/test/java/BUILD.bazel", line 86
        java_web_test_suite(<11 more arguments>)
    File "/private/var/tmp/_bazel_zellyn/c56e2a60ec67d2dd598d789409bed348/external/io_bazel_rules_webtesting/web/java.bzl", line 33, in java_web_test_suite
        wrap_web_test_suite(name = name, <4 more arguments>)
    File "/private/var/tmp/_bazel_zellyn/c56e2a60ec67d2dd598d789409bed348/external/io_bazel_rules_webtesting/web/internal/wrap_web_test_suite.bzl", line 80, in wrap_web_test_suite
        web_test_suite(name = name, <14 more arguments>)
    File "/private/var/tmp/_bazel_zellyn/c56e2a60ec67d2dd598d789409bed348/external/io_bazel_rules_webtesting/web/web.bzl", line 81, in web_test_suite
        _get_kwargs(unqualified_browser, <1 more arguments>)
    File "/private/var/tmp/_bazel_zellyn/c56e2a60ec67d2dd598d789409bed348/external/io_bazel_rules_webtesting/web/web.bzl", line 111, in _get_kwargs
        out_kwargs["tags"]
key "tags" not found in dictionary

I'm not exactly sure what's going on under the covers, but this sure seems like the wrong way to check whether a key is in a dict: https://github.com/bazelbuild/rules_webtesting/blob/0ee17d358f9a462ac272b19233c1dc1b48d509a0/web/web.bzl#L111

[DrMarcII]

Looks like a bug there. Should be if "tags" not in out_kwargs: Will try to get a fix submitted today.

[zellyn]

Hmmm. That makes me slightly doubtful that custom_browser actually works :-)

[zellyn]

fwiw, chromium is failing on our EC2 workers too… looks like that --no-sandbox flag is needed in more than one place:

$ /data/app/kochiku-worker/.cache/bazel/_bazel_kochiku-worker/0b36dc59c757f69574925eb2ab4de8ae/execroot/__main__/bazel-out/k8-fastbuild/bin/service/exemplar/src/test/java/com/squareup/service/container/exemplar/BrowserTest_chromium-local.sh.runfiles/io_bazel_rules_webtesting/third_party/chromium/chromium.out/chrome-linux/chrome
[31996:31996:1125/213655.060264:FATAL:zygote_host_impl_linux.cc(116)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux_suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
#0 0x558b61db5a79 base::debug::CollectStackTrace()
#1 0x558b61d19553 base::debug::StackTrace::StackTrace()
#2 0x558b61d2dac2 logging::LogMessage::~LogMessage()
#3 0x558b633da68e service_manager::ZygoteHostImpl::Init()
#4 0x558b6193d177 content::ContentMainRunnerImpl::Initialize()
#5 0x558b619881ba service_manager::Main()
#6 0x558b6193b731 content::ContentMain()
#7 0x558b5f82b1b3 ChromeMain
#8 0x7f01570bc3d5 __libc_start_main
#9 0x558b5f82b02a _start

Received signal 6
#0 0x558b61db5a79 base::debug::CollectStackTrace()
#1 0x558b61d19553 base::debug::StackTrace::StackTrace()
#2 0x558b61db5601 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#3 0x7f015d2445d0 <unknown>
#4 0x7f01570d0207 __GI_raise
#5 0x7f01570d18f8 __GI_abort
#6 0x558b61db4435 base::debug::BreakDebugger()
#7 0x558b61d2dfe6 logging::LogMessage::~LogMessage()
#8 0x558b633da68e service_manager::ZygoteHostImpl::Init()
#9 0x558b6193d177 content::ContentMainRunnerImpl::Initialize()
#10 0x558b619881ba service_manager::Main()
#11 0x558b6193b731 content::ContentMain()
#12 0x558b5f82b1b3 ChromeMain
#13 0x7f01570bc3d5 __libc_start_main
#14 0x558b5f82b02a _start
  r8: 00007f015d83eb00  r9: 0000000000000000 r10: 0000000000000008 r11: 0000000000000206
 r12: 00007ffc342749f8 r13: 00007ffc342749f8 r14: 00007ffc34275a20 r15: 00007ffc34275a18
  di: 0000000000007cfc  si: 0000000000007cfc  bp: 00007ffc342749a0  bx: 00007f0157461868
  dx: 0000000000000006  ax: 0000000000000000  cx: ffffffffffffffff  sp: 00007ffc34274868
  ip: 00007f01570d0207 efl: 0000000000000206 cgf: 0000000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.

[DrMarcII]

The tags bug can be worked around by including a tags attribute on your test (doesn't matter what it has in it, can even be [] I think).

I am not surprised that no-sandbox is needed in other contexts. When I created this project, I intended users to create their own browser definitions, and to use SauceLabs (or BrowserStack or whatever other provider in this space they want), not to use the ones in this project (they are mostly there as examples and for internal testing of the project). People have pushed back and said they want them defined in the project, so I acquiesced and have done some work to make them more reliable for others to use. That said, I only test these native browsers locally on Linux (and occasionally on MacOS in the past), and exclusively use SauceLabs for tests that run in my CI environment.

[zellyn]

The tags bug can be worked around by including a tags attribute on your test (doesn't matter what it has in it, can even be [] I think).

That is of course correct. I could have sworn I tried it before posting :-) btw, the chromedriver-local will not be happy unless "native" is in the list of tags you specify. The multiple flavors of default-vs-provided tag logic as you walk down the nested starlark functions are a bit confusing.