[debian] All oci-rootless integration tests failing

[ijc]

Running make test fails for me on master (db8b38227479f564f777fd91a496f043b22525aa). All three worker=oci-rootless tests are failing (and nothing else). All have the same stack trace:

Error Trace:    run.go:52
Error:      Received unexpected error failed dialing: /tmp/bktest_buildkitd863838567/buildkitd.sock
            github.com/moby/buildkit/util/testutil/integration.waitUnix
                /go/src/github.com/moby/buildkit/util/testutil/integration/util.go:85
            github.com/moby/buildkit/util/testutil/integration.runBuildkitd
                /go/src/github.com/moby/buildkit/util/testutil/integration/oci.go:152
            github.com/moby/buildkit/util/testutil/integration.(*oci).New
                /go/src/github.com/moby/buildkit/util/testutil/integration/oci.go:60
            github.com/moby/buildkit/util/testutil/integration.Run.func1
                /go/src/github.com/moby/buildkit/util/testutil/integration/run.go:47
            testing.tRunner
                /usr/local/go/src/testing/testing.go:777
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:2361

Error Trace:    run.go:62
        client_test.go:37
Error:      Should be true

Full output is in test.txt but doesn't have much more than the above.

I'm running Debian (slightly out-of-date testing, kernel 4.14.13-1) with Docker 18.05.0-ce

Could this be something related to this // TODO: make sure the user exists and subuid/subgid are configured.? I've not done any setup of that sort (as far as I know). OTOH I can't see it being done in .travis.yml and this is passing in CI it seems.

/cc @AkihiroSuda

[ijc]

with Docker 18.05.0-ce.

Typing that reminded me I was due an upgrade but I see the same thing with 18.06.0-ce.

[AkihiroSuda]

echo 1 > /proc/sys/kernel/unprivileged_userns_clone?

sub-IDs are configured in test.Dockerfile.

[ijc]

Ah yes, I expect that's it, thanks (trying it now). Might be useful to put a hint to that somewhere more prominent than docs/rootless.md to be discovered by those who just want to run make test but aren't explicitly looking into rootless stuff, e.g. log from the tests themselves.

Or perhaps the tests could auto t.Skip if they discover it to be 0?

[ijc]

That got me closer, but now I have one different failure:

    --- FAIL: TestClientIntegration/TestUser/worker=oci-rootless (5.04s)
        assertions.go:237: ^M                        ^M Error Trace:    client_test.go:361
                ^M                      run.go:60
                ^M      Error:          Received unexpected error rpc error: code = Unknown desc = executor failed running [sh -c id -nu > user]: exit code 127
                        failed to solve
                        github.com/moby/buildkit/client.(*Client).Solve.func3
                                /go/src/github.com/moby/buildkit/client/solve.go:142
                        github.com/moby/buildkit/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                                /go/src/github.com/moby/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:58
                        runtime.goexit
                                /usr/local/go/src/runtime/asm_amd64.s:2361

        oci.go:84: stdout: /usr/bin/sudo
        oci.go:87: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
        oci.go:84: stderr: /usr/bin/sudo
        oci.go:87: time="2018-07-19T15:42:46Z" level=debug msg="auto snapshotter: using native, because overlayfs is not available for /tmp/bktest_buildkitd177060793: failed to mount overlay: operation not permitted" 

Those ^@ are literal nuls (i.e. 0x0).

[tonistiigi]

@AkihiroSuda Unrelated to the issue, in addition to the skip if we can detect that it is unsupported, I think we can make the rootless worker configurable and disabled on local cases by default.

[AkihiroSuda]

like make test TESTPKGS=./client TESTFLAGS="-test.full"?

[tonistiigi]

@AkihiroSuda Could you capture that? Easiest probably would be to add WORKERS=with the default set being oci and containerd. Not a high priority. More important to figure out the issue with the test and check for unprivileged_userns_clone.

[AkihiroSuda]

The cause of the error is that the permission bit set of / is set to 0755 on overlayfs snapshotter (available only for Ubuntu) while it is set to 0700 on native snapshotter. Will open PR ASAP.

[AkihiroSuda]

PR: https://github.com/containerd/containerd/pull/2485

[ijc]

@AkihiroSuda out of interest, where was the page of NULs coming from?

[AkihiroSuda]

Maybe some bufio glitch?