docker panic in cloud image builds

[cgwalters]

https://ci.centos.org/view/Atomic/job/atomic-cahc-image-cloud-continuous/841/console

13:25:44 Running: docker run --rm --privileged --workdir /out --net=none -v /home/builder/sig-atomic-buildscripts:/in:ro -v /tmp/atomic-treecomposeXpXybp.tmp/tmp-kickstart:/out rpm-ostree-toolbox/centos-atomic-host-7-kickstart-app
13:25:44 panic: standard_init_linux.go:175: exec user process caused "permission denied" [recovered]
13:25:44    panic: standard_init_linux.go:175: exec user process caused "permission denied"
13:25:44 
13:25:44 goroutine 1 [running, locked to thread]:
13:25:44 panic(0x6f2ea0, 0xc42016c230)
13:25:44    /usr/lib/golang/src/runtime/panic.go:500 +0x1a1
13:25:44 github.com/urfave/cli.HandleAction.func1(0xc420097748)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
13:25:44 panic(0x6f2ea0, 0xc42016c230)
13:25:44    /usr/lib/golang/src/runtime/panic.go:458 +0x243
13:25:44 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc420097198, 0xc4200260c8, 0xc420097238)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
13:25:44 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc420057630, 0xaac9c0, 0xc42016c230)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
13:25:44 main.glob..func8(0xc42009aa00, 0x0, 0x0)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main_unix.go:26 +0x66
13:25:44 reflect.Value.call(0x6ddc20, 0x769b48, 0x13, 0x73c049, 0x4, 0xc420097708, 0x1, 0x1, 0x4d1728, 0x731ea0, ...)
13:25:44    /usr/lib/golang/src/reflect/value.go:434 +0x5c8
13:25:44 reflect.Value.Call(0x6ddc20, 0x769b48, 0x13, 0xc420097708, 0x1, 0x1, 0xac2700, 0xc4200976e8, 0x4da706)
13:25:44    /usr/lib/golang/src/reflect/value.go:302 +0xa4
13:25:44 github.com/urfave/cli.HandleAction(0x6ddc20, 0x769b48, 0xc42009aa00, 0x0, 0x0)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
13:25:44 github.com/urfave/cli.Command.Run(0x73c215, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d852, 0x51, 0x0, ...)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
13:25:44 github.com/urfave/cli.(*App).Run(0xc4200be000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
13:25:44 main.main()
13:25:44    /builddir/build/BUILD/docker-1b5971af3003488d3fd8add80de5125ee1e096ee/runc-b8dbc3b8e8d868723aec2fd5082e6547ec66cf58/main.go:137 +0xbd6
13:25:45 Traceback (most recent call last):
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 75, in <module>
13:25:45     main()
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose-main", line 57, in main
13:25:45     imagefactory.main(cmd)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 574, in main
13:25:45     imageouttypes=imagetypes
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/taskbase.py", line 495, in create
13:25:45     self.impl_create(**kwargs)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 386, in impl_create
13:25:45     ksdata = self.formatKS(ksfile)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/imagefactory.py", line 321, in formatKS
13:25:45     run_sync(cmd, env=child_env)
13:25:45   File "/usr/lib64/rpm-ostree-toolbox/py/rpmostreecompose/utils.py", line 36, in run_sync
13:25:45     subprocess.check_call(args, **kwargs)
13:25:45   File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
13:25:45     raise CalledProcessError(retcode, cmd)
13:25:45 subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '--privileged', '--workdir', '/out', '--net=none', '-v', '/home/builder/sig-atomic-buildscripts:/in:ro', '-v', '/tmp/atomic-treecomposeXpXybp.tmp/tmp-kickstart:/out', 'rpm-ostree-toolbox/centos-atomic-host-7-kickstart-app']' returned non-zero exit status 2

[miabbott]

I've seen this in RHEL builds.

https://bugzilla.redhat.com/show_bug.cgi?id=1382997

That BZ is marked CLOSED, but the comment history is a mess, switching between talking about 1.12 and 1.10. I think there may still be an issue there.

[miabbott]

I opened a new downstream BZ for this - https://bugzilla.redhat.com/show_bug.cgi?id=1405464

[jlebon]

This seems to have regressed in the latest Alpha release: https://s3.amazonaws.com/aos-ci/ghprb/projectatomic/rpm-ostree/b9af0c7515ea42e539930380c8e50e9ff56c0371.0.10495/output.log (from https://github.com/projectatomic/rpm-ostree/pull/561).

Booting that image to verify the Alpha version:

[root@jlebon-tmp ~]# rpm-ostree status
State: idle
Deployments:
● centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/alpha
       Version: 7.2017.4 (2017-01-06 19:07:01)
        Commit: 00f7b5b9e2352392b01412955f439f704cde9ec93baa9501b79af5ed1b866a32
        OSName: centos-atomic-host
[root@jlebon-tmp ~]# rpm -q docker
docker-1.12.4-3.el7.x86_64

[cgwalters]

Now re-fixed.