COPY fails on some base-images (cern/cc7-base)

[lukasheinrich]

Hi,

i'm continuing to workon integrating this with some of the CERN infrastructure. skopeo / orca-build / umoci (with https://github.com/openSUSE/umoci/issues/223) works nicely and I can build a new image from this Dockerfile

lheinric@lxplus071:/tmp/lheinric/test% cat Dockerfile
FROM busybox
RUN echo hello > /mydata
COPY datatocopy /datatocopy
lheinric@lxplus071:/tmp/lheinric/test% ls
datatocopy  Dockerfile

changing the FROM line to cern/cc7-base fails with a python error on shutil.copy when it tries to copy the datatocopy file

this is the log. Could this be related to some permissions set in the base image?

orca-build[INFO] Using existing image for build: ../outputimage
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:ad4325b2dd179e2103046df76bfe6efa7ac85e9ed6ab52edb33b8db399a9bd02
 81.41 MB / 81.41 MB [======================================================] 8s
Copying config sha256:a6644c6917335ef1b1285b7e6eb0e08445a798e4c230f3fd3601aa67f9c1ddd6
 280 B / 280 B [============================================================] 3s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
Traceback (most recent call last):
  File "/tmp/lheinric/bin/orca-build", line 741, in <module>
    __wrapped_main__()
  File "/tmp/lheinric/bin/orca-build", line 739, in __wrapped_main__
    main(ctx, config)
  File "/tmp/lheinric/bin/orca-build", line 705, in main
    builder.build(output=config.output, tags=config.tags, clean=config.clean, gc=config.gc)
  File "/tmp/lheinric/bin/orca-build", line 666, in build
    getattr(self, fn)(*args, isjson=step.isjson)
  File "/tmp/lheinric/bin/orca-build", line 563, in _dispatch_copy
    shutil.copy2(src, dst)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/lheinric/orca-bundle.xkqko1km/rootfs/datatocopy'

[lukasheinrich]

I notice that the rootfs directory is created without any write permissions (dr-xr-xr-x), which I think are set by umoci, so maybe this issue should be moved there, tough this line seems to suggest the rootfs path is created with a 755 mask

https://github.com/openSUSE/umoci/blob/52afb0fb11000b6229f9bc297ad85fb210f0572f/oci/layer/unpack.go#L129

lheinric@lxplus071:/tmp/lheinric/orca-bundle.uy10pjk4% ls -lr
total 2388
-rw-r--r--.  1 lheinric zp     542 Jan 18 14:29 umoci.json
-rw-r--r--.  1 lheinric zp 2429008 Jan 18 14:29 sha256_e063fcf061ac201da9bf323aef6b70a10b51721e9b005e881eef9063d4128d12.mtree
dr-xr-xr-x. 17 lheinric zp    4096 Jan 18 14:29 rootfs
-rw-r--r--.  1 lheinric zp    3588 Jan 18 14:29 config.json

[cyphar]

I will take a look at this bug today/tomorrow -- sorry for not looking at this in a while (for some reason I got un-subscribed from this repository...). Just to verify -- is cern/cc7-base publicly available?

I notice that the rootfs directory is created without any write permissions (dr-xr-xr-x), which I think are set by umoci, so maybe this issue should be moved there, tough this line seems to suggest the rootfs path is created with a 755 mask

That's very strange -- umoci should be making it with a 755 mask... I haven't seen that before.

[lukasheinrich]

yes it's available. Thanks for looking into it!

[cyphar]

I haven't been able to reproduce this problem for some reason:

orca-build[INFO] Created new image for build: /tmp/orca-build.v6_8qzzo
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [=====================================================] 39s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD: finished
orca-build[INFO] BUILD: created tags ['e60688b797f95c7bab02fad19bd7910021f963975cdf2dffc28b290d006163ff-dest']

Do you have a strange umask set in your shell that could be causing this -- or something like that? What is the output of umask? I tested this with:

% runc version
runc version 1.0.0-rc5
spec: 1.0.0
% umoci --version
umoci version 0.4.0
% skopeo version
skopeo version 0.1.26

[lukasheinrich]

hm this is my setup

umask gives

% umask
022

the tool versions are

% umoci --version
umoci version 0.4.0+dev~git8b89fabfac6c176b95078cb106c2704303564e6d
% runc --version
runc version spec: 1.0.0
% skopeo --version
skopeo version 0.1.26

% orca-build -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.aqeez1ak
orca-build[INFO] BUILD[1 of 5]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 3s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 5]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   ⨯ create runtime bundle: chown rootfs: lchown /tmp/lheinric/orca-bundle.hyumu_7p/rootfs: operation not permitted
  <--- [umoci]
orca-build[CRITICAL] Error executing subprocess: umoci unpack --image=/tmp/lheinric/orca-build.aqeez1ak:2c1a8d7ff8c4c1c7b2b4211768e79103fa4b0d6bbe5a4433df03fe8a3446861c-src /tmp/lheinric/orca-bundle.hyumu_7p failed with error code 1

[cyphar]

Ah, I misunderstood. I was running orca-build as root. You need to use --rootless, but I will take another look...

[lukasheinrich]

Ah right this was just a mistake in the post above in the OP I used rootless.

this is what ai get

lheinric@lxplus021:/tmp/lheinric% orca-build --rootless -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.q75819d9
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 2s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
  <--- [runc]
  ---> [umoci]
  <--- [umoci]
orca-build[INFO] BUILD[3 of 3]: copy ['datatocopy', '/datatocopy'] [json=False]
  ---> [umoci]
  <--- [umoci]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
Traceback (most recent call last):
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 741, in <module>
    __wrapped_main__()
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 739, in __wrapped_main__
    main(ctx, config)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 705, in main
    builder.build(output=config.output, tags=config.tags, clean=config.clean, gc=config.gc)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 666, in build
    getattr(self, fn)(*args, isjson=step.isjson)
  File "/afs/cern.ch/work/l/lheinric/orcabuild/bin/orca-build", line 563, in _dispatch_copy
    shutil.copy2(src, dst)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/lheinric/orca-bundle.zjpxpr91/rootfs/datatocopy'

[cyphar]

It's almost certainly caused by orca-build not being as clever as umoci when it comes to unprivileged copying (it doesn't know how to emulate CAP_DAC_OVERRIDE like umoci does). But I will investigate this one further...

[lukasheinrich]

thanks for looking into it!

[cyphar]

Sorry for not responding earlier --

The underlying issue is that orca-build isn't as clever as umoci when it comes to dealing with "bad" directories. Unfortunately the only real fix for this is to rewrite in Go so we can directly use umoci's packages.

The core problem is that the cern/cc7-base images have / as chmod 0555 -- which is not writable. You can fix this by copying to a sub-directory which is writable (so something like COPY datacopy /var/datacopy). Another option is to make your images not have this issue (you might need to add RUN chmod o+w / inside the image build you have set up).

The plan going forward for orca-build is that it will be part of the umoci project (but still a separate tool) and be written in Go. It will take a while unfortunately. But the above workarounds should help in the meantime.

[lukasheinrich]

Thanks @cyphar

I just chose / as it seemed the most minimal of examples. It's very reasonable I think to request people to copy stuff elsewhere. Usually we use a pattern like

FROM ...
WORKDIR /code
ADD . /code

so from your description it seems this should work. Unfortunately it seems CERNs interactive node regressed into a state that makes nsenter fail

I wonder if it's related to this https://github.com/opencontainers/runc/issues/1513

The node I'm working on should have unprivileged namespaces enabled though, maybe some update rolled that back. @davidlt did you try rootless runc recently on lxplus? I do see

cat /sys/module/namespace/parameters/unpriv_enable
Y

lheinric@lxplus021:/tmp/lheinric/wha% orca-build --rootless -t what .
orca-build[INFO] Created new image for build: /tmp/lheinric/orca-build.rym_2ou0
orca-build[INFO] BUILD[1 of 3]: from ['cern/cc7-base'] [json=False]
  ---> [skopeo]
Getting image source signatures
Copying blob sha256:297ef71a1de0345dfd52a966e8072ea4f1c38194c86df1ec46605ad56d958a69
 81.45 MB / 81.45 MB [======================================================] 3s
Copying config sha256:8a8dfdb81bfd41a0346c19564f11444ac1a3e8905206b92d79e5f9a38f131c92
 280 B / 280 B [============================================================] 0s
Writing manifest to image destination
Storing signatures
  <--- [skopeo]
orca-build[INFO] BUILD[2 of 3]: run ['echo', 'hello', '>', '/mydata'] [json=False]
  ---> [umoci]
   • rootless{usr/sbin/arping} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/sbin/clockdiff} ignoring (usually) harmless EPERM on setxattr "security.capability"
   • rootless{usr/bin/ping} ignoring (usually) harmless EPERM on setxattr "security.capability"
  <--- [umoci]
  ---> [runc]
nsenter: failed to unshare namespaces: Invalid argument
container_linux.go:348: starting container process caused "process_linux.go:279: running exec setns process for init caused \"exit status 40\""
  <--- [runc]
orca-build[CRITICAL] Error executing subprocess: runc --root=/tmp/lheinric/orca-runcroot.49u4p_ir run --bundle=/tmp/lheinric/orca-bundle.q7tuoacp orca-build-CcsvqYx301em4non1wp1f5FkCev4krQ0 failed with error code 1

[lukasheinrich]

it seems due to this CVE https://access.redhat.com/security/cve/cve-2018-1000001 we now have have max_user_namespaces = 0 (temprarily at least) :( I'll try to resume testing once they are enabled again.

we're following the discussions here:

https://bugzilla.redhat.com/show_bug.cgi?id=1533836

[cyphar]

I just chose / as it seemed the most minimal of examples. It's very reasonable I think to request people to copy stuff elsewhere. Usually we use a pattern like so from your description it seems this should work.

In principle it might work -- though the writable issue still might occur. Really I would recommend not making your images have / un-writeable by the owner. But when I finish the Go port of this project and merge it into umoci, then this should no longer be an issue. In the meantime, you can try doing COPY to a directory which is writeable in the rootfs.

[lukasheinrich]

thanks. I'll raise it with the people responsible for the image.

[lukasheinrich]

turns out one can just add a chmod +w to it and that works

FROM cern/cc7-base
RUN mkdir /code && chmod +w /code
COPY . /code

[cyphar]

I guess that implies that the default uname is wrong somehow and so the directory is not created with 755 permissions? It's nice that this is a work around for now -- but as I said we really need to have orca-build be using pkg/unpriv from umoci.

[lukasheinrich]

yeah the workaround also only works sometimes. If i switch the image

from cern/slc6-base to one build with this Dockerfile

FROM cern/slc6-base:latest
RUN adduser atlas && \
    usermod -aG wheel atlas

building a new image based on it fails (while it worked with cern/slc6-base. The /etc/gshadow file created during user creation seems to break it

FROM lukasheinrich/cern-slc6-breaking:latest
RUN  mkdir /code && chmod +w /code
RUN  echo hello > /code/mydata
COPY datatocopy /code/datatocopy