Directory PermissionError during extraction

[erikswanson]

I'm trying to restore from a backup (in a test scenario) but borg extract fails every time for me when it gets to a directory.

Extract failure:

[root@eed86ffdf96c /]# borg extract --verbose --umask 0o000 borg:/borg/cah-demo::1440546045 var/lib/cassandra
Warning: File system encoding is "ascii", extracting non-ascii filenames will not be supported.
var/lib/cassandra
var/lib/cassandra/commitlog
var/lib/cassandra/commitlog/CommitLog-4-1440542707708.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707709.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707710.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707711.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707712.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707713.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707714.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707715.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707716.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707717.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707718.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707719.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707720.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707721.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707722.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707723.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707724.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707725.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707726.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707727.log
var/lib/cassandra/commitlog/CommitLog-4-1440542707728.log
borg: Error: Local Exception.
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 903, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 859, in run
    return args.func(args)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 242, in do_extract
    archive.extract_item(dirs.pop(-1), stdout=stdout)
  File "/usr/lib64/python3.4/site-packages/borg/archive.py", line 305, in extract_item
    self.restore_attrs(path, item)
  File "/usr/lib64/python3.4/site-packages/borg/archive.py", line 330, in restore_attrs
    xattr.setxattr(fd or path, k, v, follow_symlinks=False)
  File "/usr/lib64/python3.4/site-packages/borg/xattr.py", line 101, in setxattr
    _check(func(path, name, value, len(value) if value else 0, 0), path)
  File "/usr/lib64/python3.4/site-packages/borg/xattr.py", line 36, in _check
    raise OSError(get_errno(), path)
PermissionError: [Errno 13] b'/var/lib/cassandra/commitlog'

borg: Exiting with failure status due to previous errors

Check success:

[root@eed86ffdf96c /]# borg check borg:/borg/cah-demo::1440546045
Starting repository check...
Repository check complete, no problems found.
Starting archive consistency check...
Analyzing archive 1440546045 (1/1)
Orphaned objects check skipped (needs all archives checked)
Archive consistency check complete, no problems found.

Environment details:

I'm running borg in a Docker container, and /var/lib/cassandra is a volume in a different docker container that I have mounted into the borg container by means of --volumes-from.

From inside the container running borg:

[root@eed86ffdf96c /]# mount | grep cassandra
/dev/mapper/storage-volumes on /var/lib/cassandra type xfs (rw,relatime,seclabel,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)

This is with Docker 1.8.1 on an up-to-date Fedora Atomic 22 install, so selinux is in play at the host level, but will of course appear to be off from within the container.

$ ostree admin status
* fedora-atomic 64b9c0305c485e07e729f1630ce853653d03484fcb1fa6c527f9360f78f355ae.0
    Version: 22.96
    origin refspec: fedora-atomic:fedora-atomic/f22/x86_64/docker-host

Is there any way to completely suppress xattr restoration?

[erikswanson]

Oh, because the particular system libraries might matter for reproduction, here's my Dockerfile (approximately) for my borg container:

FROM fedora:22

ENV \
BORG_VERSION=0.24.0 \
BORG_COMMIT=1e35f5ce4a7f38917afdaddeebf61a94bc7478aa

RUN dnf install -y \
gcc \
git \
libacl \
libacl-devel \
lz4 \
lz4-devel \
openssh-clients \
openssl-devel \
openssl-libs \
python3 \
python3-Cython \
python3-devel \
python3-msgpack \
&& git clone https://github.com/borgbackup/borg.git \
--branch ${BORG_VERSION} \
--single-branch \
/tmp/borg \
&& pushd /tmp/borg \
&& git checkout ${BORG_COMMIT} \
&& pip3 install . \
&& popd \
&& rm -rf /tmp/borg /root/.cache \
&& dnf remove -y \
gcc \
git \
libacl-devel \
lz4-devel \
openssl-devel \
python3-Cython \
python3-devel \
&& dnf clean all

[erikswanson]

I added some experimental logging and found that the failure occurs when borg attempts to set the 'security.selinux' xattr on the directory. I'm up and running with a local patch to xattr.py that ignores all calls to set an xattr with that name.

Solutions that come to mind are either some automatic detection of when restoring selinux xattrs is forbidden (can't blindly assume this will work, even if selinux appears to be off), or perhaps just a simple switch that makes borg record/restore only numerical properties (mtime, uid, gid, octal permissions) and nothing else.

[ThomasWaldmann]

@erikswanson ah, good you found the reason.

You could make a patch / pull request that just tries to set one xattr after the other. if it fails to set one of them, it would just ignore that and continue with the next.

[erikswanson]

I'd love to, but getting approval from $work to release a contribution of actual code to an unknown-to-the-lawyers opensource project takes hours of meetings over a period of days if not weeks. :(

[ThomasWaldmann]

@erikswanson oh :| ok, so I rather fix this on my own, that's less painful.

Do you know / can you clarify why this malfunction happens?

The backup was made with / without selinux active? The restore is tried with/without selinux active?

In borg/archive.py:330 I see it silently ignores a setxattr OSError if it is ENOTSUP (not supported). It also silently ignores chown failing (usually causes by not running as root) some lines below that.

So I am thinking about just silently ignoring if setting some specific single xattr key/value setattr is failing with EACCES (errno 13, access denied). Not sure whether that is the best thing to do though.

[ThomasWaldmann]

Note: EACCES also happens when trying to set any security.* key as a non-root user.

[ThomasWaldmann]

See PR #166.

[erikswanson]

@ThomasWaldmann So, the selinux scenario is the same for both the backup and creation: on the host (outside the container) it is active, but inside the docker container, it appears to be inactive. If it were installed, getenforce would return 'off'. I'm running borg as "root" inside the container, but it's quite clearly not fully-privileged.

Failing to restore the old selinux labeling is a non-issue for me, as by virtue of the restored file also being written out within a container, it will (in my situation, at least) have the right label. I think treating it similarly to a failure to restore uid/gid is completely reasonable.

The edge case where someone could be surprised badly is if they're using multi-category labeling to have selinux maintain separation of docker containers' files from each-other.

[erikswanson]

PR #166 LGTM. It would probably not be easy (if possible) to DTRT when trying to restore inside a container with MCS enabled, so I wouldn't worry about it. (Some checks for the particular situation then throwing a specific warning.)

[ThomasWaldmann]

OK, thanks for the feedback. I'll merge after CI shows green.