jacquetc
commented
10 months ago I continued my investigations. I am sharing here my findings and some solutions. I also realize that it's less a problem of sccache itself but more its integration in OpenSuse, so sorry about putting it here.
On OpenSuse Tumbleweed (of 16/01/2024), the problem is two-folds: AppArmor and systemd service hardening.
Stopping the builder service (sudo systemctl stop sccache-dist-builder) and using the command RUST_LOG=sccache=trace SCCACHE_START_SERVER=1 SCCACHE_NO_DAEMON=1 SCCACHE_ERROR_LOG=/tmp/sccache_builder_log.txt SCCACHE_LOG=debug sudo sccache-dist server --config /etc/sccache/builder.conf were useful to concentrate on one problem at the time.
I use a very simple Hello Word C++ program with a CMakeLists.txt and -DCMAKE_CXX_COMPILER_LAUNCHER=sccache as cmake option.
The security part is solved (I think), but there is a last problem that I hope someone would help me with ?
AppArmor
By default, even without profile, it seems that AppArmor applies some security on sofwares like bwrap. So I had to create a dedicated profile. Let's avoid deactivating a security service.
I created an AppArmor profile and filled it. I used sudo aa-genprof /usr/bin/bwrap, and used this one as a base. I added the rest line by line by trial and errors.
sudo nano /etc/apparmor.d/usr.bin.bwrap
# Last Modified: Tue Jan 16 12:27:01 2024
abi <abi/3.0>,
include <tunables/global>
# base of this profile taken from https://apparmor.narkive.com/MqKSn3mT/what-to-do-about-bubblewrap-started-from-apps-confined-with
/usr/bin/bwrap flags=(attach_disconnected) {
include <abstractions/base>
include <abstractions/ubuntu-konsole>
capability dac_override,
capability net_admin,
capability setgid,
capability setpcap,
capability setuid,
capability sys_admin,
capability sys_chroot,
capability sys_ptrace,
network netlink raw,
mount -> /,
mount options=(rw, rslave) -> /,
mount fstype=tmpfs,
mount options=(rbind) /tmp/newroot/ -> /tmp/newroot/,
mount options=(rw, rbind) /oldroot/var/cache/sccache-builder/tmp/builds/** -> /newroot/,
mount options=(rw, nosuid, nodev, remount, bind, silent, relatime) -> /newroot/,
mount fstype=proc options=(rw, nosuid, nodev, noexec) -> /newroot/proc/,
mount options=(ro, nosuid, nodev, noexec, remount, bind, silent, relatime) -> /newroot/proc/sysrq-trigger,
mount options=(rw, rbind) -> /newroot/proc/sysrq-trigger,
mount options=(rw, rbind) -> /newroot/dev/null,
mount options=(rw, rbind) -> /newroot/proc/irq/,
mount options=(ro, nosuid, nodev, noexec, remount, bind, silent, relatime) -> /newroot/proc/irq/,
mount options=(rw, rbind) -> /newroot/proc/bus/,
mount options=(ro, nosuid, nodev, noexec, remount, bind, silent, relatime) -> /newroot/proc/bus/,
mount options=(rw, rbind) -> /newroot/dev/zero,
mount options=(rw, rbind) -> /newroot/dev/full,
mount options=(rw, rbind) -> /newroot/dev/random,
mount options=(rw, rbind) -> /newroot/dev/urandom,
mount options=(rw, rbind) -> /newroot/dev/tty,
mount fstype=devpts options=(rw, nosuid, noexec) -> /newroot/dev/pts/,
mount options=(rw, silent, rprivate) -> /oldroot/,
umount /oldroot/,
umount /,
pivot_root,
/ r,
/proc/self/ns/cgroup rw,
/run/user/[0-9]*/.bubblewrap/{old,new}root/ rw,
/run/user/[0-9]*/.bubblewrap/{old,new}root/usr/ rw,
/usr/bin/bwrap mr,
/usr/bin/bwrap mr,
/usr/bin/g++ mrix,
/usr/lib64/gcc/x86_64-suse-linux/13/cc1plus mrix,
/usr/bin/c++ mrix,
/{old,new}root/** rw,
@{PROC}/@{pid}/fd/ r,
@{PROC}/@{pid}/mountinfo r,
@{PROC}/sys/kernel/overflow{gid,uid} r,
owner /proc/sys/kernel/overflowgid r,
owner /proc/sys/kernel/overflowuid r,
owner @{PROC}/@{pid}/setgroups rw,
owner @{PROC}/@{pid}/{gid,uid}_map rw,
}To update AppArmor with the changes:
sudo apparmor_parser -r /etc/apparmor.d/usr.bin.bwrap
Later I realized that using sudo ln -s /etc/apparmor.d/usr.bin.bwrap /etc/apparmor.d/disable/ would have been a useful but dirty shorcut.
Systemd hardening
OpenSuse automatically adds hardening options to the services. Finally, and strangely, I had to deactivate all of them, else I would have a bwrap: Can't mount proc on /newroot/proc: Operation not permitted error at compilation.
sudo systemctl edit sccache-dist-builder.service
GNU nano 7.2 /etc/systemd/system/sccache-dist-builder.service.d/.#override.confb8789a643c1d8e95
### Editing /etc/systemd/system/sccache-dist-builder.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file
[Service]
ProtectSystem=false
ProtectHome=false
PrivateDevices=false
ProtectHostname=false
ProtectClock=false
ProtectKernelTunables=false
ProtectKernelModules=false
ProtectKernelLogs=false
ProtectControlGroups=false
RestrictRealtime=false
Restart=on-failure
RestartSec=1min
### Edits below this comment will be discarded
### /usr/lib/systemd/system/sccache-dist-builder.service
# [Unit]
# Description=sccache-dist builder
# After=chronyd.service ntpd.service network-online.target
#
# [Service]
# # added automatically, for details please see
# # https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
# ProtectSystem=full
# ProtectHome=read-only
# PrivateDevices=true
# ProtectHostname=true
# ProtectClock=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectKernelLogs=true
# ProtectControlGroups=true
# RestrictRealtime=true
# # end of automatic additions
# User=0
# Type=simple
# CacheDirectory=sccache-builder
# Environment="RUST_LOG=sccache=info"
# Environment="SCCACHE_NO_DAEMON=1"
# ExecStart=sccache-dist server --config /etc/sccache/builder.conf
#
# [Install]
# WantedBy=multi-user.target
I also added Restart and RestartSec because of the service failing to start properly at boot-up. I added them to sccache-dist-scheduler.service too.
Restart the service(s) to apply the new config:
sudo systemctl restart sccache-dist-builder.service
sudo systemctl restart sccache-dist-scheduler.serviceLast unsolved problem
While compiling, I have this last error:
/usr/lib64/gcc/x86_64-suse-linux/13/cc1plus: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
sccache: Compiler killed by signal 1Before you ask : yes, libz is already installed. If there is an obvious solution, I don't see it right now. That's why I ask for help to the community. Do you think of anything ?
Thank you !
sylvestre
Hello,
I'm trying to make a desktop useful to speed up compilations and it seems that Sccache is the better choice for distributed compilation when I compared with Icecream or Distcc.
Configuration:
My configuration at home is:
With this configuration, all seem running as expected :
I wanted to compile a C++/ Qt project, so I installed all the needed dependencies on both machines. In the project panel of Qt Creator, I added to CMake configuration:
On ~/.bashrc, I added:
Builder service:
Scheduler service:
Problem
Compiling the C++ fails and gives me this error at each step:
It seems to have something to do with security in OpenSuse, like AppArmor or something similar, but I am stuck.
Do you have any idea ?
Here is the compilation_log.txt I can give you other files if needed.
Thank you !