search

uapi-group / specifications

UAPI Group Specifications
https://uapi-group.org/specifications/
87 stars 32 forks source link

Tracking upstream projects that do not support hermetic-usr for configuration #76

Open bluca opened 1 year ago

bluca commented 1 year ago

This issue will be used to track Linux projects that do not currently support hermetic-usr configuration style (ie: /usr/lib/foo as default, /run/foo as ephemeral local override, /etc/foo as persistent local override). The purpose is to have a cross-distribution list of items to slowly work through, to be able to have a bootable and working minimal Linux image-based system with only /usr.

This list is not definitive and will get updated as we go.

thkukuk commented 1 year ago

A first bunch, starting with a minimal OS running containers.

julian-klode commented 1 year ago

APT and dpkg both don't support it. I haven't figured out the behavior for APT yet, tbh, it's weird because essentially everything is drop-ins for apt.conf these days.

thkukuk commented 1 year ago

glibc for /etc/services (used by getservbyname/getservbyport, which make little sense and are not widely used, but there still are some applications which do). It should be moved under /usr.

getservbyname/getservbyport are using NSS as backend. There are NSS modules for nearly every location, not only /etc/services, NIS or LDAP, but also for /usr. There is e.g. https://github.com/openSUSE/libnss_usrfiles, which, with the /etc/nsswitch.conf option services: files usrfiles looks at first for entries in /etc/services and afterwards in /usr/etc/services. More such modules exists, so in my opinion, this task is solved, it's just in the responsibility of the Linux distributior to package the best fitting variant and configure it. openSUSE Tumbleweed/MicroOS are doing this since many years. So this is solved for aliases, ethers, protocols, rpc and services. /etc/shells is not handled by NSS, so solving that will be more complicated.

fbuihuu commented 1 year ago

I dare to add to the list :

There's currently https://github.com/systemd/systemd/pull/28919 to address this issue by giving the possibility to downstream to ship the config files in /usr/lib but for some reasons the idea has not been well received.

fbuihuu commented 1 year ago
poettering commented 1 year ago

@fbuihuu systemd's config files in /etc/ are just decoration. Entirely redundant, they are pretty much just helpful hints to people who want to sue traditional populated /etc/. If you delete them for modern systems that come up without /etc/ then behaviour is not changed whatsoever.

All options listed in them just give users hints on the available settings and their defaults, and those options are fully commented, hence these files are NOPs.

thkukuk commented 1 year ago

Next package list:

thkukuk commented 1 year ago

For glibc I missed yesterday:

This are now all packages from our minimal installation of MicroOS as ContainerHost beside openssh. I spoke already with the openssh developers about this some time ago, they rejected all ideas to modify configuration file handling. So we (openSUSE) reconfigured the configuration file setup to mimic hermetic-usr as far as possible, open issue is AcceptEnv, which you cannot overwrite.

Question is how far do we want to go with that list? I could go as next through a typical server installation and desktop system, but I don't know if this would be really helpful, as the list is already long and it would mix up packages with different priority.

bluca commented 1 year ago

This are now all packages from our minimal installation of MicroOS as ContainerHost beside openssh. I spoke already with the openssh developers about this some time ago, they rejected all ideas to modify configuration file handling. So we (openSUSE) reconfigured the configuration file setup to mimic hermetic-usr as far as possible, open issue is AcceptEnv, which you cannot overwrite.

Yeah I had the same experience some years past, openssh as a project is just not interested in anything that doesn't directly benefit BSD. Should we set up a Linux-focused fork, where we can co-maintain patches?

Question is how far do we want to go with that list? I could go as next through a typical server installation and desktop system, but I don't know if this would be really helpful, as the list is already long and it would mix up packages with different priority.

I think this is a good starting point, being core packages they are the highest value to fix. Once we have made a dent in the current list, then we could start looking at a server installation.

keszybz commented 1 year ago

nscd (/etc/nscd.conf)

nscd is on its way out: https://fedoraproject.org/wiki/Changes/RemoveNSCD. IIRC, the intent was to also propose the same from glibc upstream if the experiment in Fedora is successful.

ReillyBrogan commented 1 year ago

Apparmor is another one that doesn't support any kind of vendordir at the moment.

schubi2 commented 11 months ago

/etc/rsyncd.conf should be solved with: https://github.com/WayneD/rsync/pull/551

util-linux (/etc/blkid.conf): https://github.com/util-linux/util-linux/pull/2616

schubi2 commented 11 months ago

I have fixed mailx (/etc/mail.rc) in SUSE OBS: https://build.opensuse.org/request/show/1131341 I would like to do it upstream too, but I do not find the code stream to which I can make a PR. Does anyone know where mailx is hosted upstream ? I know, it is really old code :-)

bluca commented 11 months ago

In Debian we ship bsd-mailx: https://tracker.debian.org/pkg/bsd-mailx which according to package metadata comes from https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/mail/

schubi2 commented 11 months ago

In Debian we ship bsd-mailx: https://tracker.debian.org/pkg/bsd-mailx which according to package metadata comes from https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/mail/

Thanks for hint ! I have sent a patch to tech@openbsd.org .

poettering commented 11 months ago

I filed this bug against Fedora's authselect now:

https://github.com/authselect/authselect/issues/355

schubi2 commented 11 months ago

I have made a PR for fcoe-utils: https://github.com/openSUSE/fcoe-utils/pull/24

schubi2 commented 11 months ago

I have submitted a patch for selinux: https://github.com/SELinuxProject/selinux/pull/415

Conan-Kudo commented 11 months ago

DNF 5 supports hermetic configuration since 5.1.4.

https://github.com/rpm-software-management/dnf5/pull/813 https://github.com/rpm-software-management/dnf5/pull/820 https://github.com/rpm-software-management/dnf5/pull/882

keszybz commented 11 months ago

rpm-software-management/dnf5#813 rpm-software-management/dnf5#820 rpm-software-management/dnf5#882

Thanks. I filed https://github.com/rpm-software-management/dnf5/issues/1107.

DaanDeMeyer commented 11 months ago

What about ca-certificates? That one seems to be prominently missing from the list? Debian already seems to put certificates in /usr/share/ca-certificates, whereas Fedora uses /etc/pki and OpenSUSE /var/lib/ca-certificates? It'd be great if we could standardize on /usr/share/ca-certificates in some form (not familiar with the details)

schubi2 commented 11 months ago

I have submitted a patch for open-iscsi: https://github.com/open-iscsi/open-iscsi/pull/437

Conan-Kudo commented 11 months ago

What about ca-certificates? That one seems to be prominently missing from the list? Debian already seems to put certificates in /usr/share/ca-certificates, whereas Fedora uses /etc/pki and OpenSUSE /var/lib/ca-certificates? It'd be great if we could standardize on /usr/share/ca-certificates in some form (not familiar with the details)

A /usr/share/pki hierarchy as an equivalent to the one in /etc/pki would allow moving most of this stuff.

DaanDeMeyer commented 9 months ago

Another one that seems to be missing is openssl

keszybz commented 2 months ago

Since dnf5 gained support, I filed the following: https://src.fedoraproject.org/rpms/systemd/c/bedc0270e72071dd027f998629f6d633700c5b89 https://src.fedoraproject.org/rpms/setup/pull-request/12 https://src.fedoraproject.org/rpms/sudo/pull-request/29

keszybz commented 2 months ago

I marked those as fixed:

systemd "empty" main config files in /etc/systemd udev "empty" main config files in /etc/udev

https://github.com/systemd/systemd/commit/6495361c7d added the functionality to systemd. https://src.fedoraproject.org/rpms/systemd/c/29eb35530b and https://src.fedoraproject.org/rpms/systemd/c/4c7acded2a make use of this in Fedora.

poettering commented 1 month ago

I posted patches to glibc mailing list that teaches glibc NSS to look in /usr/share/ for its various database files, such as /etc/protocols (which will now be looked for in /usr/share/protocols too):

https://sourceware.org/pipermail/libc-alpha/2024-September/160190.html

Let's see if this has any chance to get in.

DaanDeMeyer commented 1 month ago

@poettering No nsswitch.conf?

cgwalters commented 1 month ago

Two notes:

poettering commented 1 month ago

@poettering No nsswitch.conf?

For now, not. This one only moves the NSS databases. If this works out I'll look into moving non-database files too, i.e. nsswitch.conf, ld.so.conf, resolv.conf, host.conf, shells. (Won't bother with nscd.conf, because nscd is obsolete, and /etc/rpc is not a concept i grok)

poettering commented 1 month ago

Two notes:

  • The nss-altfiles model is additive with /etc, which is a different thing (and often desirable)

Ah, interesting. Weirdly, /etc/aliases (and only that) in glibc's enumeration logic contains an "include" logic. if /etc/passwd had that you could use that i guess..

But I guess people who have both sets of files and want them merged should just continue to use nss-altfiles. I'll mention this in an updated commit msg, should i need to update the patch.

  • There is also /usr/lib/passwd vs /usr/share/passwd here too (which is fine, I think we actually need to do that to be compatible with systems using nss-altfiles today)

In my view of the world, stuff private to some package should be in /usr/lib/<package>, and stuff shared between packages should be in /usr/share/, hence I picked the latter here, because these files have various stakeholders I guess.

cgwalters commented 1 month ago

One thing I'd known for a while, but came up as part of the glibc patch is that there's a conflict between what OpenSUSE has been doing and what ostree has been doing. Basically that conflict is that ostree unconditionally has /usr/etc be what was installed by project as /etc - it's somewhat of an implementation detail (and yes arguably it should have been /usr/lib/ostree/default-etc or something, but I had also hoped to make it a standard but never drove that well...a lack of will for coordination on my part, also long predates uapi etc).

The conflict is that in the ostree model we can't have both /etc and /usr/etc in the image, because it's ostree that owns merging /usr/etc with /etc.

Anyways, let me summarize this briefly here (but we should file a dedicated tracker here perhaps?); there are basically at least 4-5 major approaches to /etc I know of:

The first observation I will make is that the "uapi ideal" is generally compatible with the other 3. The other 3 are attempts to deal with the fact that we don't live in that "uapi ideal" (and honestly, I doubt my use cases will before I retire...).

What I think we really need to resolve here is the conflict between the OpenSUSE approach and the ostree one because we are giving upstreams conflicting guidance. It seems to me that both should change. I think OpenSUSE should stop encouraging people to put /etc/foo.conf in /usr/etc/foo.conf because when we're doing a move and patching upstreams anyways, let's clean up the name spacing here and have it be /usr/lib/foo/foo.conf or so.

I may also change ostree...I need to dig in some but the flatcar model seems appealing. Or in the short term (it may be painful) I could try to change ostree to use /usr/lib/ostree/default-etc or so.

For example https://github.com/uapi-group/specifications/issues/100 making things conditional to move to /usr/etc are too crude IMO.

thkukuk commented 1 month ago

On Fri, Sep 27, 2024 at 1:51 PM Colin Walters @.***> wrote:

One thing I'd known for a while, but came up as part of the glibc patch is that there's a conflict between what OpenSUSE has been doing and what ostree has been doing. Basically that conflict is that ostree unconditionally has /usr/etc be what was installed by project as /etc - it's somewhat of an implementation detail (and yes arguably it should have been /usr/lib/ostree/default-etc or something, but I had also hoped to make it a standard but never drove that well...a lack of will for coordination on my part, also long predates uapi etc).

The conflict is that in the ostree model we can't have both /etc and /usr/etc in the image, because it's ostree that owns merging /usr/etc with /etc.

Anyways, let me summarize this briefly here (but we should file a dedicated tracker here perhaps?); there are basically at least 4 major approaches to /etc I know of:

  • What is promoted here, /etc starts out empty (let's call this the "uapi ideal", though I'm not entirely sure it's truly my ideal, but that's an aside)
  • What flatcar is doing (overlayfs for /etc with the lower being the defaults as installed by upstream, and /etc is a mutable persistent overlayfs upper)
  • What ostree is doing; a userspace 3 way merge; I think this is equivalent or almost equivalent to the flatcar approach (gotta remember, ostree long predates overlayfs, the ubiquitous Swiss Army knife for filesystem tricks we do today)
  • What OpenSUSE is doing (I don't quite know exactly, but it seems to rely on having upstreams install stuff in /usr/etc)

We follow https://uapi-group.org/specifications/specs/configuration_files_specification/ if the format of the configuration file allows it. If not, we only provide a fallback config if there is none in /etc. In which case admins have to manually merge changes. /usr/etc is only used for configuration files where no upstream really exists, like "services" or" protocols" or so. I personally don't like /usr/etc, too, and I had it never on the list of options, but this is what everybody did propose to accept this change. Everything like /usr/share/... or similar got vetoed.

Message ID: @.***>

Thorsten

-- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

pemensik commented 1 month ago

Any DNS caching service except systemd-resolved expects configuration in /etc/ only. With more often configured fixed DNS over TLS (or similar) service, they may have mostly static configuration, at least in some situation. But I think DNS service configuration should deserve separate issue. Especially when podman and libvirt might want to register local machine instance names to running service and not to support just single specific implementation.

But I expect they are not considered as an good alternative by systemd people, who favour their own systemd-resolved.

septatrix commented 1 month ago

Here are some more where I currently need tmpfile definitions to establish symlinks from /etc to /usr/share/factory for them to work properly:

ReillyBrogan commented 1 month ago

Lots of PAM stuff

linux-pam supports libeconf, so pam_envj, pam_limits, and pam_namespace should work if you have the --enable-vendordir flag with the configure script. We do that on Solus and it works fine. Not sure about pam_sepermit, but if it doesn't work I imagine some of the econf code can be reused with it as well.

septatrix commented 1 month ago

Lots of PAM stuff

linux-pam supports libeconf, so pam_envj, pam_limits, and pam_namespace should work if you have the --enable-vendordir flag with the configure script

It seems like Fedora had it enabled at some point but removed it again later

septatrix commented 1 month ago

Fontconfig is another contender

ReillyBrogan commented 1 month ago

Fontconfig is another contender

Fontconfig is particularly problematic as it's often built and then statically linked into various applications (Chromium/Electron does this commonly) and will have sandboxes mount the /etc/fonts directory to be read by yet another build of fontconfig outside of the distributions control.

Just making fontconfig hermetic-usr compatible will lead to fairly broken experiences for end-users (we tried it on Solus). I was able to make it work but it took creating a "stub" fontconfig config file in /usr which tmpfiles then creates a symlink at /etc/fonts/fonts.conf to.

References:

septatrix commented 1 month ago

Fontconfig is particularly problematic as it's often built and then statically linked into various applications (Chromium/Electron does this commonly) and will have sandboxes mount the /etc/fonts directory to be read by yet another build of fontconfig outside of the distributions control.

Yes, updates to fontconfig might take a while to trickle into projects which vendor their own copy. However, I do not think the sandbox is too large of a problem in itself as /usr is generally also included in the sandbox (or at least /usr/share/fontconfig) as most files from /etc/fonts/conf.d/ are symlinks to it.

evelikov commented 1 month ago

Here are a few more from my Arch machine:

septatrix commented 4 days ago

Maybe not that vital but WSL: https://github.com/microsoft/WSL/issues/12299