Drop requirement on nss-altfiles, use systemd sysusers

cgwalters commented 9 years ago

Currently, all %posts are run on the server side. This means we need nss-altfiles to help OSTree's 3 way merge of /etc.

The systemd people added systemd-sysusers, with the intention that the updates happen on boot.

A new hybrid option is running systemd-sysusers in the new root just before reboot. This would avoid system mutation on boot with all the problems that entails (e.g. booting read-only should work, etc.)

cgwalters commented 9 years ago

Here's an idea too; we don't necessarily need to force all RPMs to use sysusers. We could synthesize sysusers entries from them, as long as we know systemd-sysusers is in the target OS.

And in fact, we may need to synthesize anyways, for the case where the tree content contains uid-specific data (e.g. /usr/lib64/dbus-1/dbus-daemon-launch-helper, /etc/polkit-1/rules.d). For those cases we have to pick a uid at compose time, then tell sysusers to match that uid.

xnox commented 9 years ago

How about fixing shadow-utils to be nss-altfiles aware? That is read altfiles groups, when doing append operations to them?

cgwalters commented 9 years ago

I wrote a patch for that: http://lists.alioth.debian.org/pipermail/pkg-shadow-devel/2014-April/010149.html

However at the time, it was painful to depend on the shadow-utils patch. It still isn't applied upstream. I stopped pushing for it after I was able to work around it in rpm-ostree.

xnox commented 9 years ago

Right, I'm working on something similar. I have a set of patches that makes shadow-utils operate with /etc/* files missing (e.g. without /etc/passwd /etc/shadow etc.) and I am starting adding bits to load up nss-altfiles databases in a sensible manner. I'll check your patch to see which other bits I can borrow. Hopefully this can be fixed in shadow-utils and Linux-PAM directly, as imho both should be able to support altfiles.

cgwalters commented 9 years ago

cgwalters commented 8 years ago

An approach to dealing with setuid binaries in /usr is https://bugzilla.gnome.org/show_bug.cgi?id=722984

cgwalters commented 8 years ago

One thing I discussed with @jlebon today is we realized that for /etc, we end up making physical copies (or reflinks), which means we have an opportunity to chown them on the client machine. We can store the uid for files in /etc in a separate metadata file that ostree processes when doing deployments.

cgwalters commented 7 years ago

Systemd now has support for dynamic users: https://github.com/systemd/systemd/commit/4ffe24797cc881f1dc95f39badf6facd8061117e This addresses some of the problem space in that for probably a good chunk of services, they don't need persistent entries in any form.

dustymabe commented 7 years ago

any progress on this?

cgwalters commented 7 years ago

See https://github.com/ostreedev/ostree-releng-scripts/pull/15#issuecomment-339732974 for some interesting data.

One clear bug here is that our tmpfiles.d generation from /var really needs to synthesize back to names, not ids.

peterbaouoft commented 6 years ago

Hey, it has been a while since I have touched this, and a lot of things happened in rpm-ostree. /me wondering if this issue is still relevant? If it is, can I continue working on it( try it out)? Thanks! :)

cgwalters commented 6 years ago

This is definitely still relevant! It's a hard problem though and fairly invasive and also risky in the sense that we'd be changing an important part about how the system works and need to think about how that interacts with upgrades.

There are various hacks in the codebase that would need to be evaluated for the old/new modes. I also had a discussion recently with someone about this and he pointed out that running systemd-sysusers in %pre might not work because the new files won't have been installed.

But perhaps what we'd end up with is post-processing things back to sysusers at the end of a compose (and after package layering)?

peterbaouoft commented 6 years ago

Thanks for the reply!

It's a hard problem though and fairly invasive and also risky in the sense that we'd be changing an important part about how the system works and need to think about how that interacts with upgrades.

ah, /me will slowly grind it out then, will definitely want to learn more about this part. The only concern is it might be too time consuming to learn and process due to the complexity, but I would like to try it out :p. (if you don't mind ofc)

There are various hacks in the codebase that would need to be evaluated for the old/new modes.

ah, sure, /me will first take a look at those,(hopefully figure some part of them out by myself), then may be have a discussion first if possible? :)

But perhaps what we'd end up with is post-processing things back to sysusers at the end of a compose (and after package layering)

Thanks for the suggestion! I do not know that part very well yet. but I would like to figure that out as well, hopefully can find a way( or adopting this method) after some discussions. (/me praying):D.

peterbaouoft commented 6 years ago

Hi, Sorry for this to be taking so long. There are many parts that I don't have knowledge about, and took time to read & understand them. Do note that, some of the following are based on my own understanding, so there could be a possibility that they are "wrong", but feel free to correct me if you have found some :)!

I would like to write out my findings and doubts in the points below, each point is in bold for visual comparison: NOTE: the sections below might be long:

Nss-altfiles vs systemd-sysusers

The main reason, I think, why we are using nss-altfiles in the first place is because of its ability to read passwd and group files from /lib, and it can help us avoid the complex logic of merging /etc/passwd during OStree 3 way merge. However, some problems can be caused by using it:

1: authselect issue, a requirement of altfiles needs to be there in /etc/nsswitch.conf for passwd and group entries 2: requirement of nss-altfiles pacakge 3: need to perform separate sysusers and users into /usr/lib/passwd and /etc/passwd ( tho not sure if systemd-sysusers share this problem) 4: static uid takes places? ( may be systemd Dynamic user can help that? )

In comparison, systemd-sysusers can may be used to replace nss-altfiles as stated in the issue. It has several advantages: 1: Seems like Arch Linux adopted it 2: Each time during boot, the passwd or groups can be regenerated by systemd-sysusers if files in /usr/lib/sysusers.d/*.conf are configured correctly 3: came together with systemd, and do not necessarily require systemd entry in nsswitch.conf, but we still might need it if we need systemd Dynamic users 4: Users can call systemd-sysuers directly to add missing sysuers from arch linux user's comment

There are some potential problems I have found tho, listing them out here for reference: 1: Pagure issue 2: Seems like systemd-sysuers does not allow one to specify the login shell, and always uses the nologin binary. Might cause problems 3: Does adopting this also require us to separate normal users from sysusers? From documentation, calling systemd-sysusers will add sysusers directly to /etc/passwd. Do we also need to copy those into /usr/lib/passwd ?

Thoughts on Implementation:

Arch Linux's way of doing it seems like to be the following: In there, they have only root in /etc/passwd, and the rest entries are listed in one file /usr/lib/sysusers.d/arch.conf. test package from this issue

I am wondering if we could do the similar thing so maybe all of the sysusers can be regenerated using one file.

And, in rpmostree_context_assemble, instead of calling %pre scripts from rpm files, we can use the conf file with - as ID in it, to automatically assign an ID to it according to the sysusers.d ID section? After that, we collect ID's from existing /etc/passwd and change ownerships of the files like what we did earlier. (Will that work? )

Tho some major challenges will be 1: how do we replace / detect the %pre which does the useradd for rpms? 2: how do we add /usr/lib/sysusers/package.conf to each package? 3: Do we need to worry about interactions from Ostree side?

about how that interacts with upgrades.

I don't know too much about that yet, but I guess we can learn a bit from how arch linux approached this? I personally have not read this tho. Any suggestions for other sources to look at?

Will Dynamic Users work?

This article introduces dynamic user concept associated with systemd services. May be we can utilize that functionality too for services that do not write persistent files?

We might need to figure the following problems out if we want to use this. (Those are from my personal view): 1: How do we know if a package writes persistent files into StateDirectory, CacheDirectory etc. 2: Packages need to come with a systemd unit file in that case, does all packages have that? 3: How do we manually add the DynamicUser term into unit file if 2 applies? 4: How does this DynamicUser concept affect the systemd-sysusers section?

cgwalters commented 6 years ago

Thoughts on Implementation:

I think this would be a new sysusers entry for the check-passwd entry in the treefile. Or probably if it's omitted, we assume it as the default. Then what we do is walk the generated /etc/passwd post-compose (in rpmostree_check_passwd_groups()) and convert it to sysusers.d entries.

It looks like the "sysusers doesn't support login shell" isn't going to be a problem for FA{H,W} at least in an initial check of grep -v /sbin/nologin /usr/lib/passwd.

cgwalters commented 6 years ago

Will Dynamic Users work?

I think this is orthogonal; it's if components (packages) use dynamic users, but we shouldn't be trying to override the "upstream" on this. DynamicUser is really something that the upstream developer should be testing and enabling.

peterbaouoft commented 6 years ago

I think this would be a new sysusers entry for the check-passwd entry in the treefile. Or probably if it's omitted, we assume it as the default. Then what we do is walk the generated /etc/passwd post-compose (in rpmostree_check_passwd_groups()) and convert it to sysusers.d entries.

Thanks for the response @cgwalters , and sorry for the late reply, got distracted by other things and spent a bit more time reading the code base and other work to make sure I understand the parts :p.

I think this way will work :D, but my only concern is that rpmostree_check_passwd_groups is done only in the compose code right? (the function is called viarpmostree_check_passwd and impl_commit_tree). For package layering, if I understand correctly, the logic in rpmostree_context_assemble still suggests that we inject passwd/group into /usr/lib/{group, passwd} directly.

I guess we could do the same conversion over there, WDYT? And if all done correctly, I think we could theoretically remove the logic of migrating sysusers to /usr/lib/passwd and let systemd-sysusers generate them directly into /etc/passwd at boot time. Tho that will be happening at a later time I think, due to its impact for package layering, and ex-unified-core compose path.

peterbaouoft commented 6 years ago

I will first try the approach in compose part then :p. But before I do that, I also wanted to ask if there are any suggestions for speeding up the compose process? It usually took 30 min with my network to complete a compose, and it will be hard to test it out :(. Thanks for the advice and help :-).

cgwalters commented 6 years ago

For package layering, if I understand correctly, the logic in rpmostree_context_assemble still suggests that we inject passwd/group into /usr/lib/{group, passwd} directly.

Right, though I think what we'd do is: detect if /usr/lib/passwd exists, if not, then we don't need to do anything special for package layering; calls to useradd || true as exist in a lot of spec files today will just work out of the box I'd think.

But before I do that, I wanted to ask if there are any suggestions for speeding up the compose process?

There's --cachedir and also --cache-only. I also personally have local mirrors of all Fedora/CentOS content on an external drive; see http://yum.baseurl.org/wiki/YumMultipleMachineCaching

peterbaouoft commented 6 years ago

calls to useradd || true as exist in a lot of spec files today will just work out of the box I'd think.

Ah, I am guessing you meant the "%pre" script for each package here? If so, I think it should work :D, as long as we add the entry back into /usr/lib/sysusers.d. And correct me if I am wrong, the reason why we need to do that(separating sysusers) is to avoid the complex /etc merge happening in ostree in the later stage right?

There's --cachedir and also --cache-only. I also personally have local mirrors of all Fedora/CentOS content on an external drive; see http://yum.baseurl.org/wiki/YumMultipleMachineCaching

\o/, I believe that is exactly what I want, going to learn it and hopefully speed up the test process :D, thanks :).

NOTE: The question I asked is in no rush here, as I probably won't get a chance to touch package layering part any time soon, it is just me curious about those knowledge :p. (hopefully you won't mind)

peterbaouoft commented 6 years ago

Hi, due to some reasons(mainly personal reasons), I would not be able to finish this issue in time before the end of my internship. I would like to have a summary of the discussions, progresses, concerns, and todos here so it will be easier for people to pick the work up in the future (hopefully :p).

Current Progress of the issue:

Have a WIP PR in #1376, the implementation details can be seen in the commit msgs
Able to make a compose following this document and have a tweak of the tree file 1: nss-altfiles package removed in the tree file 2: have the following format for check-passwd and check-groups. Like the following: "check-passwd": { "type": "sysusers", "filename": "passwd" }
Able to rebase and reboot onto the composed tree using rpm-ostree rebase :$commit with some extra changes/requirements: -- Have more updated version of ostree so this ostree commit is included -- Will need to manually go into the deploy and edit /etc/gshadow files, to only have non-system user entries there. This is because systemd-sysusers will error out upon same gshadow entries

TODOS:

Integrate the logic of sysusers into deploying related actions (package layering, upgrade, deploy and rebase). Tho really, only package layering + other deploy actions are needed to be considered. There is discussion about implementation here
Resolve the workaround for shadow files, there are two ways proposed -- Use systemd-tmpfiles, for more info see comment -- Delete shadow files, or empty them, because systemd-sysuser will generate the corresponding user/group entries
Systemd-sysusers still have some corner cases. E.g: we have games:x:12:100:games:/usr/games:/sbin/nologin and games:x:20: . In that case, games has another group as its primary group, and systemd-sysuser will not handle this corner case correctly. (It will default its primary group to games)

CONCERNS:

Interaction between upstream packages, when fedora packages decide to ship systemd-sysuser configuration files. As that seems to be what ArchLinux is doing today
RHEL ships an older version of systemd, which may or may not have systemd-sysusers

peterbaouoft commented 6 years ago

If some parts in the above description, is not clear to you. Please let me know, and I will try my best to reword/ edit it. Apologies for not being able to finish it in time, and thanks =).

peterbaouoft commented 6 years ago

STATUS UPDATE: Hi, I am mostly caught up with this issue, and the progress I have left off last time. To briefly summarize, the following is what needs to be done to get this PR into a good state

1: Finish integrating the sysusers part into compose

Separate condition for skipping nss-altfiles added
Creation of rpm-ostree-base.conf that contains the composed packages sysuser entries
Removal of exisiting /usr/lib/sysusers.d to avoid duplicate declaration of entries from systemd default
Removal of /etc/gshadow by sysusers entries? --> if so, we can replicate the steps in step2

Fortunately, most of them are already done and tested in the existing commits, only slight modifications are required to finish this step

2: Handle interactions from deploying related actions --> mentioned also in the above comment.

Have the logic of remove entries from /etc/passwd complete

3: Integration testing and corner case finding...

~I am hopeful to finish the first two or at least have most of it done~ by the end of the internship. But if not, those will be the steps to take to finish, I will mention the status again in the proposed PR tmr. I will also try to keep an eye on this issue afterwards if possible...

Anyways, thanks a lot for the guidance & help provided while I was working on it. I really appreciate that!

cgwalters commented 5 years ago

Some WIP for interception in https://github.com/projectatomic/rpm-ostree/pull/1679

cgwalters commented 5 years ago

See discussion in https://github.com/projectatomic/rpm-ostree/pull/1376#issuecomment-425914340 for status.

LorbusChris commented 5 years ago

@cgwalters @jlebon @dustymabe: I have a few questions regarding this one and made some assumptions, please correct/clarify.

To me it looks like for this we need the following:

create useradd/groupadd interceptor and parse to sysusers conf during compose
create a sysusers polyfill for RHCOS (as an external lib; we don't need this in Fedora because we ship systemd-sysusers)
add option to run sysusers creation before rebooting to allow for immutable boots
drop nss-altfiles and unblock our move to authselect: https://github.com/pbrezina/authselect/issues/48 & https://github.com/coreos/fedora-coreos-tracker/issues/92

Questions/Thoughts:

What about tmpfiles; how is support for them currently? Do we have an option to create them before reboot? Do we need another polyfill for RHCOS?
How heavily should tmpfiles be used in the spec to create empty dirs? Only in /run or also in %{_localstatedir}=/var, %{_sharedstatedir}=/var/lib or even %{_sysconfdir}=/etc?
It would be great if the packaging guidelines finally changed so that all rpm packages would be able (and required) to add users and dirs/files though sysusers and tmpfiles respectively. (If all packages used sysusers, we wouldn't even need no. 1 from above, the wishful thinker would say) :
```
Source1: %{name}.sysusers
Source2: %{name}.tmpfiles
```

BuildRequires: systemd-rpm-macros %{?systemd_requires}

%pre %sysusers_create_package %{name} %{SOURCE1} %tmpfiles_create_package %{name} %{SOURCE2}

%install install -Dpm 0644 %{SOURCE1} %{buildroot}%{_sysusersdir}/%{name}.conf install -Dpm 0644 %{SOURCE2} %{buildroot}%{_tmpfilesdir}/%{name}.conf

%files %{_sysusersdir}/%{name}.conf %{_tmpfilesdir}/%{name}.conf


Unfortunately all issues regarding this last point that I could find have been closed off:
https://pagure.io/packaging-committee/issue/442
https://fedoraproject.org/wiki/Changes/SystemdSysusers
https://pagure.io/packaging-committee/issue/453

LorbusChris commented 5 years ago

I am really straying off topic a bit:

How would support for rpms with %sysusers_create_package and %tmpfiles_create_package in a Fedora container environment work? Do we have systemd inside the container and does it add users/files according to spec?

cgwalters commented 5 years ago

add option to run sysusers creation before rebooting to allow for immutable boots

You think it'd just be cleaner? Dunno...we also have tmpfiles that works the same way. Why does it matter?

cgwalters commented 5 years ago

How would support for rpms with %sysusers_create_package and %tmpfiles_create_package in a Fedora container environment work? Do we have systemd inside the container and does it add users/files according to spec?

As far as I know, nothing in sysusers requires that systemd be running as PID 1 - it's really just a CLI tool to update /etc/passwd.

podman run --rm --net=host -ti registry.fedoraproject.org/fedora:28 systemd-sysusers

exits with code 0 here.

Now, if what you're saying here is this introduces a requirement on the systemd RPM, which we commonly don't need inside at least inside "microservice style" containers...yeah that's an issue; could theoretically split out the sysusers/tmpfiles binaries into a separate package?

I feel though like the "microservice style" containers are going to be using two-phase style builds anyways, not packaging their code as RPMs. So it doesn't matter.

LorbusChris commented 5 years ago

Interesting, thanks!

The current guidelines encourage the use of %{?systemd_requires} which evals to

Requires(post): systemd 
Requires(preun): systemd 
Requires(postun): systemd

So to me it looks like many rpms one might want to run in a container as microservice may already come with a systemd install time dependency anyway. I wonder if this could be mitigated in the container by just rpm/dnf removing systemd afterwards in the container build?

You think it'd just be cleaner? Dunno...we also have tmpfiles that works the same way. Why does it matter?

Hm, I somehow thought this was a requirement. If it's not, maybe it can be dropped? Not entirely sure I know all implications this might have.

Maybe splitting out a sysusers package would free us of having to rewrite it for RHCOS? Not sure how well that package would be able to communicate with RHCOS's systemd.

cgwalters commented 5 years ago

Cross-linking https://discussion.fedoraproject.org/t/how-can-i-add-myself-to-the-libvirt-group-in-fedora-silverblue/1412

Anyone who wants to e.g. add themselves to the libvirt group should do:

grep -E '^libvirt:' /usr/lib/group >> /etc/group

However for libvirt, IMO it's better to do https://goldmann.pl/blog/2012/12/03/configuring-polkit-in-fedora-18-to-access-virt-manager/

jlebon commented 4 years ago

jlebon commented 3 years ago

Looks like this was unintentionally closed.

cgwalters commented 2 years ago

Something I just learned about today is https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/ids.nix thanks to @mkenigs

cgwalters commented 2 years ago

Hmm, I had a realization here - for any user/groups which are not actually used in the ostree commit (i.e. no files/directories owned by that user/group are mentioned), we can actually just move those to sysusers unconditionally today I believe. That would dramatically shrink the blast radius of this.

Am I missing something here? Did we just miss this obvious big step here?

travier commented 2 years ago

We could probably start by converting our passwd & group files into sysusers files and ship them. Or add a converter to rpm-ostree to create a sysusers files with hardcoded IDs from the passwd & group entries.

cgwalters commented 2 years ago

Yes, I think nothing stops us today from synthesizing sysusers entries from the stuff in passwd.

I think we could make this extra safe by having it be opt-in and conditional. In fact we already have such a thing in etc-group-members. I think we could switch the implementation of that to be generating a sysusers.d entry for the group instead of injecting it into /etc/group.

lucab commented 2 years ago

I think it is useful to first identify all static UIDs/GIDs, and teach those to rpm-ostree. Those are the initial simplest cases to handle, as they can be directly lifted into sysusers fragments with their well-known IDs.

A general translation is a little more complex as we have to handle cases like "add user <foo> to group <bar>" which may cross package boundary. This also applies to users/groups which do not own files in the ostree commit. Also, we may (or may not) face issues with lexicographic order of fragments, and the dependencies between their content.

lucab commented 2 years ago

I've found out that the list of static UIDs and GIDs for Fedora/RHEL/etc can be sourced from https://pagure.io/setup/blob/master/f/uidgid.

cgwalters commented 1 year ago

I think going forward we may want to use https://www.freedesktop.org/software/systemd/man/nss-systemd.html#Static%20Drop-In%20JSON%20User/Group%20Records

cgwalters commented 1 year ago

New thread on moving sysusers to rpm https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NEFOV236FJYS2RED2SEOV5YHDFLDX7DK/

Bigger picture, I think we can continue to make iterative progress on "thinning out" the data in the static passwd and group databases that are shipped in various places; e.g. I think there's a copy in fedora-iot too and silverblue too.

We need to be careful about this; to start I'd say we should only try dropping entries which have an exact match in sysusers. Offhand it looks like e.g. adm may be a good candidate.

travier commented 1 year ago

To make that happen we need two things:

Figure out a way to keep existing users as is on current systems that update to a non nss-altfiles image.
Have new systems populate users/group files from a mix of 'static' /usr/lib/passwd files (for example https://pagure.io/setup/blob/master/f/passwd from https://src.fedoraproject.org/rpms/setup) and sysusers configs on first boot, thus not relying on nss-altfiles anymore.

So for https://github.com/coreos/fedora-coreos-tracker/issues/1524 the goal is not to remove the line from passwd/group but instead to remove nss-altfiles from the image and make sure that the user/group gets created correctly in /etc/passwd & /etc/group.

Systems that update will need to have barrier to go through this migration step before we reset the paswd/group files to the defaults from Fedora.

cgwalters commented 1 year ago

So for https://github.com/coreos/fedora-coreos-tracker/issues/1524 the goal is not to remove the line from passwd/group but instead to remove nss-altfiles from the image

Yes, dropping nss-altfiles would be great. But that's an all-or-nothing thing. Her work will let us drain the duplicate static user/group files that are copied into multiple places in favor of canonicalizing on the bits from the setup package. So it's much more incremental.

HuijingHei commented 1 year ago

Refer to https://github.com/coreos/fedora-coreos-tracker/issues/1524#issuecomment-1623890747 (redirect discussion here)

Probably what we need to do first is add a systemd unit which keys off the firstboot and runs systemd-sysusers --root /sysroot before Ignition runs. That I believe would be safe to do now.

IMU, the unit will create group/users (according to configs under /usr/lib/sysusers.d/) in /etc/{group,passwd}, and the users might be already existed in /usr/lib/{group,passwd}, is this expected?

cgwalters commented 1 year ago

I think sysusers should query the nss subsystem which will include altfiles. So sysusers should not create users which are still in the altfiles database.

HuijingHei commented 1 year ago

I think sysusers should query the nss subsystem which will include altfiles. So sysusers should not create users which are still in the altfiles database.

IMU, as non-use of nss-altfiles in the initramfs, by default sysusers can not find users database in /lib. Look more about https://github.com/coreos/fedora-coreos-config/pull/774, seems need to mount and chroot the target, then sysusers can query the nss subsystem via altfiles, but I might misunderstood.

cgwalters commented 1 year ago

Hmm, yes, the lack of altfiles in the initramfs is potentially a problem. We probably do need to inject it there.

But I didn't think we were running sysusers from the initramfs, but only from the real root?

HuijingHei commented 1 year ago

Hmm, yes, the lack of altfiles in the initramfs is potentially a problem. We probably do need to inject it there.

Sorry, I am confusing about this, if we plan to drop nss-altfiles, why should we inject altfiles in initramfs ? Am I wrong here?

But I didn't think we were running sysusers from the initramfs, but only from the real root?

Yes, we only have bin file systemd-sysusers in initramfs (see ignition-ostree), but do not run from initramfs.

travier commented 1 year ago

More detailed proposal in https://github.com/coreos/fedora-coreos-tracker/issues/155#issuecomment-1688447446

coreos / rpm-ostree