homed activation fails for first login attemp

[obi1]

systemd version the issue has been seen with

249

Used distribution

Gentoo

Linux kernel version used (uname -a)

Linux fenrir 5.14.0 #1 SMP Mon Aug 30 10:00:40 CEST 2021 x86_64 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz GenuineIntel GNU/Linux

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

successful login after entering username and password.

Unexpected behaviour you saw

successful login only after entering the password for the second time. It doesn't matter if a correct, wrong or empty password is entered, there is no feedback just a second password prompt. After entering the correct password in the second prompt login is successful.

Steps to reproduce the problem

Boot up system. Enter a homed username and password.

Additional program output to the terminal or log subsystem illustrating the issue

Sep 02 13:54:11 fenrir systemd-homed[362]: obitest: changing state inactive → activating-for-acquire
Sep 02 13:54:11 fenrir systemd-homework[404]: None of the supplied plaintext passwords unlock the user record's hashed passwords.
Sep 02 13:54:11 fenrir systemd-homed[362]: Activation failed: Required key not available
Sep 02 13:54:11 fenrir systemd-homed[362]: obitest: changing state activating-for-acquire → inactive
Sep 02 13:54:11 fenrir systemd-homed[362]: Got notification that all sessions of user obitest ended, deactivating automatically.
Sep 02 13:54:11 fenrir systemd-homed[362]: Home obitest already deactivated, no automatic deactivation needed.
Sep 02 13:59:12 fenrir systemd-homed[362]: obitest: changing state inactive → activating-for-acquire
Sep 02 13:59:12 fenrir systemd-homework[423]: Provided password unlocks user record.
Sep 02 13:59:12 fenrir systemd-homework[423]: Allocated additional 26.2G.
Sep 02 13:59:12 fenrir systemd-homework[423]: Setting up loopback device /dev/loop0 completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Setting up LUKS device /dev/mapper/home-obitest completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Provided password unlocks user record.
Sep 02 13:59:13 fenrir systemd-homework[423]: Probing file system completed (found btrfs).
Sep 02 13:59:13 fenrir systemd-homework[423]: File system check completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Mounting file system completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Read embedded .identity file.
Sep 02 13:59:13 fenrir systemd-homework[423]: Provided password unlocks user record.
Sep 02 13:59:13 fenrir systemd-homework[423]: Reconciling user identities completed (host and header version were identical).
Sep 02 13:59:13 fenrir systemd-homework[423]: Reconciling embedded user identity completed (host and embedded version were identical).
Sep 02 13:59:13 fenrir systemd-homework[423]: Recursive changing of ownership not necessary, skipped.
Sep 02 13:59:13 fenrir systemd-homework[423]: Synchronized disk.
Sep 02 13:59:13 fenrir systemd-homework[423]: Moving to final mount point /home/obitest completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Everything completed.
Sep 02 13:59:13 fenrir systemd-homework[423]: Image size is 26.2G, file system size is 26.2G, file system payload size is 26.2G, file system free is 26.2G.
Sep 02 13:59:13 fenrir systemd-homed[362]: Home obitest is signed exclusively by our key, accepting.
Sep 02 13:59:13 fenrir systemd-homed[362]: obitest: changing state activating-for-acquire → active

Normal users (non homed) login successfuly on first attempted password. Downgrading to previous systemd version (248) solves the issue. If logging in with the same user after logout it also works on the first try as expected.

[gdonval]

More info, setting SYSTEMD_LOG_LEVEL=debug...

Nov 30 11:15:23 heaven systemd-homed[917]: Sending to worker: {..., "secret":{}}
[...]
Nov 30 11:15:23 heaven systemd-homed[917]: Successfully forked off '(sd-homework)' as PID 1510.
Nov 30 11:15:23 heaven systemd-homed[917]: username: changing state dirty → activating-for-acquire
Nov 30 11:15:23 heaven systemd-homework[1510]: None of the supplied plaintext passwords unlock the user record's hashed passwords.
Nov 30 11:15:23 heaven systemd-homed[917]: Worker reported error code ENOKEY.
Nov 30 11:15:23 heaven systemd-homed[917]: Activation failed: Required key not available
Nov 30 11:15:23 heaven systemd-homed[917]: Sent message type=error sender=n/a ...
Nov 30 11:15:23 heaven systemd-homed[917]: username: changing state activating-for-acquire → dirty
Nov 30 11:15:23 heaven systemd-homed[917]: Got notification that all sessions of user username ended, deactivating automatically.
Nov 30 11:15:23 heaven systemd-homed[917]: Home username already deactivated, no automatic deactivation needed.
Nov 30 11:15:24 heaven systemd-homed[917]: Got message type=method_call sender=:1.48 destination=org.freedesktop.home1 path=/org/freedesktop/home1 interface=org.freedes>
Nov 30 11:15:24 heaven systemd-homed[917]: Sending to worker: {...,"secret":{"password":["mysecretpassword"]}}
Nov 30 11:15:24 heaven systemd-homed[917]: Successfully forked off '(sd-homework)' as PID 1511.
Nov 30 11:15:24 heaven systemd-homed[917]: username: changing state dirty → activating-for-acquire
Nov 30 11:15:24 heaven systemd-homework[1511]: Provided password unlocks user record.

The important log bits are these:

Nov 30 11:15:23 heaven systemd-homed[917]: Sending to worker: {..., "secret":{}}
[...]
Nov 30 11:15:24 heaven systemd-homed[917]: Sending to worker: {...,"secret":{"password":["myversecretpasswordthatnowappearsinthelogs"]}}

So the first time around, the plaintext password is not forwarded to the worker.

PAM configuration

I suspect something is not quite right in there

# /etc/pam.d/system-login

#%PAM-1.0

auth       required   pam_shells.so
auth       requisite  pam_nologin.so
auth       include    system-auth

account    required   pam_access.so
account    required   pam_nologin.so
account    include    system-auth

password   include    system-auth

session    optional   pam_loginuid.so
session    optional   pam_keyinit.so       force revoke
session    include    system-auth
session    optional   pam_motd.so
session    optional   pam_mail.so          dir=/var/spool/mail standard quiet
-session   optional   pam_systemd.so
session    required   pam_env.so           user_readenv=1

# /etc/pam.d/system-auth

#%PAM-1.0

auth       required                    pam_faillock.so      preauth
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth       [success=2 default=ignore]  pam_unix.so          try_first_pass nullok
-auth      [success=1 default=ignore]  pam_systemd_home.so
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.

-account   [success=1 default=ignore]  pam_systemd_home.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

-password  [success=1 default=ignore]  pam_systemd_home.so
password   required                    pam_unix.so          try_first_pass nullok shadow sha512
password   optional                    pam_permit.so

session    required                    pam_limits.so
session    required                    pam_unix.so
session    optional                    pam_permit.so

I suspect this is not quite right, but this is the file as provided by my distribution and don't want to screw up.

---

As an aside, I know it's debug mode and I know the logs are only root accessible but displaying unhashed passwords is probably very very wrong.

[gdonval]

Adding pam_systemd_home.so to session ans use_authtok in password (as per recommendations) does not solve the issue.

[poettering]

It's actually intended behaviour: i.e. we only acquire "cheap" passwords for the first authentication attempt, and if that then fails, we'll interactively ask the user for a password. (cheap means: pw via env var or keyring, i.e. where we can just ask and have it without interactivity or anything) why? simply because passwords are just one authentication logic, there are possibly others, suchs as pkcs#11 or fido2, and in that case we don't want to ask the user pointlessly for a password, but for a request to touch the token, or ask for a PIN or so. and even if fido2/pkcs#11 is not configured, the user might not actually have a pw set, but only a recovery key, and in that case we should ask for the recovery key too.

[gdonval]

So gdm asking me to type my password twice is intended behaviour?

---

There might be some confusion between what we see in the logs here and what gdm (in particular) is doing.

[poettering]

It doesnt do that here. Its probably something else in your pam stack that triggers the 2nd request. What does your pam stack look like?

[gdonval]

As specified earlier...

Relevant files:

# /etc/pam.d/system-login

#%PAM-1.0

auth       required   pam_shells.so
auth       requisite  pam_nologin.so
auth       include    system-auth

account    required   pam_access.so
account    required   pam_nologin.so
account    include    system-auth

password   include    system-auth

session    optional   pam_loginuid.so
session    optional   pam_keyinit.so       force revoke
session    include    system-auth
session    optional   pam_motd.so
session    optional   pam_mail.so          dir=/var/spool/mail standard quiet
-session   optional   pam_systemd.so
session    required   pam_env.so           user_readenv=1

# /etc/pam.d/system-auth

#%PAM-1.0

auth       required                    pam_faillock.so      preauth
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth       [success=2 default=ignore]  pam_unix.so          try_first_pass nullok
-auth      [success=1 default=ignore]  pam_systemd_home.so
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.

-account   [success=1 default=ignore]  pam_systemd_home.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

-password  [success=1 default=ignore]  pam_systemd_home.so
password   required                    pam_unix.so          try_first_pass nullok shadow sha512
password   optional                    pam_permit.so

session    required                    pam_limits.so
session    required                    pam_unix.so
session    optional                    pam_permit.so

---

From the start:

#/etc/pam.d/gdm-password 

#%PAM-1.0

auth       include                     system-local-login
auth       optional                    pam_gnome_keyring.so

account    include                     system-local-login

password   include                     system-local-login
password   optional                    pam_gnome_keyring.so use_authtok

session    include                     system-local-login
session    optional                    pam_gnome_keyring.so auto_start

#/etc/pam.d/system-local-login 

#%PAM-1.0

auth      include   system-login
account   include   system-login
password  include   system-login
session   include   system-login

(and system-login is provided in the previous section of this post).

[poettering]

I guess pam_unix asks for a pw before pam_systemd_home can. See:

https://www.freedesktop.org/software/systemd/man/pam_systemd_home.html

For the right order

[gdonval]

Order is already respected. The only piece that is missing is pam_systemd_home in session. I have added it, in the correct order, to no avail.

[poettering]

maybe swap the modules in the auth chain too, so that pam_systemd_home can do its thing first

[gdonval]

Hum, swapping the modules did exactly what I predicted: it made me completely unable to log in :D (I'm blaming my lack of PAM knowledge as I probably did something very wrong...)

[gdonval]

So I wonder whether there is a better way but indeed swapping

auth       [success=2 default=ignore]  pam_unix.so          try_first_pass nullok
-auth      [success=1 default=ignore]  pam_systemd_home.so

to

-auth      [success=2 default=ignore]  pam_systemd_home.so
auth       [success=1 default=ignore]  pam_unix.so          try_first_pass nullok

works on arch (the key piece of information not to bork your configuration here is that success=n skips the next n lines if auth is successful so careful if you have that in your configuration). I am pretty sure there are options to retain the password when advancing from pam_unix.so to pam_systemd_home.so in the original order but this works.

@obi1 I can't reproduce your problem though. I suppose sharing your pam config would help, seeing how it works now with the same version as yours.