Account lockout

[c0c0n3]

Users can't log in through SSH with passwords anymore after deploying Age-encrypted secrets and rebooting the system. This happens both if you use Agenix or our own Agez module. Notice passwords now are yescrypt-hashed which is what PAM expects---e.g. run rg '^password ' /etc/pam.d/*. (At least in NixOS 23.11; also see this about it.)

Here's the steps to reproduce

1. Boot the dev VM.
2. Deploy the snake oil vault (odbox.vault.snakeoil.enable = true).
3. Log out of any user session.
4. Deploy the Agez vault (odbox.vault.agez.enable = true) with a fresh set of secrets.
5. Log in through SSH with both the admin and root users using a password login. This should work.
6. Ditto for direct tty logins.
7. Shut down the VM.
8. Boot the VM.
9. Log in through SSH with both the admin and root users using a password login. This should not work in both cases.
10. Log in through the tty with both the admin and root users. This should still work.

Even if you retry (9) after 30 mins, the accounts are still locked out. Notice sometimes you get slight variations on the theme where e.g. (9) works but (10) won't. Also, this is what I got when I was using SHA512 hashes instead of yescrypt

1. Boot the dev VM.
2. Deploy the snake oil vault (odbox.vault.snakeoil.enable = true).
3. Log out of any user session.
4. Deploy the Agez vault (odbox.vault.agez.enable = true) with a fresh set of secrets.
5. Log in through SSH with both the admin and root users using a password login. This should work.
6. Shut down the VM and boot the VM again.
7. Log in through SSH with both the admin and root users using a password login. This should not work in both cases.
8. Retry after half an hour. It shouldn't work.
9. But SSH+key should: ssh -i nodes/devm-aarch64/generated/ssh/id_ed25519 admin@localhost
10. Now logging in through SSH with both the admin and root users using a password login should work again.
11. Log in through the tty with both the admin and root users. This should not work anymore.

Notice passwords are actually set correctly in shadow. e.g.

$ mkpasswd -m sha-512 -S $(sudo rg admin /etc/shadow | cut -d ':' -f 2 | cut -d '$' -f 3)`
# enter admin pass, you'll see the output is the same as
$ sudo rg admin /etc/shadow | cut -d ':' -f 2

What a fantabulous mess!

[c0c0n3]

oh, forgot to mention. If you try this long enough, after a couple of reboots you may wind up with both tty and SSH logins being bust. The only way you can log in with either user is with SSH+key. Go figure!

[c0c0n3]

PAM's actually what makes TTY and SSH logins as journalctl reports

Apr 04 19:19:55 devm login[896]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/ttyAMA0 ruser= rhost=  user=admin
Apr 04 19:20:23 devm sshd[1083]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2  user=admin

[c0c0n3]

Well, it turns out Agenix/Agez/PAM actually have nothing to do with the account lockout. And the account lockout isn't really a lockout. What? Yep, that's right. The problem is between VS code and the Apple Terminal on my laptop! In fact, I was copy-pasting the passwords from VS Code into the terminal (SSH password prompt) and the password string that got sent over to the dev VM contained ANSI (colour) escape codes!

I realised that only after tracing sys calls sshd and its sub-processes were making in the dev VM:

$ sudo strace -f -s 10000 -p $(cat /run/sshd.pid) -o dump.txt

Now after getting the middle finger from the server on an SSH login with a password of root2, I looked at the dump and found the actual password SSH got was \21\33[200~root2\33[201~, see the snippet below from the dump.

1116  write(5, "\0\0\0\1\0\0\0\21\33[200~root2\33[201~", 25) = 25