search

SSSD / sssd

A daemon to manage identity, authentication and authorization for centrally-managed systems.
https://sssd.io
GNU General Public License v3.0
604 stars 247 forks source link

sssd_pam segfaults during password-based SSH-login #7061

Closed elpres closed 11 months ago

elpres commented 11 months ago

Immediately after upgrading a server from Fedora 38 to 39 SSH started rejecting password-authenticated connection attempts with "Permission denied". Luckily it was still possible to log in with a Kerberos ticket and I discovered that sssd_pam was crashing:

Relevant section from the journal: Nov 29 11:11:21 server001.domain001.local krb5_child[362908]: Pre-authentication failed: Invalid argument Nov 29 11:11:21 server001.domain001.local audit[362884]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=362884 comm="sssd_pam" exe="/usr/libexec/sssd/sssd_pam" sig=11 res=1 Nov 29 11:11:21 server001.domain001.local kernel: sssd_pam[362884]: segfault at 0 ip 000055c7f7510723 sp 00007ffc8acc8db0 error 4 in sssd_pam[55c7f74ec000+28000] likely on CPU 57 (core 1, socket 3) Nov 29 11:11:21 server001.domain001.local kernel: Code: c2 48 8d 05 77 67 00 00 48 89 45 80 48 89 c7 31 c0 e8 f1 d7 fd ff 48 c7 85 70 ff ff ff 01 00 00 00 49 8b 44 24 18 48 8b 5d a0 <48> 8b 30 44 0f b6 7b 48 48 89 df e8 3d cc fd ff 48 c7 45 98 00 00 Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=217 op=LOAD Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=218 op=LOAD Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=219 op=LOAD Nov 29 11:11:21 server001.domain001.local systemd[1]: Started systemd-coredump@3-362909-0.service - Process Core Dump (PID 362909/UID 0). Nov 29 11:11:21 server001.domain001.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@3-362909-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Nov 29 11:11:21 server001.domain001.local systemd-coredump[362910]: Process 362884 (sssd_pam) of user 0 dumped core. Module tdb.so from rpm libldb-2.8.0-1.fc39.x86_64 Module skel.so from rpm libldb-2.8.0-1.fc39.x86_64 Module server_sort.so from rpm libldb-2.8.0-1.fc39.x86_64 Module sample.so from rpm libldb-2.8.0-1.fc39.x86_64 Module rdn_name.so from rpm libldb-2.8.0-1.fc39.x86_64 Module paged_searches.so from rpm libldb-2.8.0-1.fc39.x86_64 Module memberof.so from rpm sssd-2.9.3-1.fc39.x86_64 Module mdb.so from rpm libldb-2.8.0-1.fc39.x86_64 Module liblmdb.so.0.0.0 from rpm lmdb-0.9.31-2.fc39.x86_64 Module libldb-tdb-err-map.so from rpm libldb-2.8.0-1.fc39.x86_64 Module libldb-key-value.so from rpm libldb-2.8.0-1.fc39.x86_64 Module libldb-mdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64 Module libldb-tdb-int.so from rpm libldb-2.8.0-1.fc39.x86_64 Module ldb.so from rpm libldb-2.8.0-1.fc39.x86_64 Module libcrypt.so.2 from rpm libxcrypt-4.4.36-2.fc39.x86_64 Module libssl.so.3 from rpm openssl-3.1.1-4.fc39.x86_64 Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-11.fc39.x86_64 Module libevent-2.1.so.7 from rpm libevent-2.1.12-9.fc39.x86_64 Module ldap.so from rpm libldb-2.8.0-1.fc39.x86_64 Module asq.so from rpm libldb-2.8.0-1.fc39.x86_64 Module libpath_utils.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libz.so.1 from rpm zlib-1.2.13-4.fc39.x86_64 Module libzstd.so.1 from rpm zstd-1.5.5-4.fc39.x86_64 Module liblzma.so.5 from rpm xz-5.4.4-1.fc39.x86_64 Module liblz4.so.1 from rpm lz4-1.9.4-4.fc39.x86_64 Module libcap.so.2 from rpm libcap-2.48-7.fc39.x86_64 Module libsss_cert.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libcollection.so.4 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libref_array.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libini_config.so.5 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc39.2.x86_64 Module libunistring.so.5 from rpm libunistring-1.1-5.fc39.x86_64 Module libdbus-1.so.3 from rpm dbus-1.14.10-1.fc39.x86_64 Module libcrypto.so.3 from rpm openssl-3.1.1-4.fc39.x86_64 Module libkeyutils.so.1 from rpm keyutils-1.6.1-7.fc39.x86_64 Module libkrb5support.so.0 from rpm krb5-1.21.2-2.fc39.x86_64 Module libcom_err.so.2 from rpm e2fsprogs-1.47.0-2.fc39.x86_64 Module libk5crypto.so.3 from rpm krb5-1.21.2-2.fc39.x86_64 Module libkrb5.so.3 from rpm krb5-1.21.2-2.fc39.x86_64 Module libeconf.so.0 from rpm libeconf-0.5.2-1.fc39.x86_64 Module libaudit.so.1 from rpm audit-3.1.2-5.fc39.x86_64 Module libtalloc.so.2 from rpm libtalloc-2.4.1-1.fc39.x86_64 Module libtevent.so.0 from rpm libtevent-0.15.0-1.fc39.x86_64 Module libdhash.so.1 from rpm ding-libs-0.6.2-54.fc39.x86_64 Module libsss_sbus.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libsss_iface.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libsystemd.so.0 from rpm systemd-254.5-2.fc39.x86_64 Module libsss_debug.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libsss_child.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libsss_crypt.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libtdb.so.1 from rpm libtdb-1.4.9-1.fc39.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-5.fc39.x86_64 Module libldb.so.2 from rpm libldb-2.8.0-1.fc39.x86_64 Module libpopt.so.0 from rpm popt-1.19-3.fc39.x86_64 Module libsss_util.so from rpm sssd-2.9.3-1.fc39.x86_64 Module libsss_certmap.so.0 from rpm sssd-2.9.3-1.fc39.x86_64 Module libgssapi_krb5.so.2 from rpm krb5-1.21.2-2.fc39.x86_64 Module libpam.so.0 from rpm pam-1.5.3-3.fc39.x86_64 Module sssd_pam from rpm sssd-2.9.3-1.fc39.x86_64 Stack trace of thread 362884: #0 0x000055c7f7510723 pam_passkey_auth_send.isra.0 (sssd_pam + 0x2d723) #1 0x000055c7f75118a0 pam_passkey_get_user_done (sssd_pam + 0x2e8a0) #2 0x000055c7f750ea11 pam_passkey_get_mapping_done (sssd_pam + 0x2ba11) #3 0x00007f9e3f364e40 tevent_common_invoke_immediate_handler (libtevent.so.0 + 0xbe40) #4 0x00007f9e3f364ea2 tevent_common_loop_immediate (libtevent.so.0 + 0xbea2) #5 0x00007f9e3f368a22 epoll_event_loop_once (libtevent.so.0 + 0xfa22) #6 0x00007f9e3f360894 std_event_loop_once (libtevent.so.0 + 0x7894) #7 0x00007f9e3f362e1b _tevent_loop_once (libtevent.so.0 + 0x9e1b) #8 0x00007f9e3f362f6b tevent_common_loop_wait (libtevent.so.0 + 0x9f6b) #9 0x00007f9e3f360914 std_event_loop_wait (libtevent.so.0 + 0x7914) #10 0x00007f9e3f5e9a6f server_loop (libsss_util.so + 0x50a6f) #11 0x000055c7f74f1a63 main (sssd_pam + 0xea63) #12 0x00007f9e3f18914a __libc_start_call_main (libc.so.6 + 0x2814a) #13 0x00007f9e3f18920b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2820b) #14 0x000055c7f74f2245 _start (sssd_pam + 0xf245) ELF object binary architecture: AMD x86-64 Nov 29 11:11:21 server001.domain001.local audit[362896]: USER_AUTH pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=PAM:authentication grantors=? acct="user001" exe="/usr/sbin/sshd" hostname=192.168.2.69 addr=192.168.2.69 terminal=ssh res=failed' Nov 29 11:11:21 server001.domain001.local audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-coredump@3-362909-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Nov 29 11:11:21 server001.domain001.local systemd[1]: systemd-coredump@3-362909-0.service: Deactivated successfully. Nov 29 11:11:21 server001.domain001.local sssd_pam[362919]: Starting up Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=219 op=UNLOAD Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=218 op=UNLOAD Nov 29 11:11:21 server001.domain001.local audit: BPF prog-id=217 op=UNLOAD Nov 29 11:11:22 server001.domain001.local abrt-server[362925]: Deleting problem directory ccpp-2023-11-29-11:11:21.843487-362884 (dup of ccpp-2023-11-28-13:33:41.300493-2042) Nov 29 11:11:22 server001.domain001.local abrt-notification[363023]: Process 2042 (sssd_pam) crashed in pam_passkey_auth_send.isra.0() Nov 29 11:11:24 server001.domain001.local sshd[362896]: Failed password for user001 from 192.168.2.69 port 47794 ssh2 Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=session fp=? direction=both spid=362897 suid=74 rport=47794 laddr=192.168.2.88 lport=22 exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success' Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=server fp=SHA256:40:c8:ad:c8:51:db:93:78:c3:08:4a:11:ad:e8:f3:d9:16:05:16:8f:8c:89:a1:68:f3:c8:2f:db:e0:ae:79:05 direction=? spid=362897 suid=74 exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success' Nov 29 11:11:26 server001.domain001.local sshd[362896]: Connection closed by authenticating user user001 192.168.2.69 port 47794 [preauth] Nov 29 11:11:26 server001.domain001.local audit[362896]: CRYPTO_KEY_USER pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=destroy kind=server fp=SHA256:40:c8:ad:c8:51:db:93:78:c3:08:4a:11:ad:e8:f3:d9:16:05:16:8f:8c:89:a1:68:f3:c8:2f:db:e0:ae:79:05 direction=? spid=362896 suid=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=? res=success' Nov 29 11:11:26 server001.domain001.local audit[362896]: USER_LOGIN pid=362896 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='op=login acct="user001" exe="/usr/sbin/sshd" hostname=? addr=192.168.2.69 terminal=ssh res=failed'

Also, while trying to understand what's wrong I tried logging in from a local account into an AD-backed one, and instead of being asked for a password I'm getting this message that I've never seen before:

Kerberos TGT will not be granted upon login, user experience will be affected.
Insert your passkey device, then press ENTER.

Communication with AD seems to be fine, at least things like getent passwd $USERNAME behave as expected and I can get a ticket with kinit. Both cases (the stack trace and the login message) refer to passkeys, although nothing related to them was ever configured on the server or is desired at the moment. I also manually removed sssd-passkey which seems to have been installed during the upgrade to F39, but with no effect.

sssd.conf.txt sssd_pam.log

What other logs should I provide?

alexey-tikhonov commented 11 months ago

Hi,

What other logs should I provide?

'rpm -q sssd' output and coredump would be great.

elpres commented 11 months ago 
# rpm -q sssd
sssd-2.9.3-1.fc39.x86_64

# rpm -qa | grep sssd-
sssd-common-pac-2.9.3-1.fc39.x86_64
sssd-krb5-common-2.9.3-1.fc39.x86_64
sssd-proxy-2.9.3-1.fc39.x86_64
sssd-ad-2.9.3-1.fc39.x86_64
sssd-krb5-2.9.3-1.fc39.x86_64
sssd-ldap-2.9.3-1.fc39.x86_64
sssd-ipa-2.9.3-1.fc39.x86_64
sssd-2.9.3-1.fc39.x86_64
sssd-nfs-idmap-2.9.3-1.fc39.x86_64
sssd-client-2.9.3-1.fc39.x86_64
sssd-common-2.9.3-1.fc39.x86_64
sssd-dbus-2.9.3-1.fc39.x86_64
sssd-tools-2.9.3-1.fc39.x86_64
sssd-idp-2.9.3-1.fc39.x86_64
sssd-kcm-2.9.3-1.fc39.x86_64

I scrubbed sensitive data from the log above, but it will probably be fully readable in the coredump. How can I make it available in a secure way?

justin-stephenson commented 11 months ago

Hi,

You can try adding pam_passkey_auth = False to the [pam] section of sssd.conf as a workaround for now. The new passkey feature allows storing passkey data in the AD altSecurityIdentities attribute so there is an issue here we will investigate.

I scrubbed sensitive data from the log above, but it will probably be fully readable in the coredump. How can I make it available in a secure way?

You can email our sssd developer team list sssd-maint@redhat.com (or me directly at jstephen@redhat.com) if it's okay for you.

elpres commented 11 months ago

@justin-stephenson Thanks for the tip about pam_passkey_auth = False, it indeed resolved the problem! Also, I emailed the coredump to you.

justin-stephenson commented 11 months ago

Hi @elpres thank you for the coredump. Would you be able to test the fix (removing pam_passkey_auth = False) from the COPR build provided in https://github.com/SSSD/sssd/pull/7066 ?

alexey-tikhonov commented 11 months ago

( https://copr.fedorainfracloud.org/coprs/g/sssd/pr7066/ )

elpres commented 11 months ago

The fix looks good to me; logins went back to normal. Thank you for the quick work on this!

alexey-tikhonov commented 11 months ago

Pushed PR: https://github.com/SSSD/sssd/pull/7066