pap: ERROR: Cleartext password does not match "known good" password

Hi experts,

I am newbie to radius.

Now I am building a FreeRadius server to authenticate different users.

Below is the topology:

Topology description: User1/User2/User3 want to login to Linux server via SSH, User1/User2/User3 should be authenticated by FreeRadius server during SSH logining, User1 has already been created at Linux Server before building the central AAA system, its username is already saved to /etc/passwd and its password is already saved to /etc/shadow at Linux Server, it can login to Linux Server success. User2 and User3 is not created at Linux Server.

Now we start a FreeRadius Server with below config:

root@6993e0d8f1dc:/# cat /etc/raddb/clients.conf
client localhost {
    ipaddr = 127.0.0.1
    proto = *
    secret = testing123
    require_message_authenticator = no
    nas_type     = other    # localhost isn't usually a NAS...
    limit {
        max_connections = 16
        lifetime = 0
        idle_timeout = 30
    }
}

client testnet {
    ipaddr      = 173.39.202.128/25
    secret      = testing123
}

root@6993e0d8f1dc:/# cat /etc/raddb/users
bob Cleartext-Password := "test123"
    Reply-Message := "Hello, %{User-Name}"

mario   Cleartext-Password := "test123"
    Reply-Message := "Hello, %{User-Name}"

pi  Cleartext-Password := "test123"
    Reply-Message := "Hello, %{User-Name}"

config items not listed here are using default ones.

at Linux server, we pulled pam radius source code from here, version is 77da6f5028730ded726d50732da379568a9edded, built it and installed pam_radius_auth.so, below is the config items:

 pi@raspberrypi:~ $ cat /etc/pam.d/sshd 
# PAM configuration for the Secure Shell service

auth       sufficient   pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=3

# Standard Un*x authentication.
@include common-auth

account    sufficient   pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=3
# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so

# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account  required     pam_access.so

# Standard Un*x authorization.
@include common-account

session    sufficient   pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=3
# SELinux needs to be the first session rule.  This ensures that any
# lingering context has been cleared.  Without this it is possible that a
# module could execute code in the wrong domain.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so close

# Set the loginuid process attribute.
session    required     pam_loginuid.so

# Create a new session keyring.
session    optional     pam_keyinit.so force revoke

# Standard Un*x session setup and teardown.
@include common-session

# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate

# Print the status of the user's mailbox upon successful login.
session    optional     pam_mail.so standard noenv # [1]

# Set up user limits from /etc/security/limits.conf.
session    required     pam_limits.so

# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
session    required     pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
session    required     pam_env.so user_readenv=1 envfile=/etc/default/locale

# SELinux needs to intervene at login time to ensure that the process starts
# in the proper default security context.  Only sessions which are intended
# to run in the user's context should be run after this.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so open

# Standard Un*x password updating.
@include common-password
pi@raspberrypi:~ $    
pi@raspberrypi:~ $ cat /etc/pam_radius_auth.conf 
#  /etc/pam_radius_auth.conf configuration file.
#
#  For proper security, this file SHOULD have permissions 0600,
#  that is readable by root, and NO ONE else.  If anyone other than
#  root can read this file, then they can spoof responses from the server!
#
#  There are 5 fields per line in this file.  There may be multiple
#  lines.  Blank lines or lines beginning with '#' are treated as
#  comments, and are ignored.  The fields are:
#
#  server[:port] secret [timeout [source_ip [vrf]]]
#
#  the port name or number is optional.  The default port name is
#  "radius", and is looked up from /etc/services The timeout field is
#  optional.  The default timeout is 3 seconds.
#  The source_ip field is optional and the default is none.
#  The vrf field is optional and the default is none.
#
#  For IPv6 literal addresses, the address has to be surrounded  by
#  square  brackets as usual. E.g. [2001:0db8:85a3::4].
#
#  If multiple RADIUS server lines exist, they are tried in order.  The
#  first server to return success or failure causes the module to return
#  success or failure.  Only if a server fails to response is it skipped,
#  and the next server in turn is used.
#
#  The timeout field controls how many seconds the module waits before
#  deciding that the server has failed to respond.  Timeouts MUST be
#  between 3 and 60 seconds.  If they are outside of this range, the
#  timeouts are clamped to this range.
#
#  The source_ip field can be used to make the library bind the socket
#  that connects to that particular server to a particular IP address.
#  Note: specifying a timeout field is mandatory due to config parsing,
#  but if not needed it can be just set to the default of 3.
#
#  The vrf field can be used on Linux to make the library bind the socket
#  that connects to that particualar server to a particular VRF.
#  See: https://www.kernel.org/doc/Documentation/networking/vrf.txt for
#  more information.
#  Note: specifying a source_ip field is mandatory due to config parsing,
#  but if not needed it can be just set to 0.
#
# server[:port]             shared_secret      timeout (s)  source_ip            vrf
10.79.54.196                testing123         3            0
#127.0.0.1                   secret             3
#other-server                other-secret       5            192.168.1.10         vrf-blue
#[2001:0db8:85a3::4]:1812    other6-secret      3            [2001:0db8:85a3::3]  vrf-red
#other-other-server          other-other-secret 5            0                    vrf-blue
#
# having localhost in your radius configuration is a Good Thing.
#
# See the INSTALL file for pam.conf hints.
pi@raspberrypi:~ $ 
pi@raspberrypi:~ $ cat /etc/ssh/sshd_config
#   $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem   sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   X11Forwarding no
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server
pi@raspberrypi:~ $ 
pi@raspberrypi:~ $

From the above FreeRadius Server and Linux Server's configuration, we can found that User1, User2 and User3 are added at FreeRadius's user list. when I tried to login to Linux Server using user: pi via ssh, login success, wireshark capture and freeradius shows that radius authenticate the user success. below is freeradius log:

(0) Received Access-Request Id 234 from 173.39.202.158:55987 to 172.17.0.3:1812 length 86
(0)   User-Name = "pi"
(0)   User-Password = "test123"
(0)   NAS-IP-Address = 127.0.1.1
(0)   NAS-Identifier = "sshd"
(0)   NAS-Port = 10221
(0)   NAS-Port-Type = Virtual
(0)   Service-Type = Authenticate-Only
(0)   Calling-Station-Id = "10.140.42.56"
(0) # Executing section authorize from file /etc/freeradius/sites-enabled/default
(0)   authorize {
(0)     policy filter_username {
(0)       if (&User-Name) {
(0)       if (&User-Name)  -> TRUE
(0)       if (&User-Name)  {
(0)         if (&User-Name =~ / /) {
(0)         if (&User-Name =~ / /)  -> FALSE
(0)         if (&User-Name =~ /@[^@]*@/ ) {
(0)         if (&User-Name =~ /@[^@]*@/ )  -> FALSE
(0)         if (&User-Name =~ /\.\./ ) {
(0)         if (&User-Name =~ /\.\./ )  -> FALSE
(0)         if ((&User-Name =~ /@/) && (&User-Name !~ /@(.+)\.(.+)$/))  {
(0)         if ((&User-Name =~ /@/) && (&User-Name !~ /@(.+)\.(.+)$/))   -> FALSE
(0)         if (&User-Name =~ /\.$/)  {
(0)         if (&User-Name =~ /\.$/)   -> FALSE
(0)         if (&User-Name =~ /@\./)  {
(0)         if (&User-Name =~ /@\./)   -> FALSE
(0)       } # if (&User-Name)  = notfound
(0)     } # policy filter_username = notfound
(0)     [preprocess] = ok
(0)     [chap] = noop
(0)     [mschap] = noop
(0)     [digest] = noop
(0) suffix: Checking for suffix after "@"
(0) suffix: No '@' in User-Name = "pi", looking up realm NULL
(0) suffix: No such realm "NULL"
(0)     [suffix] = noop
(0) eap: No EAP-Message, not doing EAP
(0)     [eap] = noop
(0) files: users: Matched entry pi at line 93
(0) files: EXPAND Hello, %{User-Name}
(0) files:    --> Hello, pi
(0)     [files] = ok
(0)     [expiration] = noop
(0)     [logintime] = noop
(0)     [pap] = updated
(0)   } # authorize = updated
(0) Found Auth-Type = PAP
(0) # Executing group from file /etc/freeradius/sites-enabled/default
(0)   Auth-Type PAP {
(0) pap: Login attempt with password
(0) pap: Comparing with "known good" Cleartext-Password
(0) pap: User authenticated successfully
(0)     [pap] = ok
(0)   } # Auth-Type PAP = ok
(0) # Executing section post-auth from file /etc/freeradius/sites-enabled/default
(0)   post-auth {
(0)     if (session-state:User-Name && reply:User-Name && request:User-Name && (reply:User-Name == request:User-Name)) {
(0)     if (session-state:User-Name && reply:User-Name && request:User-Name && (reply:User-Name == request:User-Name))  -> FALSE
(0)     update {
(0)       No attributes updated for RHS &session-state:
(0)     } # update = noop
(0)     [exec] = noop
(0)     policy remove_reply_message_if_eap {
(0)       if (&reply:EAP-Message && &reply:Reply-Message) {
(0)       if (&reply:EAP-Message && &reply:Reply-Message)  -> FALSE
(0)       else {
(0)         [noop] = noop
(0)       } # else = noop
(0)     } # policy remove_reply_message_if_eap = noop
(0)     if (EAP-Key-Name && &reply:EAP-Session-Id) {
(0)     if (EAP-Key-Name && &reply:EAP-Session-Id)  -> FALSE
(0)   } # post-auth = noop
(0) Sent Access-Accept Id 234 from 172.17.0.3:1812 to 173.39.202.158:55987 length 31
(0)   Reply-Message = "Hello, pi"
(0) Finished request
Waking up in 4.9 seconds.
(1) Received Accounting-Request Id 56 from 173.39.202.158:47714 to 172.17.0.3:1813 length 84
(1)   User-Name = "pi"
(1)   NAS-IP-Address = 127.0.1.1
(1)   NAS-Identifier = "sshd"
(1)   NAS-Port = 10221
(1)   NAS-Port-Type = Virtual
(1)   Acct-Status-Type = Start
(1)   Acct-Session-Id = "00010221"
(1)   Acct-Authentic = RADIUS
(1)   Calling-Station-Id = "10.140.42.56"
(1) # Executing section preacct from file /etc/freeradius/sites-enabled/default
(1)   preacct {
(1)     [preprocess] = ok
(1)     policy acct_unique {
(1)       update request {
(1)         &Tmp-String-9 := "ai:"
(1)       } # update request = noop
(1)       if (("%{hex:&Class}" =~ /^%{hex:&Tmp-String-9}/) &&       ("%{string:&Class}" =~ /^ai:([0-9a-f]{32})/i)) {
(1)       EXPAND %{hex:&Class}
(1)          --> 
(1)       EXPAND ^%{hex:&Tmp-String-9}
(1)          --> ^61693a
(1)       if (("%{hex:&Class}" =~ /^%{hex:&Tmp-String-9}/) &&       ("%{string:&Class}" =~ /^ai:([0-9a-f]{32})/i))  -> FALSE
(1)       else {
(1)         update request {
(1)           EXPAND %{md5:%{User-Name},%{Acct-Session-ID},%{%{NAS-IPv6-Address}:-%{NAS-IP-Address}},%{NAS-Identifier},%{NAS-Port-ID},%{NAS-Port}}
(1)              --> 070cd5ea05f403fa46c6d036709b9288
(1)           &Acct-Unique-Session-Id := 070cd5ea05f403fa46c6d036709b9288
(1)         } # update request = noop
(1)       } # else = noop
(1)       update request {
(1)         &Tmp-String-9 !* ANY
(1)       } # update request = noop
(1)     } # policy acct_unique = noop
(1) suffix: Checking for suffix after "@"
(1) suffix: No '@' in User-Name = "pi", looking up realm NULL
(1) suffix: No such realm "NULL"
(1)     [suffix] = noop
(1)     [files] = noop
(1)   } # preacct = ok
(1) # Executing section accounting from file /etc/freeradius/sites-enabled/default
(1)   accounting {
(1) detail: EXPAND /var/log/freeradius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/detail-%Y%m%d
(1) detail:    --> /var/log/freeradius/radacct/173.39.202.158/detail-20231116
(1) detail: /var/log/freeradius/radacct/%{%{Packet-Src-IP-Address}:-%{Packet-Src-IPv6-Address}}/detail-%Y%m%d expands to /var/log/freeradius/radacct/173.39.202.158/detail-20231116
(1) detail: EXPAND %t
(1) detail:    --> Thu Nov 16 05:07:55 2023
(1)     [detail] = ok
(1)     [unix] = ok
(1)     [exec] = noop
(1) attr_filter.accounting_response: EXPAND %{User-Name}
(1) attr_filter.accounting_response:    --> pi
(1) attr_filter.accounting_response: Matched entry DEFAULT at line 12
(1)     [attr_filter.accounting_response] = updated
(1)   } # accounting = updated
(1) Sent Accounting-Response Id 56 from 172.17.0.3:1813 to 173.39.202.158:47714 length 20
(1) Finished request
(1) Cleaning up request packet ID 56 with timestamp +19 due to done
Waking up in 4.9 seconds.
(0) Cleaning up request packet ID 234 with timestamp +19 due to cleanup_delay was reached
Ready to process requests

when I tried to login to Linux Server using user: bob or mario via ssh, login failure, wireshark capture and freeradius shows that radius authenticate the user failure. below is freeradius log:

(2) Received Access-Request Id 214 from 173.39.202.158:44109 to 172.17.0.3:1812 length 87
(2)   User-Name = "bob"
(2)   User-Password = "\010\n\r\177INC"
(2)   NAS-IP-Address = 127.0.1.1
(2)   NAS-Identifier = "sshd"
(2)   NAS-Port = 10248
(2)   NAS-Port-Type = Virtual
(2)   Service-Type = Authenticate-Only
(2)   Calling-Station-Id = "10.140.42.56"
(2) # Executing section authorize from file /etc/freeradius/sites-enabled/default
(2)   authorize {
(2)     policy filter_username {
(2)       if (&User-Name) {
(2)       if (&User-Name)  -> TRUE
(2)       if (&User-Name)  {
(2)         if (&User-Name =~ / /) {
(2)         if (&User-Name =~ / /)  -> FALSE
(2)         if (&User-Name =~ /@[^@]*@/ ) {
(2)         if (&User-Name =~ /@[^@]*@/ )  -> FALSE
(2)         if (&User-Name =~ /\.\./ ) {
(2)         if (&User-Name =~ /\.\./ )  -> FALSE
(2)         if ((&User-Name =~ /@/) && (&User-Name !~ /@(.+)\.(.+)$/))  {
(2)         if ((&User-Name =~ /@/) && (&User-Name !~ /@(.+)\.(.+)$/))   -> FALSE
(2)         if (&User-Name =~ /\.$/)  {
(2)         if (&User-Name =~ /\.$/)   -> FALSE
(2)         if (&User-Name =~ /@\./)  {
(2)         if (&User-Name =~ /@\./)   -> FALSE
(2)       } # if (&User-Name)  = notfound
(2)     } # policy filter_username = notfound
(2)     [preprocess] = ok
(2)     [chap] = noop
(2)     [mschap] = noop
(2)     [digest] = noop
(2) suffix: Checking for suffix after "@"
(2) suffix: No '@' in User-Name = "bob", looking up realm NULL
(2) suffix: No such realm "NULL"
(2)     [suffix] = noop
(2) eap: No EAP-Message, not doing EAP
(2)     [eap] = noop
(2) files: users: Matched entry bob at line 87
(2) files: EXPAND Hello, %{User-Name}
(2) files:    --> Hello, bob
(2)     [files] = ok
(2)     [expiration] = noop
(2)     [logintime] = noop
(2)     [pap] = updated
(2)   } # authorize = updated
(2) Found Auth-Type = PAP
(2) # Executing group from file /etc/freeradius/sites-enabled/default
(2)   Auth-Type PAP {
(2) pap: Login attempt with password
(2) pap: Comparing with "known good" Cleartext-Password
(2) pap: ERROR: Cleartext password does not match "known good" password
(2) pap: Passwords don't match
(2)     [pap] = reject
(2)   } # Auth-Type PAP = reject
(2) Failed to authenticate the user
(2) WARNING: Unprintable characters in the password.  Double-check the shared secret on the server and the NAS!
(2) Using Post-Auth-Type Reject
(2) # Executing group from file /etc/freeradius/sites-enabled/default
(2)   Post-Auth-Type REJECT {
(2) attr_filter.access_reject: EXPAND %{User-Name}
(2) attr_filter.access_reject:    --> bob
(2) attr_filter.access_reject: Matched entry DEFAULT at line 11
(2)     [attr_filter.access_reject] = updated
(2)     [eap] = noop
(2)     policy remove_reply_message_if_eap {
(2)       if (&reply:EAP-Message && &reply:Reply-Message) {
(2)       if (&reply:EAP-Message && &reply:Reply-Message)  -> FALSE
(2)       else {
(2)         [noop] = noop
(2)       } # else = noop
(2)     } # policy remove_reply_message_if_eap = noop
(2)   } # Post-Auth-Type REJECT = updated
(2) Delaying response for 1.000000 seconds
Waking up in 0.3 seconds.
Waking up in 0.6 seconds.
(2) Sending delayed response
(2) Sent Access-Reject Id 214 from 172.17.0.3:1812 to 173.39.202.158:44109 length 32
(2)   Reply-Message = "Hello, bob"
Waking up in 3.9 seconds.
(2) Cleaning up request packet ID 214 with timestamp +277 due to cleanup_delay was reached
Ready to process requests

Notice the line: (2) User-Password = "\010\n\r\177INC" from the above log, we can found the Access-Request message carries wrong password. Wireshark capture also proves it.

the only one different point between user pi and user bob/mario is that pi was created at Linux server but bob/mario was not. i did google search and found two different ideas:

one is here, they think a PAM module which is run before pam_radius_auth. That first module is checking the password locally, and when it's wrong, is setting the password to the "INCORRECT" string.
the other is here, they think pam_radius_auth library can’t correctly encrypt the password when there is no user defined in system (/etc/passwd).

If my issue does really result from the 2nd idea, it will be a critical issue for us. because our intention is to simplify user authentication at our network as we have many users and many linux servers, we don't want to create users one by one at every Linux Servers as which is huge workload for us. Consequently, I build a FreeRadius server to do authentication, authorization and accounting for every users. our target is that when one new user join us, the only one thing I need to do is adding him to the user list at FreeRadius Server, then he can login to Linux Servers, FreeRadius server will help authenticate him.

I am not sure: users exist at Linux system(/etc/passwd) is a mandatory prerequisite for central pam-radius-based ssh ?

Any expert could help me ?

Thanks in advance.

FreeRADIUS / pam_radius

pap: ERROR: Cleartext password does not match "known good" password #82