pbis-open 9.1.0.551 - Authentication failures for AD users

Version: 9.1.0.551 OS/Distro: Ubuntu server 18.04 64-bit - server name is "selene" Issue/Impact: We've been running pbis-open on this server since September 2019. It's recently started refusing authentication for our AD users (local users in /etc/passwd are fine). Currently getting the following messages in /var/log/auth.log:

Jan 7 08:49:26 selene sshd[2199]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.194.64.108 user=scomdjp Jan 7 08:49:28 selene sshd[2197]: error: PAM: Authentication failure for scomdjp from 10.194.64.108 Jan 7 08:49:29 selene sshd[2197]: Connection closed by 10.194.64.108 port 36756 [preauth]

The following is output from relevant commands:

1. systemctl status lwsmd.service

lwsmd.service - BeyondTrust AD Bridge Service Manager Loaded: loaded (/lib/systemd/system/lwsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-01-07 08:40:13 GMT; 2h 35min ago Process: 1157 ExecStart=/opt/pbis/sbin/lwsmd --start-as-daemon (code=exited, status=0/SUCCESS) Main PID: 1209 (lwsmd) CGroup: /system.slice/lwsmd.service ├─1209 /opt/pbis/sbin/lwsmd --start-as-daemon ├─1255 lw-container lwreg ├─1313 lw-container eventlog ├─1363 lw-container netlogon ├─1414 lw-container lwio ├─1531 lw-container reapsysl └─2616 lw-container lsass

Jan 07 09:15:51 selene lsass[1465]: [lwreg] LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] Jan 07 09:15:51 selene lsass[1465]: [lsass-ipc] lwmsg_peer_task_handle_assoc_error():lwmsg/src/peer-task.c:625: (session:99cebd2ae21d270d-4b1d26820c720533) Dropping: LWMSG_STATUS_PEER_CLOSE Jan 07 09:15:56 selene lsass[1465]: [lsass] LsaSrvIpcCheckPermissions():lsass/server/api/ipc_state.c:79: Permission granted for (uid = 0, gid = 0, pid = 2418) to open LsaIpcServer Jan 07 09:15:56 selene lsass[1465]: [lsass-ipc] lwmsg_peer_log_accept():lwmsg/src/peer-log.c:230: (session:b23e54e1e9d41ca5-c3fc0b0d637861e8) Accepted Jan 07 09:15:56 selene lsass[1465]: LwKrb5SetThreadDefaultCachePath():lwadvapi/threaded/lwkrb5.c:479: Switched gss krb5 credentials path from to FILE:/var/lib/pbis/krb5cc_lsass.AD.HUD.AC.UK Jan 07 09:15:56 selene lsass[1465]: [lsass] MemCacheFindUserByName():lsass/server/auth-providers/ad-open-provider/memcache.c:1003: User cache entry for scomdjp not found Jan 07 09:15:56 selene lsass[1465]: [lsass] AD_CheckExpiredObject():lsass/server/auth-providers/ad-open-provider/online.c:1804: Using cache entry for sid S-1-5-21-1219361320-872739099-178173116-33806, updated 4505 seconds ago Jan 07 09:15:58 selene lsass[1465]: [lsass-ipc] lwmsg_peer_task_handle_assoc_error():lwmsg/src/peer-task.c:625: (session:b23e54e1e9d41ca5-c3fc0b0d637861e8) Dropping: LWMSG_STATUS_PEER_CLOSE

2. /opt/pbis/bin/lwsm list

lwreg running (container: 1255) dcerpc stopped eventlog running (container: 1313) lsass running (container: 2616) lwio running (container: 1414) netlogon running (container: 1363) rdr running (io: 1414) reapsysl running (container: 1531) usermonitor stopped

3. /opt/pbis/domainjoin-cli query

Name = selene Domain = AD.HUD.AC.UK Distinguished Name = CN=SELENE,OU=SCOM-SERVERS,OU=SCOM,OU=Staff,DC=AD,DC=HUD,DC=AC,DC=UK

4. pbis status

LSA Server Status:

Compiled daemon version: 9.1.0.551 Packaged product version: 9.1.551.2 Uptime: 0 days 1 hours 55 minutes 24 seconds

[Authentication provider: lsa-activedirectory-provider]

    Status:        Online
    Mode:          Un-provisioned
    Domain:        AD.xxx.xxx.xx
    Domain SID:    S-1-5-21-1219361320-872739099-178173116
    Forest:        AD.xxx.xxx.xx
    Site:          Valid name
    Online check interval:  300 seconds
    [Trusted Domains: 1]

    [Domain: AD]

            DNS Domain:       AD.xxx.xxx.xx
            Netbios name:     AD
            Forest name:      AD.xxx.xxx.xx
            Trustee DNS name:
            Client site name: Valid Name
            Domain SID:       S-1-5-21-1219361320-872739099-178173116
            Domain GUID:      2334fafe-65f9-4bc3-9054-5a0cd990ab82
            Trust Flags:      [0x001d]
                              [0x0001 - In forest]
                              [0x0004 - Tree root]
                              [0x0008 - Primary]
                              [0x0010 - Native]
            Trust type:       Up Level
            Trust Attributes: [0x0000]
            Trust Direction:  Primary Domain
            Trust Mode:       In my forest Trust (MFT)
            Domain flags:     [0x0001]
                              [0x0001 - Primary]

            [Domain Controller (DC) Information]

                    DC Name:              Bailey.AD.xxx.xxx.xx
                    DC Address:           10.64.240.253
                    DC Site:              Valid Name
                    DC Flags:             [0x0000f3fd]
                    DC Is PDC:            yes
                    DC is time server:    yes
                    DC has writeable DS:  yes
                    DC is Global Catalog: yes
                    DC is running KDC:    yes

            [Global Catalog (GC) Information]

                    GC Name:              Brazil.AD.xxx.xxx.xx
                    GC Address:           172.17.193.13
                    GC Site:              Valid Name
                    GC Flags:             [0x0000f1fc]
                    GC Is PDC:            no
                    GC is time server:    yes
                    GC has writeable DS:  yes
                    GC is running KDC:    yes

5. /opt/pbis/bin/enum-users

This prints an exhaustive list of users, correctly identifies Name / UID / GID / Gecos / Shell and Home Dir entries

**6. attach logs

/opt/pbis/bin/lwsm set-log-target -p lsass - file /tmp/lsass.log
/opt/pbis/bin/lwsm set-log-level -p lsass - debug
attach log**

20200107112223:ALWAYS: Logging started 20200107112239:ALWAYS:LwSmSetMaxLogLevel():lwsm/server/logger.c:662: Log level changed to DEBUG 20200107112242:VERBOSE:lsass:LsaUmpCheckUsers():lsass/server/auth-providers/ad-open-provider/lsaum_p.c:697: Lsa User Manager - checking user credentials refresh list 20200107112245:VERBOSE:lsass:LsaSrvIpcCheckPermissions():lsass/server/api/ipc_state.c:79: Permission granted for (uid = 0, gid = 0, pid = 4295) to open LsaIpcServer 20200107112245:VERBOSE:lsass-ipc:lwmsg_peer_log_accept():lwmsg/src/peer-log.c:230: (session:a1c52f2ef7e9ed05-b8b02558a1153703) Accepted 20200107112245:DEBUG:LwKrb5SetThreadDefaultCachePath():lwadvapi/threaded/lwkrb5.c:479: Switched gss krb5 credentials path from to FILE:/var/lib/pbis/krb5cc_lsass.AD.xxx.xxx.xx 20200107112245:DEBUG:lsass:MemCacheFindUserByName():lsass/server/auth-providers/ad-open-provider/memcache.c:1003: User cache entry for scomdjp not found 20200107112245:VERBOSE:lsass:AD_CheckExpiredObject():lsass/server/auth-providers/ad-open-provider/online.c:1804: Using cache entry for sid S-1-5-21-1219361320-872739099-178173116-33806, updated 167 seconds ago 20200107112245:VERBOSE:lsass:LsaSrvIpcCheckPermissions():lsass/server/api/ipc_state.c:79: Permission granted for (uid = 0, gid = 0, pid = 4295) to open LsaIpcServer 20200107112245:VERBOSE:lsass-ipc:lwmsg_peer_log_accept():lwmsg/src/peer-log.c:230: (session:f0b727aba072ec12-6b38ad27f0d97aa4) Accepted 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:RegTransactGetValueW():lwreg/client/clientipc.c:809: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:DEBUG:lwreg:LwNtRegGetValueA():lwreg/client/regntclient.c:793: Status: LW_STATUS_OBJECT_NAME_NOT_FOUND = 0xC0000034 (-1073741772)] 20200107112245:VERBOSE:lsass-ipc:lwmsg_peer_task_handle_assoc_error():lwmsg/src/peer-task.c:625: (session:f0b727aba072ec12-6b38ad27f0d97aa4) Dropping: LWMSG_STATUS_PEER_CLOSE 20200107112248:VERBOSE:lsass:LsaSrvIpcCheckPermissions():lsass/server/api/ipc_state.c:79: Permission granted for (uid = 0, gid = 0, pid = 4297) to open LsaIpcServer 20200107112248:VERBOSE:lsass-ipc:lwmsg_peer_log_accept():lwmsg/src/peer-log.c:230: (session:a4296161756cb446-57f18bdd98ce7325) Accepted 20200107112248:DEBUG:LwKrb5SetThreadDefaultCachePath():lwadvapi/threaded/lwkrb5.c:479: Switched gss krb5 credentials path from to FILE:/var/lib/pbis/krb5cc_lsass.AD.HUD.AC.UK 20200107112248:DEBUG:lsass:MemCacheFindUserByName():lsass/server/auth-providers/ad-open-provider/memcache.c:1003: User cache entry for scomdjp not found 20200107112248:VERBOSE:lsass:AD_CheckExpiredObject():lsass/server/auth-providers/ad-open-provider/online.c:1804: Using cache entry for sid S-1-5-21-1219361320-872739099-178173116-33806, updated 170 seconds ago 20200107112250:VERBOSE:lsass-ipc:lwmsg_peer_task_handle_assoc_error():lwmsg/src/peer-task.c:625: (session:a4296161756cb446-57f18bdd98ce7325) Dropping: LWMSG_STATUS_PEER_CLOSE 20200107112251:VERBOSE:lsass-ipc:lwmsg_peer_task_handle_assoc_error():lwmsg/src/peer-task.c:625: (session:a1c52f2ef7e9ed05-b8b02558a1153703) Dropping: LWMSG_STATUS_PEER_CLOSE

The command that returns the error is an attempt by a user to login to the server using an SSH login, such as ssh user>@<server. IF on a command line, the "Password:" prompt is returned to the user, so they see no "error" on their side, but the logs are showing the above.

I've looked at some articles suggesting refreshing the ad cache on the server, which hasn't stopped the problem, so am a bit stuck!

I'm reluctant to go back to local accounts, bypassing the AD authentication, because that was the whole point of moving to it, to reduce the number of accoutn logins users had to remember!

If any one has any thoughts, it'd be apprceciated. Thanks Dave

BeyondTrust / pbis-open

pbis-open 9.1.0.551 - Authentication failures for AD users #264