Closed richard-allen closed 6 years ago
cooperlees
commented
7 years ago I believe the succeed on fail is a client decision (I could be wrong). I would double check to see with the child segfault if any message gets sent back to your router.
I don't think you're doing anything wrong and just some love between PAM + this old codebase probably needs some debugging like you've found. Maybe asking in some PAM github repo, forum etc. might be the next best step, as, I too, am no PAM guru.
richard-allen
commented
7 years ago Looking at the gdb output, I just had a terrifying thought, which I promptly tested and yes it's true. It matters not what I type into the first factor prompt, it succeeds. ALso, it matters not what I type into the second factor prompt. Still succeeds :) I have to mess up the username field before logins fail
cooperlees
commented
7 years ago Well this is either a bug in PAM / Tacacs interaction with PAM or your PAM config from what I see. I've changed teams at Facebook and don't have time to debug this, nor do I have the ability to create a repro of your setup (We don't use dual factor on TACACS+ devices, we do elsewhere to get into production). So if you can fix this and show how you've tested it I am sure we will happily accept a PR.
richard-allen
commented
7 years ago Ok... Do you think anyone at facebook would be interested in diagnosing this with me? I could provide a clone of this VM complete with all the setup in no time if needed and give access to the system to work this out and a test router to trigger the bug. PS: can you see anything abvious errors in the PAM config? It's just basicly calling system-auth for most things and that should be pretty well tested.
cooperlees
commented
7 years ago We won't have time to help here sir. If you work it out, PRs will be welcome.
cooperlees
commented
6 years ago Can't help on custom LDAP + PAM issues we can not reproduce in our environment.
Im trying to set up tac_plus+ to use two factor authentication. My setup is as follows:
In ldap I have two users. One configured for two factor and one not.
[root@ipa ~]# id rikkatest uid=1130400006(rikkatest) gid=1130400006(rikkatest) groups=1130400004(cisco-enable),1130400006(rikkatest) [root@ipa ~]# id netvik uid=1130400009(netvik) gid=1130400009(netvik) groups=1130400004(cisco-enable),1130400008(service_accounts),1130400009(netvik)
[root@ipa ~]# ssh rikkatest@localhost First Factor: Second Factor: Last login: Tue Jul 25 16:44:20 2017 from localhost -sh-4.2$
[root@ipa netvik]# ssh netvik@localhost Password: Last login: Tue Jul 25 17:18:43 2017 -sh-4.2$
[root@ipa ~]# cat /etc/pam.d/tac_plus
%PAM-1.0
auth include system-auth account required pam_nologin.so account include system-auth password include system-auth session optional pam_keyinit.so force revoke session include system-auth session required pam_loginuid.so
[root@ipa ~]# cat /etc/tac_plus.conf accounting file = /var/log/tacacs.log
key = testing123
host = 94.142.159.65 { key = testing123 } host = pat.ok.is { key = testing123 }
group = enable { login = PAM }
user = netvik { member = enable }
user = rikkatest { member = enable }
(Yes, that's a lousy key, but only here for testing)
Running daemon as such: [root@ipa ~]# tac_plus -C /etc/tac_plus.conf -L -p 49 -d1016 -g
Then I have a Cisco router configured to authenticate against this tac_plus server and it "works". First the plain user with no two factor auth:
[ra@hamburger ~]$ telnet 10.199.6.87 Trying 10.199.6.87... Connected to 10.199.6.87. Escape character is '^]'.
User Access Verification
Username: netvik Password:
Router>
so far so good. Daemon stdout has:
Reading config Version F4.0.4.19-fb Initialized 1 tac_plus server F4.0.4.19-fb starting socket FD 4 AF 2 socket FD 5 AF 10 uid=0 euid=0 gid=0 egid=0 s=26448032 session request from pat.ok.is sock=6 connect from pat.ok.is [94.142.159.65] Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/START size=38 validation request from pat.ok.is PACKET: key=testing123 version 192 (0xc0), type 1, seq no 1, flags 0x1 session_id 3891870596 (0xe7f93b84), Data length 26 (0x1a) End header Packet body hex dump: 0x1 0x1 0x1 0x1 0x0 0x6 0xc 0x0 0x74 0x74 0x79 0x35 0x37 0x38 0x31 0x30 0x2e 0x31 0x39 0x39 0x2e 0x36 0x2e 0x31 0x31 0x30 type=AUTHEN/START, priv_lvl = 1 action=login authen_type=ascii service=login user_len=0 port_len=6 (0x6), rem_addr_len=12 (0xc) data_len=0 User: port: tty578 rem_addr: 10.199.6.110 data: End packet Authen Start request choose_authen returns 1 cfg_get_hvalue: name=94.142.159.65 attr=prompt cfg_get_phvalue: returns NULL cfg_get_hvalue: name=pat.ok.is attr=prompt cfg_get_phvalue: returns NULL Writing AUTHEN/GETUSER size=55 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 2, flags 0x1 session_id 3891870596 (0xe7f93b84), Data length 43 (0x2b) End header Packet body hex dump: 0x4 0x0 0x25 0x0 0x0 0x0 0xa 0x55 0x73 0x65 0x72 0x20 0x41 0x63 0x63 0x65 0x73 0x73 0x20 0x56 0x65 0x72 0x69 0x66 0x69 0x63 0x61 0x74 0x69 0x6f 0x6e 0xa 0xa 0x55 0x73 0x65 0x72 0x6e 0x61 0x6d 0x65 0x3a 0x20 type=AUTHEN status=4 (AUTHEN/GETUSER) flags=0x0 msg_len=37, data_len=0 msg: 0xa User Access Verification 0xa data: End packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/CONT size=23 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 3, flags 0x1 session_id 3891870596 (0xe7f93b84), Data length 11 (0xb) End header Packet body hex dump: 0x6 0x0 0x0 0x0 0x0 0x6e 0x65 0x74 0x76 0x69 0x6b type=AUTHEN/CONT user_msg_len 6 (0x6), user_data_len 0 (0x0) flags=0x0 User msg: netvik User data: End packet cfg_get_value: name=netvik isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM choose_authen chose default_fn Calling authentication function cfg_get_value: name=netvik isuser=1 attr=nopassword rec=1 cfg_get_value: recurse group = enable cfg_get_intvalue: returns 0 cfg_get_value: name=netvik isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM cfg_get_value: name=netvik isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM pam_verify netvik pam_tacacs received 1 pam_messages pat.ok.is tty578: PAM_PROMPT_ECHO_OFF Writing AUTHEN/GETPASS size=28 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 4, flags 0x1 session_id 3891870596 (0xe7f93b84), Data length 16 (0x10) End header Packet body hex dump: 0x5 0x1 0xa 0x0 0x0 0x0 0x50 0x61 0x73 0x73 0x77 0x6f 0x72 0x64 0x3a 0x20 type=AUTHEN status=5 (AUTHEN/GETPASS) flags=0x1 msg_len=10, data_len=0 msg: Password: data: End packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/CONT size=33 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 5, flags 0x1 session_id 3891870596 (0xe7f93b84), Data length 21 (0x15) End header Packet body hex dump: 0x10 0x0 0x0 0x0 0x0 0x36 0x28 0x5b 0x35 0x31 0x76 0x22 0x4c 0x42 0x52 0x66 0x43 0x71 0x6d 0x7b 0x38 type=AUTHEN/CONT user_msg_len 16 (0x10), user_data_len 0 (0x0) flags=0x0 User msg: 6([51v"LBRfCqm{8 User data: End packet pam_acct_mgmt returns PAM_SUCCESS pam_verify returns 1 cfg_get_value: name=netvik isuser=1 attr=expires rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns NULL Password has not expired
cfg_get_value: name=netvik isuser=1 attr=acl rec=1
cfg_get_value: recurse group = enable
cfg_get_pvalue: returns NULL
login query for 'netvik' tty578 from pat.ok.is accepted
login success: user=netvik device=pat.ok.is port=94.142.159.65 client=tty578
Writing AUTHEN/SUCCEED size=18
PACKET: key=testing123
version 192 (0xc0), type 1, seq no 6, flags 0x1
session_id 3891870596 (0xe7f93b84), Data length 6 (0x6)
End header
Packet body hex dump:
0x1 0x0 0x0 0x0 0x0 0x0
type=AUTHEN status=1 (AUTHEN/SUCCEED) flags=0x0
msg_len=0, data_len=0
msg:
data:
End packet
cfg_get_hvalue: name=94.142.159.65 attr=key
cfg_get_phvalue: returns testing123
pat.ok.is: disconnect
Then I try the two factor user:
[ra@hamburger ~]$ telnet 10.199.6.87 Trying 10.199.6.87... Connected to 10.199.6.87. Escape character is '^]'.
User Access Verification
Username: rikkatest First Factor:
Second Factor:
Router>
This seems to have worked. However the daemons stdout has:
Reading config Version F4.0.4.19-fb Initialized 1 tac_plus server F4.0.4.19-fb starting socket FD 4 AF 2 socket FD 5 AF 10 uid=0 euid=0 gid=0 egid=0 s=23187616 session request from pat.ok.is sock=6 connect from pat.ok.is [94.142.159.65] Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/START size=38 validation request from pat.ok.is PACKET: key=testing123 version 192 (0xc0), type 1, seq no 1, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 26 (0x1a) End header Packet body hex dump: 0x1 0x1 0x1 0x1 0x0 0x6 0xc 0x0 0x74 0x74 0x79 0x35 0x37 0x38 0x31 0x30 0x2e 0x31 0x39 0x39 0x2e 0x36 0x2e 0x31 0x31 0x30 type=AUTHEN/START, priv_lvl = 1 action=login authen_type=ascii service=login user_len=0 port_len=6 (0x6), rem_addr_len=12 (0xc) data_len=0 User: port: tty578 rem_addr: 10.199.6.110 data: End packet Authen Start request choose_authen returns 1 cfg_get_hvalue: name=94.142.159.65 attr=prompt cfg_get_phvalue: returns NULL cfg_get_hvalue: name=pat.ok.is attr=prompt cfg_get_phvalue: returns NULL Writing AUTHEN/GETUSER size=55 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 2, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 43 (0x2b) End header Packet body hex dump: 0x4 0x0 0x25 0x0 0x0 0x0 0xa 0x55 0x73 0x65 0x72 0x20 0x41 0x63 0x63 0x65 0x73 0x73 0x20 0x56 0x65 0x72 0x69 0x66 0x69 0x63 0x61 0x74 0x69 0x6f 0x6e 0xa 0xa 0x55 0x73 0x65 0x72 0x6e 0x61 0x6d 0x65 0x3a 0x20 type=AUTHEN status=4 (AUTHEN/GETUSER) flags=0x0 msg_len=37, data_len=0 msg: 0xa User Access Verification 0xa data: End packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/CONT size=26 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 3, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 14 (0xe) End header Packet body hex dump: 0x9 0x0 0x0 0x0 0x0 0x72 0x69 0x6b 0x6b 0x61 0x74 0x65 0x73 0x74 type=AUTHEN/CONT user_msg_len 9 (0x9), user_data_len 0 (0x0) flags=0x0 User msg: rikkatest User data: End packet cfg_get_value: name=rikkatest isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM choose_authen chose default_fn Calling authentication function cfg_get_value: name=rikkatest isuser=1 attr=nopassword rec=1 cfg_get_value: recurse group = enable cfg_get_intvalue: returns 0 cfg_get_value: name=rikkatest isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM cfg_get_value: name=rikkatest isuser=1 attr=login rec=1 cfg_get_value: recurse group = enable cfg_get_pvalue: returns PAM pam_verify rikkatest pam_tacacs received 2 pam_messages pat.ok.is tty578: PAM_PROMPT_ECHO_OFF Writing AUTHEN/GETPASS size=32 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 4, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 20 (0x14) End header Packet body hex dump: 0x5 0x1 0xe 0x0 0x0 0x0 0x46 0x69 0x72 0x73 0x74 0x20 0x46 0x61 0x63 0x74 0x6f 0x72 0x3a 0x20 type=AUTHEN status=5 (AUTHEN/GETPASS) flags=0x1 msg_len=14, data_len=0 msg: First Factor: data: End packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/CONT size=29 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 5, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 17 (0x11) End header Packet body hex dump: 0xc 0x0 0x0 0x0 0x0 0x72 0x69 0x6b 0x6b 0x61 0x74 0x65 0x73 0x74 0x31 0x32 0x33 type=AUTHEN/CONT user_msg_len 12 (0xc), user_data_len 0 (0x0) flags=0x0 User msg: rikkatest123 User data: End packet pat.ok.is tty578: PAM_PROMPT_ECHO_OFF Writing AUTHEN/GETPASS size=33 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 6, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 21 (0x15) End header Packet body hex dump: 0x5 0x1 0xf 0x0 0x0 0x0 0x53 0x65 0x63 0x6f 0x6e 0x64 0x20 0x46 0x61 0x63 0x74 0x6f 0x72 0x3a 0x20 type=AUTHEN status=5 (AUTHEN/GETPASS) flags=0x1 msg_len=15, data_len=0 msg: Second Factor: data: End packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Waiting for packet cfg_get_hvalue: name=94.142.159.65 attr=key cfg_get_phvalue: returns testing123 Read AUTHEN/CONT size=23 PACKET: key=testing123 version 192 (0xc0), type 1, seq no 7, flags 0x1 session_id 3576905187 (0xd5333de3), Data length 11 (0xb) End header Packet body hex dump: 0x6 0x0 0x0 0x0 0x0 0x31 0x38 0x38 0x38 0x33 0x36 type=AUTHEN/CONT user_msg_len 6 (0x6), user_data_len 0 (0x0) flags=0x0 User msg: 188836 User data: End packet Segmentation fault (core dumped)
Looking into the core:
[root@ipa ~]# gdb /usr/bin/tac_plus core.31038 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/bin/tac_plus...Reading symbols from /usr/lib/debug/usr/bin/tac_plus.debug...done. done. [New LWP 31038] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `tac_plus -C /etc/tac_plus.conf -L -p 49 -d1016 -g'. Program terminated with signal 11, Segmentation fault.
0 0x00000000004122b5 in pam_tacacs (nmsg=2, pmpp=0x7ffe5c4bba70, prpp=0x7ffe5c4bba60, appdata_ptr=0x0) at pwlib.c:568
568 prpp[i]->resp = (char *) tac_malloc(acp->user_msg_len + 1); (gdb) where
0 0x00000000004122b5 in pam_tacacs (nmsg=2, pmpp=0x7ffe5c4bba70, prpp=0x7ffe5c4bba60, appdata_ptr=0x0) at pwlib.c:568
1 0x00007f4816f23b88 in prompt_2fa (pamh=pamh@entry=0x161d510, pi=pi@entry=0x7ffe5c4bbb50, prompt_fa1=0x7f4816f279fe "First Factor: ",
2 0x00007f4816f25426 in get_authtok_for_authentication (flags=, pi=0x7ffe5c4bbb50, pamh=0x161d510) at src/sss_client/pam_sss.c:1584
3 pam_sss (task=, pamh=0x161d510, pam_flags=, argc=, argv=) at src/sss_client/pam_sss.c:1826
4 0x00007f481903bf1a in _pam_dispatch_aux (use_cached_chain=, resumed=, h=, flags=0, pamh=0x161d510) at pam_dispatch.c:110
5 _pam_dispatch (pamh=pamh@entry=0x161d510, flags=0, choice=choice@entry=1) at pam_dispatch.c:426
6 0x00007f481903b7e0 in pam_authenticate (pamh=0x161d510, flags=) at pam_auth.c:34
7 0x0000000000412774 in pam_verify (user=0x161d0f0 "rikkatest", passwd=0x161d400 "", data=0x7ffe5c4bbf80) at pwlib.c:672
8 0x00000000004114da in verify (name=0x161d0f0 "rikkatest", passwd=0x161d400 "", data=0x7ffe5c4bbf80, recurse=1) at pwlib.c:158
9 0x0000000000409130 in tac_login (data=0x7ffe5c4bbf80, p=0x161d400) at default_fn.c:315
10 0x0000000000408c5b in default_fn (data=0x7ffe5c4bbf80) at default_fn.c:170
11 0x000000000040371a in authenticate (datap=0x7ffe5c4bbf80, typep=0x7ffe5c4bbef0) at authen.c:319
12 0x00000000004031ed in do_start (pak=0x161d0c0 "\300\001\001\001\325\063", <incomplete sequence \343>) at authen.c:139
13 0x0000000000402f53 in authen (pak=0x161d0c0 "\300\001\001\001\325\063", <incomplete sequence \343>) at authen.c:53
14 0x00000000004168eb in start_session () at tac_plus.c:853
15 0x0000000000416760 in main (argc=8, argv=0x7ffe5c4bc678) at tac_plus.c:764
apparently pwlib's, line 568: prpp[i]->resp = (char *) tac_malloc(acp->user_msg_len + 1); Is not making the system happy. Some checking:
(gdb) print prpp $1 = (struct pam_response *) 0x7ffe5c4bba60 (gdb) print i $2 = 1 (gdb) print nmsg $3 = 2 (gdb) print prpp[0] $4 = (struct pam_response ) 0x161da10 (gdb) print prpp[1] $5 = (struct pam_response ) 0x5 (gdb) print prpp[0]->resp $6 = 0x162e320 "rikkatest123" (gdb) print prpp[1]->resp Cannot access memory at address 0x5 (gdb) print appdata_ptr $7 = (void ) 0x0
(yes, thats a lousy password and it's just for testing)
Im not into PAM on this level but to me it looks like pam library is calling pam_tacacs() in the daemon with either nmsg off by one or something missing from prpp
Any ideas what is going on? Have I stumbled across a bug or am I doing something drastically wrong? And should the login succeed when the daemon crashes like that?