search

ubccr / mokey

FreeIPA self-service account management portal
BSD 3-Clause "New" or "Revised" License
194 stars 46 forks source link

Fedora 33 Error: failed sending AS_REQ to KDC: failed to communicate with KDC #80

Closed SomePersonSomeWhereInTheWorld closed 3 years ago

SomePersonSomeWhereInTheWorld commented 3 years ago

Any idea why this wouldn't be working? freeipa-server-4.9.3-1.fc33.x86_64

 mokey -c  /etc/mokey/mokey.yaml  --debug server
INFO[0000] Using template dir: /usr/share/mokey/templates
FATA[0000] [Root cause: Networking_Error] Networking_Error: AS Exchange Error: failed sending AS_REQ to KDC: failed to communicate with KDC. Attempts made with TCP (lookup _kerberos._tcp.OURDOMAIN.EDU on 127.0.0.1:53: no such host) and then UDP (lookup _kerberos._udp.OURDOMAIN.EDU on 127.0.0.1:53: no such host)
ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
nc -vvvzu 127.0.0.1 53
Ncat: Version 7.80 ( https://nmap.org/ncat )
NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:02001002:system library:fopen:No such file or directory
libnsock nsock_iod_new2(): nsock_iod_new (IOD #1)
libnsock nsock_connect_udp(): UDP connection requested to 127.0.0.1:53 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [127.0.0.1:53]
Ncat: Connected to 127.0.0.1:53.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #2)
libnsock nsock_write(): Write request for 1 bytes to IOD #1 EID 19 [127.0.0.1:53]
libnsock nsock_timer_create(): Timer created - 2000ms from now.  EID 28
libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 19 [127.0.0.1:53]
libnsock nsock_read(): Read request from IOD #1 [127.0.0.1:53] (timeout: -1ms) EID 34
libnsock nsock_trace_handler_callback(): Callback: TIMER SUCCESS for EID 28
Ncat: UDP packet sent successfully
Ncat: 1 bytes sent, 0 bytes received in 2.02 seconds.
libnsock nsock_trace_handler_callback(): Callback: READ KILL for EID 34 [127.0.0.1:53]
libnsock nsock_iod_delete(): nsock_iod_delete (IOD #1)
libnsock nsock_iod_delete(): nsock_iod_delete (IOD #2)
aebruno commented 3 years ago

Double check you can resolve those dns records. For example: 

$ dig -t SRV +short _kerberos._tcp.OURDOMAIN.EDU

Should return the hostnames of your freeipa servers.

SomePersonSomeWhereInTheWorld commented 3 years ago

dig -t SRV +short _kerberos._tcp.OURDOMAIN.EDU

Returns nothing as our IT department has disabled TCP for port 53. I've asked for an exclusion. UDP works though. Is TCP required?

When I use a browser on the server I do get the following page from http://localhost:8080/

Certificate System
Certificate System
-

The Certificate System is an enterprise-class open source Certificate Authority (CA). It is a full-featured system, and has been hardened by real-world deployments. It supports all aspects of certificate lifecycle management, including key archival, OCSP and smartcard management, and much more.

Enter
aebruno commented 3 years ago

Those dns records are required. Does this return anything? It should return the hostnames of your freeipa servers:

$ dig -t SRV +short _kerberos._udp.int.ccr.buffalo.edu
SomePersonSomeWhereInTheWorld commented 3 years ago 
dig -t SRV +short _kerberos._udp.int.ccr.buffalo.edu

No does not return anything.

I know FreeIPA and Kerberos are working and admin is logged in and I can run a dig list this:

dig +tcp 8.8.8.8

; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc33 <<>> +tcp 8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 34505
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 6e77b82359098a81eb672b16607f3178d4f2cafa5841f4eb (good)
;; QUESTION SECTION:
;8.8.8.8.                       IN      A

;; AUTHORITY SECTION:
.                       792     IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2021042002 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Apr 20 15:54:32 EDT 2021
;; MSG SIZE  rcvd: 139

Also, rdns = false in/etc/krb5.conf.

aebruno commented 3 years ago

Check this krb5.conf setting which tells clients to use dns to locate the KDCs: 

dns_lookup_kdc = true

Also check that the host you're running mokey on is enrolled in freeipa.

SomePersonSomeWhereInTheWorld commented 3 years ago

dns_lookup_kdc = true

This is a test server and I don't have the ability to edit the official University DNS records. Is there a work around for this?

Also check that the host you're running mokey on is enrolled in freeipa

Yes same server

aebruno commented 3 years ago

dns_lookup_kdc = true

This is a test server and I don't have the ability to edit the official University DNS records. Is there a work around for this?

You could try setting dns_lookup_kdc = false in your krb5.conf and setting the IP address of your KDCs manually.

SomePersonSomeWhereInTheWorld commented 3 years ago

I set dns_lookup_kdc = false and now getting this:

[Root cause: Encrypting_Error] KRBMessage_Handling_Error: AS Exchange Error: failed setting AS_REQ PAData for pre-authentication required < Encrypting_Error: error getting key from credentials: matching key not found in keytab. Looking for [mokey] realm: OURDOMAIN.EDU kvno: 0 etype: 18" 

klist -kt /etc/krb5.keytab -e
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (aes256-cts-hmac-sha1-96)
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (aes128-cts-hmac-sha1-96)
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (aes128-cts-hmac-sha256-128)
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (aes256-cts-hmac-sha384-192)
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (camellia128-cts-cmac)
   2 01/27/21 10:43:06 host/ourdomain.edu@OURDOMAIN.EDU (camellia256-cts-cmac)
aebruno commented 3 years ago

Did you create a keytab for the mokey user? See the install docs here on the section Create a user account and role in FreeIPA with the "Modify users and Reset passwords" privilege. You set the path to this keytab in the mokey.yaml file:

#------------------------------------------------------------------------------
# Keytab file and username for mokey to user for operations requiring elevated
# privileges (should have "Modify users and Reset passwords" privilege in
# FreeIPA)
#------------------------------------------------------------------------------
keytab: "/path/to/client.keytab"
ktuser: "username"
SomePersonSomeWhereInTheWorld commented 3 years ago

Yes:

#------------------------------------------------------------------------------
# Keytab file and username for mokey to user for operations requiring elevated
# privileges (should have "Modify users and Reset passwords" privilege in
# FreeIPA)
#------------------------------------------------------------------------------
keytab: "/etc/mokey/keytab/mokeyapp.keytab"
ktuser: "mokey"
ls -l /etc/mokey/keytab/mokeyapp.keytab
-rw-r----- 1 root mokey 164 Apr 20 13:29 /etc/mokey/keytab/mokeyapp.keytab
klist -kt /etc/mokey/keytab/mokeyapp.keytab -e
Keytab name: FILE:/etc/mokey/keytab/mokeyapp.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 04/20/21 13:29:15 mokeyapp@OURDOMAIN.EDU (aes256-cts-hmac-sha1-96)
   1 04/20/21 13:29:15 mokeyapp@OURDOMAIN.EDU (aes128-cts-hmac-sha1-96)
aebruno commented 3 years ago

Could be a bug? Not sure as we have not tested this on freeipa 4.9.x.

SomePersonSomeWhereInTheWorld commented 3 years ago

Could be a bug? Not sure as we have not tested this on freeipa 4.9.x.

Nope getting closer:

mokey -c  /etc/mokey/mokey.yaml  --debug server
INFO[0000] Using template dir: /usr/share/mokey/templates
INFO[0000] IPA server:
WARN[0000] **WARNING*** SSL/TLS not enabled. HTTP communication will not be encrypted and vulnerable to snooping.
INFO[0000] Running on http://0.0.0.0:8080/
FATA[0000] listen tcp 0.0.0.0:8080: bind: address already in use

While I was testing I was using ktuser: "mokey" rather than `ktuser: "mokeyapp"

netstat -a -n -o | grep 8080
tcp        0      0 127.0.0.1:59630         127.0.0.1:8080          ESTABLISHED keepalive (8.69/0/0)
tcp        0      0 127.0.0.1:59632         127.0.0.1:8080          ESTABLISHED keepalive (0.20/0/0)
tcp6       0      0 :::8080                 :::*                    LISTEN      off (0.00/0/0)
tcp6       0      0 127.0.0.1:8080          127.0.0.1:59630         ESTABLISHED off (0.00/0/0)
tcp6       0      0 127.0.0.1:8080          127.0.0.1:59632         ESTABLISHED off (0.00/0/0)
lsof -i :8080
COMMAND    PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    335493 pkiuser   89u  IPv6 1093444      0t0  TCP *:webcache (LISTEN)
ps -ef | grep 335493
pkiuser   335493       1 14 16:43 ?        00:00:42 /usr/lib/jvm/java-11-openjdk/bin/java -Dcom.redhat.fips=false -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start

Tomcat is a part of FreeIPA, no?

aebruno commented 3 years ago

Nice! yes you can change the port with (just need to pick an open port):

#------------------------------------------------------------------------------
# Secure webserver port to listen on
#------------------------------------------------------------------------------
port: 8080
SomePersonSomeWhereInTheWorld commented 3 years ago

Yes indeed! From debug:

INFO[0017] Redirect URL wyaf=/

From system logs:

Credentials allowed by configuration
Apr 20 16:58:45 ourserver gssproxy[7317]:    GSSX_RES_INIT_SEC_CONTEXT( status: { 1 { 1 2 840 113554 1 2 2 } 0 "The routine must be called again to complete its function" "" [  ] } context_handle: { [ ......H............ ] [  ] 0 { 1 2 840 113554 1 2 2 } "" "" 0 314 1 0 } output_token: [ ...0....H.......... ] [ { [ 73796e635f63726564730 ] [ ....ouruser.OURDOMAIN... ] } ] )
Apr 20 16:58:45 ourserver mokey[336224]: time="2021-04-20T16:58:45-04:00" level=info msg="Redirect URL" wyaf=/

So the web page just reloads to itself after a login attempt. Sounds like this, what's the fix? I tried your previous suggestion.

SomePersonSomeWhereInTheWorld commented 3 years ago

On some users it appears to work and the browser screen will show:

A reset password email has been sent. Check your email for further instructions.

However in the system logs I see:

level=error msg="Forgotpw: user already has active token"

And in the Apache logs:

[Wed Apr 21 13:54:49.342971 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Wed Apr 21 13:54:49.343089 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: WSGI jsonserver.__call__:
[Wed Apr 21 13:54:49.343169 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: KerberosWSGIExecutioner.__call__:
[Wed Apr 21 13:54:49.367291 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: Created connection context.ldap2_139680801951312
[Wed Apr 21 13:54:49.367401 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: WSGI WSGIExecutioner.__call__:
[Wed Apr 21 13:54:49.367813 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: raw: user_show('ouruser', all=True, version='2.156', no_members=False)
[Wed Apr 21 13:54:49.368031 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: user_show('ouruser', rights=False, all=True, raw=False, version='2.156', no_members=False)
[Wed Apr 21 13:54:49.396525 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: INFO: [jsonserver_kerb] mokeyapp@OURDOMAIN.EDU: user_show('ouruser', all=True, version='2.156', no_members=False): SUCCESS
[Wed Apr 21 13:54:49.396651 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: [jsonserver_kerb] mokeyapp@OURDOMAIN.EDU: user_show('ouruser', all=True, version='2.156', no_members=False): SUCCESS etime=28954265
[Wed Apr 21 13:54:49.397762 2021] [wsgi:error] [pid 366876:tid 367236] [remote x.x.x.x:52264] ipa: DEBUG: Destroyed connection context.ldap2_139680801951312

Any ideas?

aebruno commented 3 years ago

@RobbieTheK can you provide some more context here? Are you resetting a users password? What steps did you do to set up the user?

SomePersonSomeWhereInTheWorld commented 3 years ago

@RobbieTheK can you provide some more context here? Are you resetting a users password? What steps did you do to set up the user?

Sure. Steps to reproduce are simple. Fill out the forgot password. Don't check email. Try forgot password again and see logs. The subsequent forgot password email is never sent.

aebruno commented 3 years ago

@RobbieTheK Thanks for the extra info. Currently, the user can't request a new token until the previous one expires.

SomePersonSomeWhereInTheWorld commented 3 years ago

@RobbieTheK Thanks for the extra info. Currently, the user can't request a new token until the previous one expires.

OK perhaps this is a feature request, to alert the user on the landing page in this condition. It can be confusing if a user sees a message that says another message has been sent, perhaps to alert the user that a password request was already performed?

aebruno commented 3 years ago

@RobbieTheK agreed. Just filed #86

aebruno commented 3 years ago

So going back to your original issue, can you confirm mokey works with Fedora 33?

SomePersonSomeWhereInTheWorld commented 3 years ago

So going back to your original issue, can you confirm mokey works with Fedora 33?

Yes indeed just some of these usability issues. Here's another potential feature request. If a user's password was reset in the FreeIPA UI, they will be prompted by kinit via CLI to reset their password. However the Mokey UI returns Invalid login which is true but again may confuse users. I suppose an easy fix may be to update the "Invalid login" to also say something like "check your login with kinit to see if your password needs resetting" or something of the ilk.

From /var/log/messages:

level=error msg="Failed login attempt" ipa_client_error="IPA login failed with HTTP status code: 401" uid=xxxx

From Apache error log:

[wsgi:error] Password expired.  You must change it now.
[wsgi:error] [pid 619170:tid 619467] [remote x.x.x.x:35730] Enter new password:
[wsgi:error] [pid 619170:tid 619467] [remote x.x.x.x:35730]
[wsgi:error] [pid 619170:tid 619467] [remote x.x.x.x:35730] ipa: DEBUG: stderr=kinit: Cannot read password while getting initial credentials
[wsgi:error] [pid 619170:tid 619467] [remote x.x.x.x:35730]
[wsgi:error] [pid 619170:tid 619467] [remote x.x.x.x:35730] ipa: INFO: 401 Unauthorized: kinit: Cannot read password while getting initial credentials
aebruno commented 3 years ago

Yes indeed just some of these usability issues

Excellent! Thanks for testing this out on Fedora 33. Glad to hear it's working.

Here's another potential feature request. If a user's password was reset in the FreeIPA UI, they will be prompted by kinit via CLI to reset their password. However the Mokey UI returns Invalid login

See #84

I'm going to go ahead and close this issue out. Feel free to submit a new one if you find other issues/features etc. Thanks for all the great feedback.