Closed pki-bot closed 4 years ago
Comment from jamesmasson (@james-masson) at 2015-11-12 15:03:25
Version information
[root@foo ~]# rpm -qa | grep ipa
python-iniparse-0.4-9.el7.noarch
libipa_hbac-1.12.2-58.el7_1.17.x86_64
ipa-server-4.1.0-18.el7.centos.4.x86_64
libipa_hbac-python-1.12.2-58.el7_1.17.x86_64
sssd-ipa-1.12.2-58.el7_1.17.x86_64
ipa-python-4.1.0-18.el7.centos.4.x86_64
ipa-admintools-4.1.0-18.el7.centos.4.x86_64
ipa-client-4.1.0-18.el7.centos.4.x86_64
[root@foo ~]# rpm -qa | grep pki
pki-tools-10.2.6-7.el7.centos.x86_64
dogtag-pki-server-theme-10.2.6-1.el7.centos.noarch
pki-base-10.2.6-7.el7.centos.noarch
pki-server-10.2.6-7.el7.centos.noarch
dogtag-pki-console-theme-10.2.6-1.el7.centos.noarch
pki-console-10.2.6-1.el7.centos.noarch
krb5-pkinit-1.12.2-15.el7_1.x86_64
pki-ca-10.2.6-7.el7.centos.noarch
Comment from jamesmasson (@james-masson) at 2015-11-12 15:10:33
I have this all working for IPA Sub-CA certs signed by a seperate external Dogtag instance.
There is something specific about certificates signed by this particular (non-dogtag) CA, which Dogtag (inside IPA) chokes on, but otherwise passes all validation.
Comment from jamesmasson (@james-masson) at 2015-11-12 18:20:21
The debug logs seem to indicate the 'caSigningCert cert-pki-ca' is the certificate it has problems with.
[12/Nov/2015:12:41:35][localhost-startStop-1]: CertUtils:
verifySystemCerts() cert tag=signing
[12/Nov/2015:12:41:35][localhost-startStop-1]: CertUtils:
verifySystemCertByTag(signing)
[12/Nov/2015:12:41:35][localhost-startStop-1]: CertUtils:
verifySystemCertByNickname(caSigningCert cert-pki-ca,SSLCA)
[12/Nov/2015:12:41:35][localhost-startStop-1]: CertUtils:
verifySystemCertByNickname(): calling isCertValid()
[12/Nov/2015:12:41:35][localhost-startStop-1]: CertUtils:
verifySystemCertByNickname() failed: caSigningCert cert-pki-ca
[12/Nov/2015:12:41:35][localhost-startStop-1]: SignedAuditEventFactory:
create()
message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=caSigningCert
cert-pki-ca] CIMC certificate verification
But further checking seems to indicate the cert passes the relevant checks ( Y A L )
[root@foo ~]# certutil -V -u Y -d /var/lib/pki/pki-tomcat/alias -n
'caSigningCert cert-pki-ca'
certutil: certificate is valid
[root@foo ~]# certutil -V -u A -d /var/lib/pki/pki-tomcat/alias -n
'caSigningCert cert-pki-ca'
certutil: certificate is valid
[root@foo ~]# certutil -V -u L -d /var/lib/pki/pki-tomcat/alias -n
'caSigningCert cert-pki-ca'
certutil: certificate is valid
Comment from cfu (@cfu) at 2015-11-16 21:01:04
The -O option of certutil could also help see if the certs really chains up or not:
e.g. certutil -d /var/lib/pki/pki-tomcat/alias -O -n "ocspSigningCert cert-pki-ca"
If it chains up, the output of this command would be the cert in question plus all the ca certs that chains all the way to the root cert.
Another thing worth investigation are the validities of the certs in comparison with the CA signing cert, and then in comparison with the system were you are doing installation.
Comment from jamesmasson (@james-masson) at 2015-11-17 18:27:09
All the certs chain correctly
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
root.com CT,c,
ocspSigningCert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "root.com"
"root.com" [CN=root.com]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "ocspSigningCert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
"ocspSigningCert cert-pki-ca" [CN=OCSP Subsystem,O=LOCAL]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "caSigningCert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "root.com"
"root.com" [CN=root.com]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "ocspSigningCert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
"ocspSigningCert cert-pki-ca" [CN=OCSP Subsystem,O=LOCAL]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "subsystemCert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
"subsystemCert cert-pki-ca" [CN=CA Subsystem,O=LOCAL]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "Server-Cert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
"Server-Cert cert-pki-ca" [CN=foo.local,O=LOCAL]
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -O -n "auditSigningCert cert-pki-ca"
"root.com" [CN=root.com]
"caSigningCert cert-pki-ca" [CN=Certificate Authority,O=LOCAL]
"auditSigningCert cert-pki-ca" [CN=CA Audit,O=LOCAL]
Validity is good too
[root@foo ~]# date
Tue Nov 17 16:23:09 UTC 2015
[root@foo ~]# openssl x509 -in ipa.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
43:79:78:46:73:cc:25:3d:70:8d:92:1b:25:b4:6c:bb:5a:87:65:66
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=root.com
Validity
Not Before: Nov 17 16:17:13 2015 GMT
Not After : Nov 28 11:17:13 2015 GMT
Subject: O=LOCAL, CN=Certificate Authority
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
Would it help if I signed a CSR for you, so you can replicate this issue locally?
Comment from jamesmasson (@james-masson) at 2015-11-17 18:49:30
Root CA cert is valid too
[root@foo ~]# date
Tue Nov 17 16:48:51 UTC 2015
[root@foo ~]# openssl x509 -in vaultca.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
15:24:4c:26:9b:a9:af:ac:ce:60:dc:f5:bc:29:b6:c8:02:92:fd:73
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=root.com
Validity
Not Before: Nov 12 12:19:23 2015 GMT
Not After : Dec 11 16:19:23 2015 GMT
Subject: CN=root.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
Comment from cfu (@cfu) at 2015-11-23 20:42:05
Hi, could you display cert pretty print for any of the system certs, e.g. "ocspSigningCert cert-pki-ca" so I can take a look of the validity?
Also, I noticed that you have very short validity for the CA's, is this intentional?
Comment from cfu (@cfu) at 2015-11-30 20:35:54
Replying to [comment:8 cfu]:
Hi, could you display cert pretty print for any of the system certs, e.g. "ocspSigningCert cert-pki-ca" so I can take a look of the validity?
Also, I noticed that you have very short validity for the CA's, is this intentional?
Hi, just wondering if there is an update? thanks.
Comment from jamesmasson (@james-masson) at 2015-12-01 14:59:10
Would you like a demo environment for this problem? It's 100% reproducible. Or I'm happy to sign a CSR for you.
Yes, we have deliberately short certificate lifetimes. The same length lifetimes are no problem when using certificates signed by an external Dogtag instance. Varying the lifetimes has no effect, have gone between a few days and a few months.
Here's a pretty printed OCSP cert
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -L -n "ocspSigningCert cert-pki-ca"
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: PKCS 1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=LOCAL"
Validity:
Not Before: Tue Dec 01 12:48:34 2015
Not After : Mon Nov 20 12:48:34 2017
Subject: "CN=OCSP Subsystem,O=LOCAL"
Subject Public Key Info:
Public Key Algorithm: PKCS 1 RSA Encryption
RSA Public Key:
Modulus:
ef:ac:bd:25:d6:48:b3:b0:38:17:2c:d4:11:39:04:e3:
62:99:66:07:7c:6f:b6:84:a7:62:3b:cd:01:51:ee:e3:
95:37:aa:37:3f:af:dc:e7:21:ea:1b:9d:97:aa:82:6a:
58:fe:2e:36:a4:7a:78:e3:9f:2c:3a:1b:3f:f7:9a:d0:
10:e4:57:d4:dd:d6:85:30:cb:91:e6:24:32:5f:a3:b3:
90:64:cb:db:7d:71:b9:3b:e9:85:cd:06:39:72:04:bf:
bc:a0:d9:7b:88:72:b9:6b:97:74:65:fd:92:2b:23:0c:
8d:f4:ce:58:02:8f:46:32:b3:70:0f:e6:8c:c2:fc:6d:
4d:f6:cd:03:d7:e1:64:ea:a3:5e:8b:6c:86:b6:f0:73:
a4:37:94:71:c3:8e:a5:24:24:93:34:8b:ba:cf:6f:7e:
28:3c:1f:1f:9a:78:90:3e:97:cf:a5:e8:e8:c0:43:df:
be:eb:da:1c:0c:d0:71:01:36:de:c7:66:cc:2b:9a:00:
7d:52:1a:7c:65:5a:3c:a5:5f:9b:3b:de:25:dc:cf:3d:
0e:5e:51:c8:39:bf:06:23:e2:1d:cd:54:4c:d7:2f:89:
83:70:16:9a:64:49:7a:89:5a:28:61:c5:ce:f9:28:65:
8b:1b:89:3d:9e:5a:4c:bd:60:db:66:ec:dd:9f:73:f7
Exponent: 65537 (0x10001)
Signed Extensions:
Name: Certificate Authority Key Identifier
Key ID:
4c:6c:57:90:2d:05:cd:57:e5:28:72:80:62:ef:56:56:
e7:a2:18:0f
Name: Certificate Key Usage
Critical: True
Usages: Digital Signature
Non-Repudiation
Certificate Signing
CRL Signing
Name: Authority Information Access
Method: PKIX Online Certificate Status Protocol
Location:
URI: "http://foo.local:80/ca/ocsp"
Name: Extended Key Usage
OCSP Responder Certificate
Signature Algorithm: PKCS 1 SHA-256 With RSA Encryption
Signature:
06:0a:f0:ab:53:42:f7:82:98:14:a5:e0:19:3e:a5:0a:
8e:5d:b9:eb:42:e2:ad:7e:f3:e3:1a:04:f8:f4:dd:7f:
1a:07:39:fa:a7:5b:b5:27:02:2f:ba:16:ac:71:c4:9b:
e3:f0:af:fd:0f:ab:d2:11:42:d2:4e:ff:ea:68:9b:48:
fd:c8:f7:d4:fd:36:0a:0e:f2:7a:3b:61:be:01:37:c0:
69:a2:a8:c7:af:84:3d:15:50:7d:12:77:40:f1:f2:d0:
c1:fd:5b:05:7e:4a:5b:c8:64:8d:d2:10:df:6f:46:c3:
49:8f:ae:c7:36:42:6a:8a:49:4a:7c:0f:29:06:b2:c5:
9f:58:bb:a6:38:97:b2:d6:95:3e:db:32:da:41:f4:b1:
15:2a:92:0a:38:59:12:e3:b9:36:be:d2:3d:0f:08:8e:
e6:00:14:92:44:a1:e2:9c:59:7d:b1:46:19:09:0e:67:
fc:aa:bb:94:1e:af:47:10:fc:7d:b2:c2:68:e9:29:d9:
15:c4:a4:e9:3e:90:c4:f2:12:a9:28:39:c9:23:9c:b0:
fb:cd:51:56:91:d9:7c:a2:bd:9e:1f:3c:b9:89:14:e4:
2f:48:2d:cc:24:fa:f9:c0:4a:ea:cf:c5:ae:16:d8:ba:
71:6c:83:7a:c5:e5:e1:c7:5a:08:bb:96:1f:9b:9a:30
Fingerprint (SHA-256):
29:E3:70:35:87:8D:68:92:44:18:85:7A:4B:59:4A:90:1B:CE:62:CD:E9:A3:D6:BC:5F:99:37:20:8D:5C:4C:EE
Fingerprint (SHA1):
56:07:41:20:F0:AE:96:87:65:75:C8:75:B4:39:F4:6B:FF:08:8B:59
Certificate Trust Flags:
SSL Flags:
User
Email Flags:
User
Object Signing Flags:
User
Comment from cfu (@cfu) at 2015-12-01 21:27:34
your signing CA expired before it signed the OCSP system cert? That doesn't seem right.
Validity
Not Before: Nov 17 16:17:13 2015 GMT
Not After : Nov 28 11:17:13 2015 GMT
Subject: O=LOCAL, CN=Certificate Authority
==== Validity: Not Before: Tue Dec 01 12:48:34 2015 Not After : Mon Nov 20 12:48:34 2017 Subject: "CN=OCSP Subsystem,O=LOCAL"
Comment from jamesmasson (@james-masson) at 2015-12-01 22:04:15
I think you're confusing the previous IPA CA cert with the root cert.
See comment 7 for the root ( CA that signed the IPA cert - refered to as vaultCA ) certificate details
Comment from cfu (@cfu) at 2015-12-01 23:57:14
Replying to [comment:13 james-masson]:
I think you're confusing the previous IPA CA cert with the root cert.
See comment 7 for the root ( CA that signed the IPA cert - refered to as vaultCA ) certificate details
No, I was NOT talking at the root cert, as it looks fine. Please read them again again, The OCSP signing cert was signed by "Subject: O=LOCAL, CN=Certificate Authority", and according to comment 6, "Subject: O=LOCAL, CN=Certificate Authority" has the validity of Not Before: Nov 17 16:17:13 2015 GMT Not After : Nov 28 11:17:13 2015 GMT , while in comment 10, it shows: Issuer: "CN=Certificate Authority,O=LOCAL" Validity: Not Before: Tue Dec 01 12:48:34 2015 Not After : Mon Nov 20 12:48:34 2017 Subject: "CN=OCSP Subsystem,O=LOCAL"
Which tells me that the OCSP cert does not become valid until 12/1, while its issuer, "CN=Certificate Authority,O=LOCAL" already expired by Nov 28 11:17:13 2015 GMT.
Comment from jamesmasson (@james-masson) at 2015-12-02 00:16:17
The IPA CA cert is not the same one as before - as you mentioned, it expired and became invalid as expected.
Every time I update this ticket I generate a new CSR and sign it.
Here's this iteration's IPA certificate
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -L -n "caSigningCert cert-pki-ca"
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
27:e5:19:98:a8:8f:11:b1:31:24:1f:9b:20:a9:3b:63:
f1:31:b8:73
Signature Algorithm: PKCS 1 SHA-256 With RSA Encryption
Issuer: "CN=root.com"
Validity:
Not Before: Tue Dec 01 12:46:09 2015
Not After : Sun Dec 06 12:46:09 2015
Subject: "CN=Certificate Authority,O=LOCAL"
Subject Public Key Info:
Public Key Algorithm: PKCS 1 RSA Encryption
RSA Public Key:
Modulus:
Comment from cfu (@cfu) at 2015-12-02 00:36:45
ok, that explains it. thanks. Another possibility is https://bugzilla.redhat.com/show_bug.cgi?id=1282892 In the bug, you will see that I was not sure when it became broken, so it is unclear if the issue exists in the bits used by IPA. Endi is going to add some debugging to assist with the investigation.
Comment from cfu (@cfu) at 2015-12-02 00:42:29
Replying to [comment:16 cfu]:
ok, that explains it. thanks. Another possibility is https://bugzilla.redhat.com/show_bug.cgi?id=1282892 In the bug, you will see that I was not sure when it became broken, so it is unclear if the issue exists in the bits used by IPA. Endi is going to add some debugging to assist with the investigation.
actually, I just read the previous comments where you showed that the certs chained up. If that's case, then it's probably not 1282892. Either way, the extra debugging Endi is working on will help.
Comment from edewata (@edewata) at 2015-12-02 03:03:12
Hi James,
Could you install the following build and try again? https://copr-be.cloud.fedoraproject.org/results/edewata/pki-epel/epel-7-x86_64/00144216-pki-core/
Please attach the debug log in /var/log/pki/pki-tomcat/ca/debug. It's supposed to show exactly where the validation fails in PKI. If this is still not sufficient, we may need to dig deeper into NSS/JSS.
Comment from jamesmasson (@james-masson) at 2015-12-02 12:28:11
It seems to be pointing the finger at the IPA CA signing cert - which is what I would expect. I've attached the full file, but here's a snippet.
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCerts() cert tag=signing
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertByTag(signing)
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(caSigningCert cert-pki-ca, SSLCA)
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(): calling isCertValid()
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: caSigningCert cert-pki-ca
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
[02/Dec/2015:10:21:21][localhost-startStop-1]: CertUtils: verifySystemCertsByTag() failed: java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
[02/Dec/2015:10:21:21][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=] CIMC certificate verification
[02/Dec/2015:10:21:21][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=] CIMC certificate verification
java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:855)
at com.netscape.cmscore.cert.CertUtils.verifySystemCertByTag(CertUtils.java:946)
at com.netscape.cmscore.cert.CertUtils.verifySystemCerts(CertUtils.java:1061)
at com.netscape.cmscore.apps.CMSEngine.verifySystemCerts(CMSEngine.java:1732)
at com.netscape.certsrv.apps.CMS.verifySystemCerts(CMS.java:1380)
at com.netscape.cms.selftests.common.SystemCertsVerification.runSelfTest(SystemCertsVerification.java:193)
at com.netscape.cmscore.selftests.SelfTestSubsystem.runSelfTestsAtStartup(SelfTestSubsystem.java:861)
at com.netscape.cmscore.selftests.SelfTestSubsystem.startup(SelfTestSubsystem.java:1811)
at com.netscape.cmscore.apps.CMSEngine.startupSubsystems(CMSEngine.java:1843)
at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1291)
at com.netscape.certsrv.apps.CMS.startup(CMS.java:200)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1609)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:536)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:123)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1272)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5210)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5493)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:672)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1862)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[02/Dec/2015:10:21:21][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=SELFTESTS_EXECUTION][SubjectID=$System$][Outcome=Failure] self tests execution (see selftests.log for details)
Here's the pretty print of the cert IPA CA cert
[root@foo ~]# certutil -d /var/lib/pki/pki-tomcat/alias -L -n 'caSigningCert cert-pki-ca'
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
51:62:8d:ea:2c:28:b6:d3:68:07:88:dc:d1:d6:18:d7:
0a:a9:7b:6e
Signature Algorithm: PKCS 1 SHA-256 With RSA Encryption
Issuer: "CN=root.com"
Validity:
Not Before: Wed Dec 02 10:17:51 2015
Not After : Mon Dec 07 10:17:51 2015
Subject: "CN=Certificate Authority,O=LOCAL"
Subject Public Key Info:
Public Key Algorithm: PKCS 1 RSA Encryption
RSA Public Key:
Modulus:
cd:3a:7b:8a:cc:5c:8a:78:c7:e4:63:ad:e0:86:60:5e:
50:d5:8b:f3:15:ae:9a:5a:e9:17:14:cb:6b:b0:bc:af:
78:5a:4e:96:cc:a5:ef:60:e6:4a:68:18:f9:ed:98:68:
c8:3e:9c:8f:0e:75:3c:8c:19:f3:be:ab:9e:50:e4:c6:
78:13:e9:0b:2b:83:81:24:2d:ac:7d:5c:41:1d:3c:d7:
9b:84:82:1b:b5:6d:79:a6:f3:c9:df:88:37:3f:66:6d:
63:ce:3b:ed:23:76:4a:61:78:0e:a6:53:38:06:5e:63:
0f:fb:c4:96:96:c4:31:9f:00:c1:c9:7e:0f:d8:75:0d:
3f:03:62:47:49:d6:5b:f2:6e:b1:b1:60:ae:ed:10:ec:
13:39:5a:a7:3a:72:6a:b6:04:60:3b:2a:64:59:71:71:
7d:4b:a1:ff:6d:0b:50:06:e2:f3:0b:e7:ad:09:96:af:
a0:30:3e:de:72:ce:cf:47:4e:eb:f2:e7:97:b1:19:2a:
5a:18:eb:06:25:37:17:d4:73:2f:3a:53:c7:e5:6d:86:
0b:8a:0c:3e:1f:b8:f8:1f:f6:55:c9:b7:3e:f1:84:d7:
b6:37:40:31:85:5d:43:16:59:3b:9e:1b:21:3a:77:9e:
82:ed:e4:31:ba:94:31:cd:6f:c2:7f:cc:8c:d9:33:33
Exponent: 65537 (0x10001)
Signed Extensions:
Name: Certificate Subject Key ID
Data:
38:af:c0:f3:7c:31:d6:79:6a:81:c0:8b:4f:17:d9:58:
12:16:7a:ba
Name: Certificate Authority Key Identifier
Key ID:
dd:60:70:f2:fe:f3:bf:ed:01:65:64:7f:35:93:67:f6:
23:12:b0:07
Name: Certificate Basic Constraints
Critical: True
Data: Is a CA with no maximum path length.
Name: Certificate Key Usage
Critical: True
Usages: Digital Signature
Non-Repudiation
Certificate Signing
CRL Signing
Signature Algorithm: PKCS 1 SHA-256 With RSA Encryption
Signature:
48:13:b3:55:91:94:db:8f:8f:63:b9:91:c9:64:5a:29:
f4:ab:5d:61:35:47:ce:9f:40:33:6f:e0:e9:34:84:63:
74:51:39:dc:d8:71:aa:56:23:5a:63:df:59:dc:dd:47:
8c:f5:57:2a:16:50:c4:ed:b5:4b:ca:a3:d9:e5:f3:d4:
af:59:a2:48:99:84:96:f1:4e:ad:a3:c6:55:b2:1c:19:
22:4a:b5:eb:6c:13:5a:76:1a:24:82:1d:0f:e0:32:1b:
45:25:57:6a:3a:19:06:dc:4b:33:b6:be:75:55:88:ed:
5d:d7:99:9c:b1:6f:01:2c:29:84:7d:f2:89:2f:21:75:
02:c8:78:68:94:cf:8c:72:b5:2c:74:24:71:33:bb:6d:
7b:47:27:35:d1:9d:f2:52:ed:3f:f7:df:ae:4b:72:66:
2b:3d:ba:e6:f8:ac:ce:ad:37:93:64:b2:c3:bd:52:e4:
8c:06:e4:55:cf:76:89:42:fe:95:2f:18:60:ad:6f:ae:
fe:69:bd:59:e1:49:94:a7:a6:f0:ce:f0:61:49:9a:e1:
61:76:c0:4e:34:c4:be:8b:d0:6c:f0:1c:ed:95:5b:7e:
8a:d3:08:d4:69:8e:82:8a:99:cc:05:8f:34:96:3f:a0:
84:0f:1d:94:c9:60:18:40:f1:5c:ea:fa:e6:29:7a:ef
Fingerprint (SHA-256):
C7:06:6A:4B:66:1A:A9:A0:39:FD:B0:48:6D:ED:8F:19:69:FD:A8:4C:F4:ED:46:0A:60:53:30:3B:9F:4C:EA:F2
Fingerprint (SHA1):
50:50:47:21:CD:3E:35:DF:5E:B1:7D:D0:DD:C5:25:51:58:7E:4B:E3
Certificate Trust Flags:
SSL Flags:
Valid CA
Trusted CA
User
Trusted Client CA
Email Flags:
Valid CA
Trusted CA
User
Object Signing Flags:
Valid CA
Trusted CA
User
Comment from jamesmasson (@james-masson) at 2015-12-02 12:30:33
debug log, with extra debug! pki_debug.log.gz
Comment from edewata (@edewata) at 2015-12-03 20:56:24
Hi James,
Thanks for the info. It looks like the certutil and JSS are calling the same validation function (i.e. CERT_VerifyCertificate()) but they are handling the result differently. I will need to do further debugging in JSS/NSS. Could you sign the following CSR so I can reproduce the problem? Thanks!
-----BEGIN NEW CERTIFICATE REQUEST-----
MIICnzCCAYcCAQAwWjE3MDUGA1UEChMuYWJjLmlkbS5sYWIuZW5nLmJycS5yZWRo
YXQuY29tIFNlY3VyaXR5IERvbWFpbjEfMB0GA1UEAxMWQ0EgU2lnbmluZyBDZXJ0
aWZpY2F0ZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPEzaIJd1QcL
6D+KOmU377fac31VjchHcjBunqcvGx5KS2jKWzaNg72+6FLULYczrgfJQZnPgNVE
ZxnXwbFSTzPVpXOiTlOR9FF9ohS0GR3p2kob5UK51B933FIx+rcHdqAoezJF6lFI
+LGjhEHpl7fsK/Fd5cgqaRtknJOVk0RFKrWV27FOA5xPHiltfY+qagNb4Bou2xK7
Y+cEWnlnX7z7+yMKowFCX1efl4pmYSLI9riTD9lk3KNYpvZlbf3ee1CQ5X7SfP39
zFN7jkU5W5Bk1fZIZLFGcsO1h59Pm3bSypzmhQ8kdS4v7RO++UaNo6k9BV8fV+y5
yGLwk9/S8AMCAwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQCXJVqH1ebNk5CXPoQE
e1CPGnmU7Y4neQn3/Cx7gy+O7M4KygfmSGybxCyIut2JXBdaapWS4K4tZcE8cxkW
a5Vit0Z1FyLqkT9Chwdz9dbTK/CO7nLhdHmJ9XLsE2hnzfxeuSk1IAKMqCGTVMkn
PzqWHXqMgzPgn6aTbhiNE/v2zvCRMLu9JmefzMyTD8ImByT1o805S3GqvkDBg6j8
3thCHk5n3VVaZM8Puldka8uYW6XyFzRD4/SitcY+CaJxYHQNMYVMkymbSJPF0ZVM
ilnA5My4RD9R72U00sId5/SYUlYq/J7OveSv4mQyG6WZCvZdBDRFLRj0gj2d6gFe
68pt
-----END NEW CERTIFICATE REQUEST-----
Comment from jamesmasson (@james-masson) at 2015-12-03 22:37:25
Here you go...
IPA cert
-----BEGIN CERTIFICATE-----
MIIDTjCCAjagAwIBAgIUZ5WLs6nR3K8vd6ycT5zV05NNYfgwDQYJKoZIhvcNAQEL
BQAwEzERMA8GA1UEAxMIcm9vdC5jb20wHhcNMTUxMjAzMjAzMzM5WhcNMTUxMjI0
MTYzMzM5WjBaMTcwNQYDVQQKEy5hYmMuaWRtLmxhYi5lbmcuYnJxLnJlZGhhdC5j
b20gU2VjdXJpdHkgRG9tYWluMR8wHQYDVQQDExZDQSBTaWduaW5nIENlcnRpZmlj
YXRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8TNogl3VBwvoP4o6
ZTfvt9pzfVWNyEdyMG6epy8bHkpLaMpbNo2Dvb7oUtQthzOuB8lBmc+A1URnGdfB
sVJPM9Wlc6JOU5H0UX2iFLQZHenaShvlQrnUH3fcUjH6twd2oCh7MkXqUUj4saOE
QemXt+wr8V3lyCppG2Sck5WTREUqtZXbsU4DnE8eKW19j6pqA1vgGi7bErtj5wRa
eWdfvPv7IwqjAUJfV5+XimZhIsj2uJMP2WTco1im9mVt/d57UJDlftJ8/f3MU3uO
RTlbkGTV9khksUZyw7WHn0+bdtLKnOaFDyR1Li/tE775Ro2jqT0FXx9X7LnIYvCT
39LwAwIDAQABo1MwUTAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSYG1d2bOXo
80R9iG0EhqvIxzEY8DAfBgNVHSMEGDAWgBQJjNngux7zFLBCiKPPvdSXGliG+DAN
BgkqhkiG9w0BAQsFAAOCAQEAfPYfp2BHDWYoU7VvFswmmHDCWZxnGIszQBq+pzD1
1wnUg6EKVVDNhsLlfIjKeDHdng/69lf7ApIbOWwYKC3sicAYpBrQVz/GuZ3CM8TI
TU6JyVJpRw9TSnzcIMEU2OQP85elkSIUg/5UrYcBKaPLZ1kvIQ3BO+Wa2+xz50K2
ApFssY+zjc6ZV0PT/8yB4VIfORiMMcEiqccOEsbPpQf+IP2HJGIaJNYAWoT7mS8Z
WwSpK5rhTTKdB5CYjmMduiibrH7b5+Jsc9EpfYEC0yLcbp/5bwLPdPcOYwBwGYMQ
tA28FDyd5IpikKNhOWQ5ZWbMpKlW9Cbj4Bn0gxtd4K6+ew==
-----END CERTIFICATE-----
New root cert - with a bit more validity time...
-----BEGIN CERTIFICATE-----
MIIDQzCCAiugAwIBAgIUQPZZ3dzbDvPykG3hqBHtmk/S3DIwDQYJKoZIhvcNAQEL
BQAwEzERMA8GA1UEAxMIcm9vdC5jb20wHhcNMTUxMjAzMjAyNzU1WhcNMTYwMTAy
MjAyNzU1WjATMREwDwYDVQQDEwhyb290LmNvbTCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBALB2b5Z1YJ3riEUeDyiW8YPZgYbdyh0FD+AZyQr6e8lAyynE
rSOliN4k9kLTb6E42IFLxXD+OPlTy445Cd7+jUNln0WqKFH5lVi7t5h0DjmtKPQC
dy4ocxYfIUUrs7m4FL85g0m53v6gLR06fJSHLOqrpwYowio6+hV1/Fq37rp73VAF
YBK6XC3IPOdDjMuAjJFavBhtADXCJ+jJR2nulyO4DeVP03PzkTxYA9w3K88AFUL5
4Afl0VvFn+6FBJj0WCKER1SW0H9D6AWjQd7h0m3vavcURHF3vGHlTlKBoKAI4Ai1
FgdNs616ylK1jYvl+wOJOp5M7KpOzA2CzjF6MN0CAwEAAaOBjjCBizAOBgNVHQ8B
Af8EBAMCAa4wEwYDVR0lBAwwCgYIKwYBBQUHAwkwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQUCYzZ4Lse8xSwQoijz73UlxpYhvgwHwYDVR0jBBgwFoAUCYzZ4Lse
8xSwQoijz73UlxpYhvgwEwYDVR0RBAwwCoIIcm9vdC5jb20wDQYJKoZIhvcNAQEL
BQADggEBAJXYS/PWVcvl2rTX0UTnmHzJuWcI7+p/QqcsPqv6y0WnaB1zQPwQ3MSj
2E5aGroPhA3pSgZV56LmVm8XOGnk0R25jWZq1JRW0unCMjLb55XVwJkRZ6yAG8P1
n3eSsf3IHxLZZLg4LOscqvT1ImZYSYWZIs2JnIOn0ndNeDN+slSydza52VGfF1Sj
vJ3cAusLD4Jsiao3pvVLY+ybtZ1q3D0JNCxIMVp4gH3V4lAb3hLiQw8XQQjmwTFh
qQUoVQc8Mq/JXR5NpIqXmXQQM4bYRnrF5xT2+AuvxnWB/23OMlWyAPuOM3dCuzNT
8B1orCn3zI/Kh2o48PD8m4Qg9Fe2L8E=
-----END CERTIFICATE-----
Comment from edewata (@edewata) at 2015-12-04 03:27:07
Thanks for the certs. Unfortunately I'm unable to reproduce the problem with these certs on Fedora 23. I may need to try this on a RHEL machine (I don't have a CentOS machine). Could you clarify which CentOS version you are using, and how did you get the PKI 10.2.6-7.el7.centos packages? To my understanding the latest RHEL only has PKI 10.2.5-x.
Comment from jamesmasson (@james-masson) at 2015-12-04 12:31:37
I've been experimenting with a couple of PKI version over the last few months I've been working on this issue.
The 10.2.6-7 version was downloaded as a SRPM from rawhide, then built locally on the Centos7 machine.
I'm happy to offer you a public VM to test with, if you wish.
I've just reverted back to a stock, fully patched Centos7 image ( which did update NSS, interestingly ), with stock IPA/PKI - and now I get a different error
[root@foo ~]# rpm -qa | sort | egrep "ipa|pki|nss"
ipa-admintools-4.1.0-18.el7.centos.4.x86_64
ipa-client-4.1.0-18.el7.centos.4.x86_64
ipa-python-4.1.0-18.el7.centos.4.x86_64
ipa-server-4.1.0-18.el7.centos.4.x86_64
jansson-2.4-6.el7.x86_64
krb5-pkinit-1.12.2-15.el7_1.x86_64
libipa_hbac-1.12.2-58.el7_1.18.x86_64
libipa_hbac-python-1.12.2-58.el7_1.18.x86_64
libsss_nss_idmap-1.12.2-58.el7_1.18.x86_64
mod_nss-1.0.8-33.el7.x86_64
nss-3.19.1-7.el7_1.2.x86_64
nss-devel-3.19.1-7.el7_1.2.x86_64
nss-softokn-3.16.2.3-13.el7_1.x86_64
nss-softokn-devel-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-devel-3.16.2.3-13.el7_1.x86_64
nss-sysinit-3.19.1-7.el7_1.2.x86_64
nss-tools-3.19.1-7.el7_1.2.x86_64
nss-util-3.19.1-4.el7_1.x86_64
nss-util-devel-3.19.1-4.el7_1.x86_64
openssh-6.6.1p1-12.el7_1.x86_64
openssh-clients-6.6.1p1-12.el7_1.x86_64
openssh-server-6.6.1p1-12.el7_1.x86_64
openssl-1.0.1e-42.el7.9.x86_64
openssl-devel-1.0.1e-42.el7.9.x86_64
openssl-libs-1.0.1e-42.el7.9.x86_64
pki-base-10.1.2-7.el7.noarch
pki-ca-10.1.2-7.el7.noarch
pki-server-10.1.2-7.el7.noarch
pki-tools-10.1.2-7.el7.x86_64
python-iniparse-0.4-9.el7.noarch
python-nss-0.16.0-2.el7.x86_64
sssd-ipa-1.12.2-58.el7_1.18.x86_64
2015-12-04T10:08:43Z DEBUG args='/usr/bin/certutil' '-d' '/etc/httpd/alias' '-f' XXXXXXXX '-R' '-k' 'rsa' '-g' '2048' '-s' 'CN=IPA RA,O=LOCAL' '-z' '/tmp/tmpQdGKqw' '-a'
2015-12-04T10:08:43Z DEBUG Process finished, return code=0
2015-12-04T10:08:43Z DEBUG stdout=
Certificate request generated by Netscape certutil
Phone: (not specified)
Common Name: IPA RA
Email: (not specified)
Organization: LOCAL
State: (not specified)
Country: (not specified)
-----BEGIN NEW CERTIFICATE REQUEST-----
MIICZjCCAU4CAQAwITEOMAwGA1UEChMFTE9DQUwxDzANBgNVBAMTBklQQSBSQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMk4Q8QYUSDNJMctkx4OBfQT
dkOsGdd6X9aTsiQPeAfcG3gSM0uFgO/ZuvPVDvNJ9pHVzB5ZexTdkXQfbIFlG8+L
ztnFC9u+4hEEcpiOJCOvjmoxUSM4oEThMR7k1nnVERS3URsxVOIWeTsuZv7JIuq5
UU6xwuh8FQ44+dLJdW0kh0UpbwpE1vVWAYaygK6xJvsLYjN8YsVK8zxRYZ/hB8B9
o83ebjz9+79fhDDUwZ5HYHUDkitf/np53LMB6+wYxXOcIc8wKVK1IIz0sR6LjiiQ
0chC78xiTtFpdc9FAi+a92fuXinGGnC2oRjHIXZkW3OMQKMKokTm6uMShuc505sC
AwEAAaAAMA0GCSqGSIb3DQEBCwUAA4IBAQCn6KX65zJgvBf0l8NxP0jZEnFLe/mS
ijSCYukGT3oE5rKZxICV0YFSOT+hRpg1iOjK6e/6L2gTB4RJzDOv0caimJUfnFIA
geycd2Gr1M/waRZyibXG6QqvMLJaTSVVDZmEzdkEdGCSV/67r+u6M7RwVojly3aB
xpcM07Pag0/h4yrHMxUjEtokiyixZ1wdQZONo8FEuYvNGhe4uizn471jlenfYvvS
4IN83URgMeGOQ8pgXoSZVYpLCrKIYjkjPQ6OcuQmS06xAep9X/EtbDtRiKEcm4EC
gLrk1o+IpesA+WKELpPbnLpRp2onLb7vwv+HnLUJw8lViLc/Mgt81jtJ
-----END NEW CERTIFICATE REQUEST-----
2015-12-04T10:08:43Z DEBUG stderr=
Generating key. This may take a few moments...
2015-12-04T10:08:43Z DEBUG Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 382, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 372, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1149, in __request_ra_certificate
self.requestId = item_node[0].childNodes[0].data
IndexError: list index out of range
2015-12-04T10:08:43Z DEBUG [error] IndexError: list index out of range
2015-12-04T10:08:43Z DEBUG File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 646, in run_script
return_value = main_function()
File "/sbin/ipa-server-install", line 1170, in main
ca_signing_algorithm=options.ca_signing_algorithm)
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 520, in configure_instance
self.start_creation(runtime=210)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 382, in start_creation
run_step(full_msg, method)
File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 372, in run_step
method()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1149, in __request_ra_certificate
self.requestId = item_node[0].childNodes[0].data
2015-12-04T10:08:43Z DEBUG The ipa-server-install command failed, exception: IndexError: list index out of range
PKI does come up now - but it's the IPA install code that fails.
I've attached the IPA install and PKI debug logs.
Comment from jamesmasson (@james-masson) at 2015-12-04 12:32:22
stock centos ipa install log stock_centos7_ipaserver-install.log.gz
Comment from jamesmasson (@james-masson) at 2015-12-04 12:32:44
stock centos7 pki debug log stock_centos7_ca_debug.log.gz
Comment from jamesmasson (@james-masson) at 2015-12-04 12:46:12
Taking the same, stock centos7 IPA, and upgrading PKI to your debug builds, gives the original error - PKI fails to come up.
[root@foo tmp]# rpm -qa | sort | egrep "^ipa|^pki|^nss"
ipa-admintools-4.1.0-18.el7.centos.4.x86_64
ipa-client-4.1.0-18.el7.centos.4.x86_64
ipa-python-4.1.0-18.el7.centos.4.x86_64
ipa-server-4.1.0-18.el7.centos.4.x86_64
nss-3.19.1-7.el7_1.2.x86_64
nss-devel-3.19.1-7.el7_1.2.x86_64
nss-softokn-3.16.2.3-13.el7_1.x86_64
nss-softokn-devel-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-devel-3.16.2.3-13.el7_1.x86_64
nss-sysinit-3.19.1-7.el7_1.2.x86_64
nss-tools-3.19.1-7.el7_1.2.x86_64
nss-util-3.19.1-4.el7_1.x86_64
nss-util-devel-3.19.1-4.el7_1.x86_64
pki-base-10.2.6-10.el7.centos.noarch
pki-ca-10.2.6-10.el7.centos.noarch
pki-core-debuginfo-10.2.6-10.el7.centos.x86_64
pki-javadoc-10.2.6-10.el7.centos.noarch
pki-kra-10.2.6-10.el7.centos.noarch
pki-ocsp-10.2.6-10.el7.centos.noarch
pki-server-10.2.6-10.el7.centos.noarch
pki-symkey-10.2.6-10.el7.centos.x86_64
pki-tks-10.2.6-10.el7.centos.noarch
pki-tools-10.2.6-10.el7.centos.x86_64
pki-tps-10.2.6-10.el7.centos.x86_64
[04/Dec/2015:10:40:02][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(): calling isCertValid()
[04/Dec/2015:10:40:02][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: caSigningCert cert-pki-ca
[04/Dec/2015:10:40:02][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
[04/Dec/2015:10:40:02][localhost-startStop-1]: CertUtils: verifySystemCertsByTag() failed: java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
[04/Dec/2015:10:40:02][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=] CIMC certificate verification
[04/Dec/2015:10:40:02][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=] CIMC certificate verification
java.lang.Exception: Invalid certificate caSigningCert cert-pki-ca
at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:855)
at com.netscape.cmscore.cert.CertUtils.verifySystemCertByTag(CertUtils.java:946)
at com.netscape.cmscore.cert.CertUtils.verifySystemCerts(CertUtils.java:1061)
Comment from jamesmasson (@james-masson) at 2015-12-04 13:06:17
It doesn't look like IPA 4.2 builds on EL/Centos7 yet - there was talk of a COPR repo a while back, but it's gone a bit quiet.
Again, happy to give you access to a box to demo this problem.
Comment from edewata (@edewata) at 2015-12-09 07:19:19
I've created JSS and PKI builds that would provide more details on the certificate validation error:
Would you be able to try them? Thanks.
Comment from edewata (@edewata) at 2015-12-10 04:34:05
Here are the EPEL builds:
These builds include the fix for ticket 850.
Comment from jamesmasson (@james-masson) at 2015-12-10 12:52:15
With just the JSS RPMs, here's the debug log
[root@foo ~]# rpm -qa | sort | egrep "^ipa|^pki|^nss"
ipa-admintools-4.1.0-18.el7.centos.4.x86_64
ipa-client-4.1.0-18.el7.centos.4.x86_64
ipa-python-4.1.0-18.el7.centos.4.x86_64
ipa-server-4.1.0-18.el7.centos.4.x86_64
nss-3.19.1-7.el7_1.2.x86_64
nss-devel-3.19.1-7.el7_1.2.x86_64
nss-softokn-3.16.2.3-13.el7_1.x86_64
nss-softokn-devel-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-devel-3.16.2.3-13.el7_1.x86_64
nss-sysinit-3.19.1-7.el7_1.2.x86_64
nss-tools-3.19.1-7.el7_1.2.x86_64
nss-util-3.19.1-4.el7_1.x86_64
nss-util-devel-3.19.1-4.el7_1.x86_64
pki-base-10.2.6-10.el7.centos.noarch
pki-ca-10.2.6-10.el7.centos.noarch
pki-core-debuginfo-10.2.6-10.el7.centos.x86_64
pki-javadoc-10.2.6-10.el7.centos.noarch
pki-kra-10.2.6-10.el7.centos.noarch
pki-ocsp-10.2.6-10.el7.centos.noarch
pki-server-10.2.6-10.el7.centos.noarch
pki-symkey-10.2.6-10.el7.centos.x86_64
pki-tks-10.2.6-10.el7.centos.noarch
pki-tools-10.2.6-10.el7.centos.x86_64
pki-tps-10.2.6-10.el7.centos.x86_64
Comment from jamesmasson (@james-masson) at 2015-12-10 12:52:45
with_jss_debug_rpm pki_debug_v3.log.gz
Comment from jamesmasson (@james-masson) at 2015-12-10 12:55:22
With the new PKI RPMS, the second phase of install just fails.
[root@foo ~]# rpm -qa | sort | egrep "^ipa|^pki|^nss|^jss"
ipa-admintools-4.1.0-18.el7.centos.4.x86_64
ipa-client-4.1.0-18.el7.centos.4.x86_64
ipa-python-4.1.0-18.el7.centos.4.x86_64
ipa-server-4.1.0-18.el7.centos.4.x86_64
jss-4.2.6-38.el7.centos.x86_64
jss-debuginfo-4.2.6-38.el7.centos.x86_64
jss-javadoc-4.2.6-38.el7.centos.x86_64
nss-3.19.1-7.el7_1.2.x86_64
nss-devel-3.19.1-7.el7_1.2.x86_64
nss-softokn-3.16.2.3-13.el7_1.x86_64
nss-softokn-devel-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64
nss-softokn-freebl-devel-3.16.2.3-13.el7_1.x86_64
nss-sysinit-3.19.1-7.el7_1.2.x86_64
nss-tools-3.19.1-7.el7_1.2.x86_64
nss-util-3.19.1-4.el7_1.x86_64
nss-util-devel-3.19.1-4.el7_1.x86_64
pki-base-10.2.5-6.el7.centos.noarch
pki-ca-10.2.5-6.el7.centos.noarch
pki-core-debuginfo-10.2.5-6.el7.centos.x86_64
pki-javadoc-10.2.5-6.el7.centos.noarch
pki-kra-10.2.5-6.el7.centos.noarch
pki-ocsp-10.2.5-6.el7.centos.noarch
pki-server-10.2.5-6.el7.centos.noarch
pki-symkey-10.2.5-6.el7.centos.x86_64
pki-tks-10.2.5-6.el7.centos.noarch
pki-tools-10.2.5-6.el7.centos.x86_64
pki-tps-10.2.5-6.el7.centos.x86_64
[root@foo ~]# /sbin/ipa-server-install -U -a password -p password --hostname=foo.local -r LOCAL -n local --mkhomedir --setup-dns --forwarder=8.8.8.8 --external-ca
The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.
This includes:
* Configure a stand-alone CA (dogtag) for certificate management
* Configure the Network Time Daemon (ntpd)
* Create and configure an instance of Directory Server
* Create and configure a Kerberos Key Distribution Center (KDC)
* Configure Apache (httpd)
* Configure DNS (bind)
Warning: skipping DNS resolution of host foo.local
Checking forwarders, please wait ...
Using reverse zone(s) 33.168.192.in-addr.arpa.
The IPA Master Server will be configured with:
Hostname: foo.local
IP address(es): 192.168.33.100
Domain name: local
Realm name: LOCAL
BIND DNS server will be configured to serve IPA domain with:
Forwarders: 8.8.8.8
Reverse zone(s): 33.168.192.in-addr.arpa.
Configuring NTP daemon (ntpd)
[1/4]: stopping ntpd
[2/4]: writing configuration
[3/4]: configuring ntpd to start on boot
[4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv): Estimated time 1 minute
[1/38]: creating directory server user
[2/38]: creating directory server instance
[3/38]: adding default schema
[4/38]: enabling memberof plugin
[5/38]: enabling winsync plugin
[6/38]: configuring replication version plugin
[7/38]: enabling IPA enrollment plugin
[8/38]: enabling ldapi
[9/38]: configuring uniqueness plugin
[10/38]: configuring uuid plugin
[11/38]: configuring modrdn plugin
[12/38]: configuring DNS plugin
[13/38]: enabling entryUSN plugin
[14/38]: configuring lockout plugin
[15/38]: creating indices
[16/38]: enabling referential integrity plugin
[17/38]: configuring certmap.conf
[18/38]: configure autobind for root
[19/38]: configure new location for managed entries
[20/38]: configure dirsrv ccache
[21/38]: enable SASL mapping fallback
[22/38]: restarting directory server
[23/38]: adding default layout
[24/38]: adding delegation layout
[25/38]: creating container for managed entries
[26/38]: configuring user private groups
[27/38]: configuring netgroups from hostgroups
[28/38]: creating default Sudo bind user
[29/38]: creating default Auto Member layout
[30/38]: adding range check plugin
[31/38]: creating default HBAC rule allow_all
[32/38]: initializing group membership
[33/38]: adding master entry
[34/38]: configuring Posix uid/gid generation
[35/38]: adding replication acis
[36/38]: enabling compatibility plugin
[37/38]: tuning directory server
[38/38]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring certificate server (pki-tomcatd): Estimated time 3 minutes 30 seconds
[1/8]: creating certificate server user
[2/8]: configuring certificate server instance
The next step is to get /root/ipa.csr signed by your CA and re-run /sbin/ipa-server-install as:
/sbin/ipa-server-install --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate
[root@foo ~]# /sbin/ipa-server-install --external-cert-file=/root/ipa.crt --external-cert-file=/root/vaultca.crt
The log file for this installation can be found in /var/log/ipaserver-install.log
CA is not installed yet. To install with an external CA is a two-stage process.
First run the installer with --external-ca.
Comment from jamesmasson (@james-masson) at 2015-12-10 14:30:10
I've created a VM in AWS to demo the issue. centos@54.229.193.39
Drop me your ssh public key, and I'll add you to it.
Workflow is
Comment from edewata (@edewata) at 2015-12-10 20:00:39
I had to create new EPEL 7.1 builds:
Now I'm getting this exception during selftest:
java.security.cert.CertificateException: Invalid certificate: (-8179) Peer's Certificate issuer is not recognized.
at org.mozilla.jss.CryptoManager.verifyCertificateNowNative(Native Method)
at org.mozilla.jss.CryptoManager.verifyCertificate(CryptoManager.java:1548)
at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:851)
...
Perhaps someone more familiar with NSS could explain what this error message means when validating the IPA certificate.
Comment from edewata (@edewata) at 2015-12-11 03:31:39
jmagne did some investigation and found this variable in /etc/sysconfig/pki-tomcat:
NSS_ENABLE_PKIX_VERIFY=1
The file is created after running installation step 1. I'm not sure what this variable actually does, but if you unset the variable, the selftest will no longer fail.
Comment from jamesmasson (@james-masson) at 2015-12-11 12:42:14
/etc/sysconfig/pki-tomcat seems to be updated with NSS_ENABLE_PKIX_VERIFY in step 2
[7/27]: enable PKIX certificate path discovery and validation
However, there's no way to skip NSS_ENABLE_PKIX_VERIFY being added - using
chattr +i /etc/sysconfig/pki-tomcat
causes the installer part2 to blow up.
It's not possible to run the installer part2 a second time, or continue a previously failed part2 install.
Is there a workaround for this?
Comment from jamesmasson (@james-masson) at 2015-12-11 13:16:01
Found it... /usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py
def enable_pkix(self):
installutils.set_directive(self.dogtag_constants.SYSCONFIG_FILE_PATH,
'DONT_NSS_ENABLE_PKIX_VERIFY', '1',
quotes=False, separator='=')
.... rename the variable to something that won't be recognised :-)
Presume this is still being investigated, so eventually my workaround won't be needed?
Comment from jamesmasson (@james-masson) at 2015-12-11 17:33:30
Mozilla's conclusions on their own library are pretty damning:
https://wiki.mozilla.org/SecurityEngineering/Certificate_Verification#libpkix
Comment from edewata (@edewata) at 2015-12-11 17:41:54
Since the variable was added by IPA code, you might want to check with the IPA team the reason behind it, whether it's OK to disable it, and if there's a fix/workaround (e.g. by adding a CLI parameter or by moving the variable to step 1). For PKI, we will fix ticket 850 to display NSS error message during selftest, but I don't believe there is anything to fix for this particular issue.
Comment from jamesmasson (@james-masson) at 2015-12-14 12:40:28
I've raised a new bug against IPA - https://fedorahosted.org/freeipa/ticket/5546
Comment from jamesmasson (@james-masson) at 2015-12-14 19:53:25
IPA 4.2 has just been released for Centos 7 - Unfortunately, we're now back with the same problem.
[14/Dec/2015:16:47:52][localhost-startStop-1]: CertUtils: verifySystemCerts() cert tag=audit_signing
[14/Dec/2015:16:47:52][localhost-startStop-1]: CertUtils: verifySystemCertByTag(audit_signing)
[14/Dec/2015:16:47:52][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(auditSigningCert cert-pki-ca,ObjectSigner)
[14/Dec/2015:16:47:52][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(): calling isCertValid()
[14/Dec/2015:16:47:52][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() passed: auditSigningCert cert-pki-ca
[14/Dec/2015:16:47:52][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Success][CertNickName=auditSigningCert cert-pki-ca] CIMC certificate verification
java.lang.Exception: SystemCertsVerification: system certs verification failure
at com.netscape.cms.selftests.common.SystemCertsVerification.runSelfTest(SystemCertsVerification.java:198)
at com.netscape.cmscore.selftests.SelfTestSubsystem.runSelfTestsAtStartup(SelfTestSubsystem.java:861)
at com.netscape.cmscore.selftests.SelfTestSubsystem.startup(SelfTestSubsystem.java:1797)
at com.netscape.cmscore.apps.CMSEngine.startupSubsystems(CMSEngine.java:1701)
at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1148)
at com.netscape.certsrv.apps.CMS.startup(CMS.java:200)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1602)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:123)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1272)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1197)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1087)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5210)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5493)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:672)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1862)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
NSS_ENABLE_PKIX_VERIFY is not enabled.
Comment from jmagne (@jmagne) at 2015-12-14 21:40:58
Hi:
Has the test VM mentioned above been updated to be able to show this most recent failure? Also, if not possible to view the failure on a host, would it be possible to have the nss db in question zipped up and sent over to look at?
Comment from jamesmasson (@james-masson) at 2015-12-15 18:56:31
Apologies - this was an expired local cert that happened to occur at the same time as the IPA 4.2 upgrade.
A fresh install cycle with a new cert, and the hacked 'cainstance.py' still works.
Comment from ftweedal (@frasertweedale) at 2015-12-16 04:01:02
Having read the comments on this ticket and the FreeIPA 5546 I think we can close this with resolution "invalid" and turn our attention to why FreeIPA ever enabled the envvar and resolving the matter there.
Comment from jamesmasson (@james-masson) at 2017-02-27 14:06:06
Metadata Update from @james-masson:
This issue was migrated from Pagure Issue #1697. Originally filed by jamesmasson (@james-masson) on 2015-11-12 14:57:38:
Following on from this thread - https://www.redhat.com/archives/freeipa-users/2015-October/msg00257.html
Dogtag ( as part of IPA) seems to refuse to use a signed CA cert which passes validation by IPA install (python) and openssl.
1. I use ipa-server-install in two-phase mode to create a CSR.
IPA CSR
2. I sign the CSR, using a private root CA.
ROOT CA CERT
3. I verify the issued IPA Sub-CA cert, against the ROOT ca.
IPA SUB-CA Cert
4. I complete the two-phase IPA install
5. Result
Dogtag hangs on startup (as part of second-phase IPA install )
dogtag debug log
Output of certificate verification tests.