search

ESGF / esgf-installer

ESGF P2P Node Installer
https://esgf.llnl.gov/
Other
21 stars 21 forks source link

Restrcited download testing leads to SAML signature error #647

Closed nathanlcarlson closed 5 years ago

nathanlcarlson commented 5 years ago

After a 3.0 install it is desired to test if access control on published data. To do this the following steps were taken.

  1. Publish sample cmip5 data to the fresh install.
  2. Create a user on the production esgf-node.llnl.gov and subscribe to the CMIP5 Research group.
  3. Add the esgf-node.llnl.gov to the whitelist on the fresh install.
  4. Attempt to download a sample .nc file via the HTTPServer using the OpenId given to the user created on the produciton node.

This is where the following error is occuring as reported in /esg/content/thredds/logs/threddsServlet.log. 

2018-10-18T08:59:52.784 -0700 [  62369989][        ] DEBUG - org.opensaml.xml.signature.impl.BaseSignatureTrustEngine - Signature validation using candidate validation credential failed
org.opensaml.xml.validation.ValidationException: Unable to evaluate key against signature
        at org.opensaml.xml.signature.SignatureValidator.validate(SignatureValidator.java:73) ~[xmltooling-1.2.2.jar:?]
        at org.opensaml.xml.signature.impl.BaseSignatureTrustEngine.verifySignature(BaseSignatureTrustEngine.java:141) [xmltooling-1.2.2.jar:?]
        at org.opensaml.xml.signature.impl.ExplicitKeySignatureTrustEngine.validate(ExplicitKeySignatureTrustEngine.java:109) [xmltooling-1.2.2.jar:?]
        at esg.security.common.SAMLBuilder.validateSignature(SAMLBuilder.java:512) [esgf-security-2.7.18.jar:?]
        at esg.security.authn.service.impl.SAMLAuthenticationStatementFacadeImpl.validateAssertion(SAMLAuthenticationStatementFacadeImpl.java:199) [esgf-security-2.7.18.jar:?]
        at esg.security.authn.service.impl.SAMLAuthenticationStatementFacadeImpl.getAuthentication(SAMLAuthenticationStatementFacadeImpl.java:134) [esgf-security-2.7.18.jar:?]
        at esg.orp.app.AuthenticationFilter.attemptValidation(AuthenticationFilter.java:118) [esg-orp-2.9.10.jar:?]
        at esg.orp.app.AccessControlFilterTemplate.doFilter(AccessControlFilterTemplate.java:63) [esg-orp-2.9.10.jar:?]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.20]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.20]
        at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.6.2.jar:2.6.2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [catalina.jar:8.5.20]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [catalina.jar:8.5.20]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [catalina.jar:8.5.20]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [catalina.jar:8.5.20]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [catalina.jar:8.5.20]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [catalina.jar:8.5.20]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) [catalina.jar:8.5.20]
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) [catalina.jar:8.5.20]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [catalina.jar:8.5.20]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [catalina.jar:8.5.20]
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486) [tomcat-coyote.jar:8.5.20]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-coyote.jar:8.5.20]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-coyote.jar:8.5.20]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457) [tomcat-coyote.jar:8.5.20]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-coyote.jar:8.5.20]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.5.20]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: org.apache.xml.security.signature.XMLSignatureException: Signature length not correct: got 256 but was expecting 512
        at org.apache.xml.security.signature.XMLSignature.checkSignatureValue(Unknown Source) ~[xmlsec-1.4.2.jar:?]
        at org.opensaml.xml.signature.SignatureValidator.validate(SignatureValidator.java:68) ~[xmltooling-1.2.2.jar:?]
        ... 29 more
2018-10-18T08:59:52.785 -0700 [  62369990][        ] ERROR - org.opensaml.xml.signature.impl.ExplicitKeySignatureTrustEngine - Failed to verify signature using either KeyInfo-derived or directly trusted credentials
2018-10-18T08:59:52.785 -0700 [  62369990][        ] DEBUG - esg.security.common.SAMLBuilder - Validated assertion signature: result=false
nathanlcarlson commented 5 years ago

Log statements prior to the error:

2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.opensaml.xml.signature.impl.SignatureUnmarshaller - Starting to unmarshall Apache XML-Security-based SignatureImpl element
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.opensaml.xml.signature.impl.SignatureUnmarshaller - Constructing Apache XMLSignature object
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.apache.xml.security.utils.ElementProxy - setElement("ds:Signature", "")
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.apache.xml.security.utils.ElementProxy - setElement("ds:SignedInfo", "")
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.apache.xml.security.utils.ElementProxy - setElement("ds:SignatureMethod", "")
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.opensaml.xml.signature.impl.SignatureUnmarshaller - Adding canonicalization and signing algorithms, and HMAC output length to Signature
2018-10-18T08:59:52.779 -0700 [  62369984][        ] DEBUG - org.opensaml.xml.signature.impl.SignatureUnmarshaller - Adding KeyInfo to Signature
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.apache.xml.security.algorithms.JCEMapper - Request for URI http://www.w3.org/2000/09/xmldsig#rsa-sha1
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.credential.criteria.EvaluableCredentialCriteriaRegistry - Registry located evaluable criteria class org.opensaml.xml.security.credential.criteria.EvaluableKeyAlgorithmCredentialCriteria for criteria class org.opensaml.xml.securit
y.criteria.KeyAlgorithmCriteria
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.credential.criteria.EvaluableCredentialCriteriaRegistry - Registry located evaluable criteria class org.opensaml.xml.security.credential.criteria.EvaluableEntityIDCredentialCriteria for criteria class org.opensaml.xml.security.cr
iteria.EntityIDCriteria
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.credential.criteria.EvaluableCredentialCriteriaRegistry - Registry located evaluable criteria class org.opensaml.xml.security.credential.criteria.EvaluableUsageCredentialCriteria for criteria class org.opensaml.xml.security.crite
ria.UsageCriteria
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.signature.impl.BaseSignatureTrustEngine - Attempting to verify signature and establish trust using KeyInfo-derived credentials
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.keyinfo.BasicProviderKeyInfoCredentialResolver - Found 0 key names: []
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.keyinfo.BasicProviderKeyInfoCredentialResolver - No credentials were found, calling empty credentials post-processing hook
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.keyinfo.BasicProviderKeyInfoCredentialResolver - A total of 0 credentials were resolved
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.security.credential.criteria.EvaluableCredentialCriteriaRegistry - Registry could not locate evaluable criteria for criteria class org.opensaml.xml.security.keyinfo.KeyInfoCriteria
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.signature.impl.BaseSignatureTrustEngine - Failed to verify signature and/or establish trust using any KeyInfo-derived credentials
2018-10-18T08:59:52.781 -0700 [  62369986][        ] DEBUG - org.opensaml.xml.signature.impl.ExplicitKeySignatureTrustEngine - Attempting to verify signature using trusted credentials
2018-10-18T08:59:52.782 -0700 [  62369987][        ] INFO  - org.opensaml.xml.security.credential.criteria.EvaluableEntityIDCredentialCriteria - Could not evaluate criteria, credential contained no entity ID
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.opensaml.xml.signature.SignatureValidator - Attempting to validate signature using key from supplied credential
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.opensaml.xml.signature.SignatureValidator - Creating XMLSignature object
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.opensaml.xml.signature.SignatureValidator - Validating signature with signature algorithm URI: http://www.w3.org/2000/09/xmldsig#rsa-sha1
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.opensaml.xml.signature.SignatureValidator - Validation credential key algorithm 'RSA', key instance class 'sun.security.rsa.RSAPublicKeyImpl'
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.apache.xml.security.signature.XMLSignature - SignatureMethodURI = http://www.w3.org/2000/09/xmldsig#rsa-sha1
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.apache.xml.security.signature.XMLSignature - jceSigAlgorithm    = SHA1withRSA
2018-10-18T08:59:52.782 -0700 [  62369987][        ] DEBUG - org.apache.xml.security.signature.XMLSignature - jceSigProvider     = SunRsaSign
2018-10-18T08:59:52.783 -0700 [  62369988][        ] DEBUG - org.apache.xml.security.signature.XMLSignature - PublicKey = Sun RSA public key, 4096 bits
...
2018-10-18T08:59:52.784 -0700 [  62369989][        ] DEBUG - org.apache.xml.security.utils.SignerOutputStream - Canonicalized SignedInfo:
2018-10-18T08:59:52.784 -0700 [  62369989][        ] DEBUG - org.apache.xml.security.utils.SignerOutputStream - <ds:SignedInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
...
nathanlcarlson commented 5 years ago

It appears there is a misstep in the re-implementation of the --install-keypair command. This was discovered by using the master branch esg-node --install-keypair and being able to successfully download the sample cmip5 data. I will review the --install-keypair steps and attempt to determine the problem point.

nathanlcarlson commented 5 years ago

A strange error message has been appearing in Jenkins auto-installs

Extra  DEBUG    | esg_functions.py - 921 - call_binary - 10/16/2018 03:04:58 PM - binary_name: /esg/tools/idptools/bin/extkeytool
  DEBUG    | esg_functions.py - 922 - call_binary - 10/16/2018 03:04:58 PM - arguments: ['-importkey', '-keystore', '/esg/config/tomcat/keystore-tomcat', '-alias', 'my_esgf_node', '-storepass', 'EsgfLLNL', '-keypass', 'EsgfLLNL', '-keyfile', '/esg/tools/idptools/key.der', '-certfile', '/esg/tools/idptools/cert.bundle', '-provider', 'org.bouncycastle.jce.provider.BouncyCastleProvider']
extkeytool usage:
-exportkey      [-v] [-rfc] [-alias <alias>] [-keystore <keystore>] 
         [-storepass <storepass>] [-storetype <storetype>]
         [-keypass <keypass>] [-provider <provider_class_name>] 
         [-file <output_file>] 

-importkey      [-v] [-secret] [-alias <alias>] [-keyfile <key_file>]
         [-keystore <keystore>] [-storepass <storepass>]
         [-storetype <storetype>] [-keypass <keypass>] 
         [-provider <provider_class_name>] [-certfile <cert_file>] 
         [-algorithm <key_algorithm>]
EXTKEYTOOL_HOME: /esg/tools/idptools
Could not import key: key alias (my_esgf_node) already exists
Cannot Perform Operation: Could not import key: key alias (my_esgf_node) already exists

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: my_esgf_node
...{Key Details}...

Initially I ignored it because of 

Could not import key: key alias (my_esgf_node) already exists

This implied what needed to be done was already done. Looking closely today I discovered that the key being put there is 2048 bit, whereas the one that is supposed to be there is 4096 bit. It appears that the print statement "Creating Empty Keystore" is incorrect. It is in fact creating a keystore with this 2048 bit key with alias "my_esgf_node" in it. This blocks the import of the real key because it is given the matching alias "my_esgf_node". I don't know how this works in 2.x and not 3.x.

There is more than one way to have Java KeyStore (JKS) with the required keys. As a solution, which has been manually test, I found the following.

A pkcs12 keystore can be generated and imported into a JKS.

openssl pkcs12 -export -in {hostcert} -inkey {hostkey} -nodes -passout pass:{tempkeystore_pass} -certfile {cachain} -out tempkeystore.p12

Then the import

keytool -importkeystore -srckeystore tempkeystore.p12 -srcstoretype pkcs12 -destkeystore /esg/config/tomcat/keystore-tomcat -deststoretype JKS -srcstorepass {tempkeystore_pass} -deststorepass {JKS pass}

This will be implemented and tested to verify via a fresh install.

nathanlcarlson commented 5 years ago

More manual testing indicates that the following flags are needed on the import

-srcalias 1 -destalias my_esgf_node

1 is the alias assigned to the key in the pkcs12 keystore and my_esgf_node is the currently used alias by default in the JKS.

nathanlcarlson commented 5 years ago

I discovered there is a pyjks module (Java KeyStore) and this, in conjunction with pyopenssl, are now being used to solve this issue.

nathanlcarlson commented 5 years ago

A simpler keystore fix was made in #654. As noted there more can be done to improve the JKS management going on, but it will have to be an incremental process.