search

NationalSecurityAgency / datawave

DataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.
https://code.nsa.gov/datawave
Apache License 2.0
563 stars 244 forks source link

Error as I migrated to DW version 2.6 #463

Open rqualis-altamiracorp opened 5 years ago

rqualis-altamiracorp commented 5 years ago

I did a new clone of the DW repo and tried to run a test query using the TestUser keys and getting the error below. I am not sure what the cause. I was abel to run the curl statement on a previous clone (2.5)

The curl command I ran was:

curl --silent --write-out 'HTTP_STATUS_CODE:%{http_code};TOTAL_TIME:%{time_total};CONTENT_TYPE:%{content_type}' --insecure --cert /opt/TestScripts/certificates/testUser.pem --key /opt/TestScripts/certificates/testUser.key.rsa --cacert /opt/TestScripts/certificates/testUser.ca --header 'Content-Type: application/x-www-form-urlencoded' --header 'Accept: application/json' -X POST https://5.11.135.14:8443/DataWave/Query/EventQuery/createAndNext

returned:

'HTTP_STATUS_CODE:000;TOTAL_TIME:0.061;CONTENT_TYPE:''HTTP_STATUS_CODE:000;TOTAL_TIME:0.000;CONTENT_TYPE:'ErrorForbidden'HTTP_STATUS_CODE:403;TOTAL_TIME:0.147;CONTENT_TYPE:text/html;charset=UTF-8'

Error log

19-06-13 02:40:04,248 TRACE [datawave.security.auth.DatawaveAuthenticationMechanism] (default task-2) Authenticating with proxiedEntities=null and proxiedIssuers=null 2019-06-13 02:40:04,249 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=Test A. User, OU=Example Developers, O=Example Corp, C=US] into [cn=test a. user, ou=example developers, o=example corp, c=us] 2019-06-13 02:40:04,249 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=EXAMPLE CORP CA, O=Example Corp, C=US] into [cn=example corp ca, o=example corp, c=us] 2019-06-13 02:40:04,249 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=Test A. User, OU=Example Developers, O=Example Corp, C=US] into [cn=test a. user, ou=example developers, o=example corp, c=us] 2019-06-13 02:40:04,249 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=EXAMPLE CORP CA, O=Example Corp, C=US] into [cn=example corp ca, o=example corp, c=us] 2019-06-13 02:40:04,250 TRACE [datawave.security.auth.DatawaveAuthenticationMechanism] (default task-2) Computed credential = DatawaveCredential[userName="cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>", certificate="[ [ .........

]"] 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: initialize(Subject, CallbackHandler, Map, Map) 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: getDatawaveCredential() 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: getDatawaveCredential() 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: validateCredential 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) alias = cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us> 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) identity = null 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: validateCertificateCredential(DatawaveCredential)[datawave.security.login.DatawaveCertVerifier@7d6b41dd] 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) Cert Validation result : true 2019-06-13 02:40:04,252 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: validateCertificateCredential(DatawaveCredential) 2019-06-13 02:40:04,253 DEBUG [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) Failing login due to EJB exception No user found for cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>: datawave.security.authorization.AuthorizationException: No user found for cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>

keith-ratcliffe commented 5 years ago

Seems like the DatawaveUserService impl that you've configured for your deployment is just unaware of this test user identity. Which service are you using? TestDatawaveUserService, RemoteDatawaveUserService, other?

rqualis-altamiracorp commented 5 years ago

I haven't changed anything. I just create the yml from the examples in the sample-config. I wanted to get the bare 2.6 up without changing much. How do I check with *DatawaveUserService the baselined 2.6 is using - just for my edification?

keith-ratcliffe commented 5 years ago

That leads me to believe that you intend to run with the authorization microservice(?) If so, then you should have Wildfly security configured for RemoteDatawaveUserService by overriding any necessary defaults here, in the config for your environment profile (dev.properties, I'm assuming). At minimum, you need security.use.remoteauthservice=true which is mutually exclusive with TestDatawaveUserService (i.e., security.use.testauthservice must be false)

Those settings get written to $WILDFLY_HOME/standalone/configuration/standalone-full.xml and determine which CDI-managed DatawaveUserService impl gets injected at runtime

rqualis-altamiracorp commented 5 years ago
  1. "...authorization microservice(?).." = Yes (If I am correct this is the way Datawave will be going)
  2. "...should have Wildfly security configured for RemoteDatawaveUserService by overriding any necessary defaults here..." = Yes I did override defaults in the default.properties
  3. Yes, the config for my env is dev.properties.
  4. I did not have: security.use.remoteauthservice=true it was set to false. This security.use.testauthservice was set to false. I will set it to true and review the other settings and try again and post the result.

HERE ARE MY SETTINGS IN THAT AREA:

############################
# Server Identity
############################
server.cert.basename=
# Subject DN of the server cert
server.dn=
# Issuer DN of the server cert
issuer.dn=
server.trust.store=

############################
#
# Security Settings
#
############################
# Whether or not to use the remote authorization service
security.use.remoteauthservice=true
# Whether or not to use the test authorization service that loads canned users
security.use.testauthservice=false
# Spring context entry defining the location of test authorization service entries
security.testauthservice.context.entry=
# JSON-encoded DatawaveUser objects to use in the test authorization service
security.testauthservice.users=
# Configuration for the remote DatawaveUser service
#
# Find the host and port of the service using a SRV DNS lookup
security.remoteuserservice.srv.lookup.enabled=false
# The DNS servers to use for the SRV lookup
security.remoteuserservice.srv.lookup.servers=127.0.0.1
# The port on which the DNS server that serves SRV records is listening
security.remoteuserservice.srv.lookup.port=8600
# The scheme to use when connecting to the remote user service
security.remoteuserservice.scheme=https
# The host to connect to (or do a SRV lookup on) for the remote user service
security.remoteuserservice.host=localhost
# The port to connect to (unless a SRV lookup was performed) for the remote user service
security.remoteuserservice.port=8643
rqualis-altamiracorp commented 5 years ago

After redeploying, I am not getting the following error:

AMQ222216: Security problem while creating session: AMQ119031: Unable to validate user from invm:0. Username: DATAWAVE; SSL certificate subject DN: unavailable 04:18:13,637 ERROR [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ154003: Unable to reconnect org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@798f9ea7 destination=topic/AccumuloTableCache destinationType=javax.jms.Topic ack=Auto-acknowledge durable=true clientID=null subscription=ModificationCacheMessageBean user=DATAWAVE password=**** maxSession=50): ActiveMQSecurityException[errorType=SECURITY_EXCEPTION message=AMQ119031: Unable to validate user from invm:0. Username: DATAWAVE; SSL certificate subject DN: unavailable] at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:423) at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:319) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:288) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:237) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1327) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:672) at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:297) at org.apache.activemq.artemis.ra.ActiveMQResourceAdapter.createSession(ActiveMQResourceAdapter.java:1599) at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.setupSession(ActiveMQActivation.java:479) at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.setup(ActiveMQActivation.java:312) at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.reconnect(ActiveMQActivation.java:682) at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation$SetupActivation.run(ActiveMQActivation.java:722) at org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter$WorkWrapper.run(ActiveMQResourceAdapter.java:161) at org.jboss.jca.core.workmanager.WorkWrapper.runWork(WorkWrapper.java:445) at org.jboss.as.connector.services.workmanager.WildflyWorkWrapper.runWork(WildflyWorkWrapper.java:69) at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:223) at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29) at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789) at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44) at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809) at java.lang.Thread.run(Thread.java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:485)

I am using the default setting below:

    <property name="dw.hornetq.system.password" value="secret"/>
rqualis-altamiracorp commented 5 years ago

I still got the same error:

'HTTP_STATUS_CODE:000;TOTAL_TIME:0.012;CONTENT_TYPE:''HTTP_STATUS_CODE:000;TOTAL_TIME:0.000;CONTENT_TYPE:'ErrorForbidden'HTTP_STATUS_CODE:403;TOTAL_TIME:0.158;CONTENT_TYPE:text/html;charset=UTF-8'

keith-ratcliffe commented 5 years ago

Hard to troubleshoot without having more context for your stack trace

keith-ratcliffe commented 5 years ago
rqualis-altamiracorp commented 5 years ago

I am still working on this issue - will debug some more:

New error: 'HTTP_STATUS_CODE:000;TOTAL_TIME:0.028;CONTENT_TYPE:''HTTP_STATUS_CODE:000;TOTAL_TIME:0.000;CONTENT_TYPE:''HTTP_STATUS_CODE:415;TOTAL_TIME:1.064;CONTENT_TYPE:'

this is a The HTTP 415 Unsupported Media Type client error response code indicates that the server refuses to accept the request because the payload format is in an unsupported format.

The format problem might be due to the request's indicated Content-Type or Content-Encoding, or as a result of inspecting the data directly.

log file

02:20:42,726 WARN [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-1) RESTEASY002150: resteasy.scan.providers is no longer supported. Use a servlet 3.0 container and the ResteasyServletInitializer 02:20:42,727 WARN [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-1) RESTEASY002150: resteasy.scan is no longer supported. Use a servlet 3.0 container and the ResteasyServletInitializer 02:20:42,727 WARN [org.jboss.resteasy.resteasy_jaxrs.i18n] (default task-1) RESTEASY002150: resteasy.scan.resources is no longer supported. Use a servlet 3.0 container and the ResteasyServletInitializer

rqualis-altamiracorp commented 5 years ago
  1. setup-wildfly.sh script did not report any errors.
  2. I will review my config files again and try to see if I can figure it out. If not, then I will post more details
rqualis-altamiracorp commented 5 years ago

Hi,

If there is some specific information I can provide you with the give context, please let me know.

I used this command to build: mvn -Pdev,examples -Dservices -Ddeploy -Dtar -Ddist -DskipTests clean install

There were no errors with wildly startup

I changed the curl statement to (I copied your TestUser etc certs to my local dir):

### My Curl command curl --silent --write-out 'HTTP_STATUS_CODE:%{http_code};TOTAL_TIME:%{time_total};CONTENT_TYPE:%{content_type}' --insecure --cert /opt/TestScripts/certificates/spring-boot-starter-datawave/testUser.pem --key /opt/TestScripts/certificates/spring-boot-starter-datawave/testUser.key.rsa --cacert /opt/TestScripts/certificates/spring-boot-starter-datawave/testUser.ca -X POST https://5.11.135.14:8443/DataWave/Query/EventQuery/createAndNext

The logs/accesslog file had

18.209.87.147 - - "cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>" [16/Jun/2019:01:41:34 +0000] "POST /DataWave/Query/EventQuery/createAndNext HTTP/1.1" 415 - 831 default task-1

The Security.log content after running the curl command above is:

2019-06-16 01:50:55,217 TRACE [datawave.security.auth.DatawaveAuthenticationMechanism] (default task-2) Authenticating with proxiedEntities=null and proxiedIssuers=null 2019-06-16 01:50:55,217 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=Test A. User, OU=Example Developers, O=Example Corp, C=US] into [cn=test a. user, ou=example developers, o=example corp, c=us] 2019-06-16 01:50:55,217 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=EXAMPLE CORP CA, O=Example Corp, C=US] into [cn=example corp ca, o=example corp, c=us] 2019-06-16 01:50:55,217 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=Test A. User, OU=Example Developers, O=Example Corp, C=US] into [cn=test a. user, ou=example developers, o=example corp, c=us] 2019-06-16 01:50:55,217 TRACE [datawave.security.util.ProxiedEntityUtils] (default task-2) Normalized [CN=EXAMPLE CORP CA, O=Example Corp, C=US] into [cn=example corp ca, o=example corp, c=us] 2019-06-16 01:50:55,219 TRACE [datawave.security.auth.DatawaveAuthenticationMechanism] (default task-2) Computed credential = DatawaveCredential[userName="cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>", certificate="[ [ Version: V3 Subject: CN=Test A. User, OU=Example Developers, O=Example Corp, C=US Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

.......... Validity: [From: Fri Feb 02 17:22:00 UTC 2018, To: Wed Feb 01 17:22:00 UTC 2023] Issuer: CN=EXAMPLE CORP CA, O=Example Corp, C=US ..........

[2]: ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[ CA:false PathLen: undefined ]

[3]: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth ]

[4]: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Key_Encipherment ]

[5]: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 70 00 40 3C 41 AB 2F 00 17 11 99 02 CC 36 9A 1B p.@<A./......6.. 0010: 95 45 A4 D5 .E.. .......... ]"] 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: initialize(Subject, CallbackHandler, Map, Map) 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: getDatawaveCredential() 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: getDatawaveCredential() 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: validateCredential 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) alias = cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us> 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) identity = null 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) enter: validateCertificateCredential(DatawaveCredential)[datawave.security.login.DatawaveCertVerifier@5d6d9713] 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) Cert Validation result : true 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: validateCertificateCredential(DatawaveCredential) 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: validateCredential 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) User 'DatawavePrincipal{name='cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>', proxiedUsers=[DatawaveUser{name='cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>', userType=USER, auths=[PUBLIC, PRIVATE, FOO, BAR, DEF, A, B, C, D, E, F, G, H, I, DW_USER, DW_SERV, DW_ADMIN, JBOSS_ADMIN], roles=[AuthorizedUser, Administrator, JBossAdministrator], creationTime=-1}]}' authenticated, loginOk=true 2019-06-16 01:50:55,221 DEBUG [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) exit: login() 2019-06-16 01:50:55,221 TRACE [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) Committing login for DatawavePrincipal{name='cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>', proxiedUsers=[DatawaveUser{name='cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us>', userType=USER, auths=[PUBLIC, PRIVATE, FOO, BAR, DEF, A, B, C, D, E, F, G, H, I, DW_USER, DW_SERV, DW_ADMIN, JBOSS_ADMIN], roles=[AuthorizedUser, Administrator, JBossAdministrator], creationTime=-1}]}@1046823876. loginOk=true 2019-06-16 01:50:55,222 DEBUG [datawave.security.login.DatawavePrincipalLoginModule] (default task-2) [3] Groups for cn=test a. user, ou=example developers, o=example corp, c=us<cn=example corp ca, o=example corp, c=us> { Administrator : AuthorizedUser : JBossAdministrator }

keith-ratcliffe commented 5 years ago

@rqualis-altamiracorp , any updates with this issue? can it be closed?

rqualis-altamiracorp commented 5 years ago

I had to switch gear to close off a different task that is taking some time.