search

marklogic / java-client-api

Java client for the MarkLogic enterprise NoSQL database
https://docs.marklogic.com/guide/java
Apache License 2.0
59 stars 72 forks source link

Marklogic java client Kerberos support limits using the TGT Cache only #995

Closed mwarnes closed 4 years ago

mwarnes commented 6 years ago

The Kerberos support documented for the ML Java API at the below link requires that the user issue “kinit” to create and cache a TGT with the Kerberos Key Distribution Center. This restricts any Java application to be run interactively by the user ``

https://docs.marklogic.com/guide/java/intro#id_70914

If the requirement is to run the Java application as a scheduled task or unattended then no TGT will be present in the cache and the application will fail with the following exception.

Exception in thread "main" com.marklogic.client.FailedRequestException: Unable to obtain password from user

The usual method to deal with this, which does not require running “kinit” manually is to use a KeyTab instead. We have a KB article which documents how to do this with the XCC/J library (https://help.marklogic.com/Knowledgebase/Article/View/455/15/authenticating-xccj-applications-using-a-kerberos-keytab )but this will not work with the Java API code as there is an explicit call to reset Kerberos configuration within the HTTPKerberosAuthInterceptor.KerberosLoginConfiguration class

options.put("refreshKrb5Config", "true");

I’ve written a small update to this class that will allow using a KeyTab if you set the system property “com.marklogic.krb5.keytab.path” and specify a valid Kerberos KeyTab path to use, e.g.

// private DatabaseClient client;
private static String appServerHostName = "mlsrv.marklogic.local";
private static String kdcPrincipalUser = "mluser1@MLKRB.LOCAL";
private static int appServerHostPort = 8011;
private static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());

public static void main(String[] args) {

    System.setProperty("com.marklogic.krb5.keytab.path","/tmp/user.keytab");
    LOG.info("trying to connect using the Kerberos Auth Context");
    DatabaseClient client = DatabaseClientFactory.newClient(appServerHostName,
            appServerHostPort, new DatabaseClientFactory.KerberosAuthContext(kdcPrincipalUser));
    client.newServerEval().xquery("1+1");
}

Output with Debug enabled showing Keytab being used if keytab property is set.

[user1@mlsrv ~]# java -jar ./KerberosAPI.jar SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /tmp/user.keytab refreshKrb5Config is false principal is mluser1@MLKRB.LOCAL tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is mluser1@MLKRB.LOCAL Will use keytab Commit Succeeded

If no keytab property is set, then the Kerberos functionality remains as it is and the TGT is acquired from the cache if available following kinit.

[user1@mlsrv ~]# java -jar ./KerberosAPI.jar SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Debug is true storeKey false useTicketCache true useKeyTab false doNotPrompt true ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is true principal is mluser2@MLKRB.LOCAL tryFirstPass is false useFirstPass is false storePass is false clearPass is false Refreshing Kerberos configuration Acquire TGT from Cache Principal is mluser2@MLKRB.LOCAL Commit Succeeded

The changes to the HTTPKerberosAuthInterceptor.KerberosLoginConfiguration class are small, only the lines highlighted below so I don’t know if you want to consider adding them as is or if you would like me to create a Pull Request on the Java API GitHub site?

      options.put("refreshKrb5Config", "true");
      if (System.getProperty("com.marklogic.krb5.keytab.path") == null) {
        options.put("useTicketCache", "true");
        options.put("doNotPrompt", "true");
      } else {
        options.put("useKeyTab", "true");
        options.put("keyTab", System.getProperty("com.marklogic.krb5.keytab.path"));
        options.put("storeKey", "true");
      }
ehennum commented 6 years ago

It might be best to configure the use of Kerberos via the KerberosAuthContext interface instead of using custom environment variables, taking into account:

https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html

https://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html

mwarnes commented 6 years ago

I added a Pull request with a rewrite of my original modification, per Erik previous comment I have modified the KerberosAuthContext interface to allow configuring all options supported by the Krb5LoginModule.

https://docs.oracle.com/javase/8/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html

ehennum commented 6 years ago

Hi, Martin -- this enhancement seems to be heading in a good direction. I'm wondering whether:

That way, we would have separation of concerns between defining and using the configuration.

KerberosAuthContext could have immutable state by saving the result of KerberosConfig.toOptions()

What do you think?

mwarnes commented 6 years ago

Hi Erik,

Yes, that sounds reasonable, I can make those changes and submit another Pull request.

Also, I noticed in HTTPKerberosAuthInterceptor with the call to create the LoginContext, parameter 3 (CallBackHandler) is null.

LoginContext lc = new LoginContext("Krb5LoginContext", subject, null,
    (krbConfig != null) ? new KerberosLoginConfiguration(krbConfig) : new KerberosLoginConfiguration());

This means that we can only support 1-3 of the Login methods supported by the Krb5LoginContext

  1. ticket cache
  2. keytab
  3. shared state
  4. user prompt

Do you think it's worthwhile adding a callback handler to the code for completeness to handle the odd case where it may fall through and prompt the user for a Kerberos principle and password?

Rgds .. Martin

ehennum commented 6 years ago

Hi, Martin: Thanks for refining the solution.

Regarding the callback, I'm wondering whether, given the most typical use of the Java API is on the middle tier, prompting the user for the Kerberos principle and password during the request wouldn't come up very often.

If that's correct, should we opt for simplicity for now?

mwarnes commented 6 years ago

Hi Erik,

I've made the following changes:

Moved KerberosConfif to a nested static class within DatabaseClientFactory Changed KerberosConfig setters to with*() fluents that return KerberosConfig Added toOptions() method that returns the map from KerberosConfig Added a KerberosAuthContext() constructor that takes a KerberosConfig object

This separates the creation of the KerberosConfig object and its use in a KerberosAuthContext as per your suggestion.

Example

            KerberosConfig krbConfig = new DatabaseClientFactory.KerberosConfig().withPrincipal(kdcPrincipalUser)
                    .withUseKeyTab(true)
                    .withKeyTab("/tmp/user.keytab")
                    .withDebug(true);

            DatabaseClient client = DatabaseClientFactory.newClient(appServerHostName,
                    appServerHostPort,
                    new DatabaseClientFactory.KerberosAuthContext(krbConfig));

The original method of passing a Principal string to KerberosAuthContext remains unchanged.

I've created a new Pull Request for review

Rgds .. Martin

ehennum commented 6 years ago

Thanks, @mwarnes , for the enhanced pull request.

My only lingering concern is the mutability of KerberosConfig after it is supplied to the newClient() factory.

I'm wondering whether the API should keep the result of Collections.unmodifiableMap(KerberosConfig.toOptions()) in its internal state instead of the KerberosConfig input.

If you have concerns or other ideas, let's chat about it next week.

ehennum commented 6 years ago

@mwarnes , now that @anu3990 has merged the pull request (thanks!), do you have any suggestions to @georgeajit for a straightforward sanity check on the configuration changes to use as a regress test?

georgeajit commented 5 years ago

Contents of the email sent attached here: @mwarnes I was validating the fix for https://github.com/marklogic/java-client-api/issues/995 .We in QA have a Jenkins job that uses cached TGT to validate Java client user logins and carryout other tests. I modified that Jenkins job so that credentials are not cached. On running the following the snippet, we always get “Unable to obtain password from user”

public void setUp() throws KeyManagementException, NoSuchAlgorithmException, Exception { SSLContext sslcontext = null; // Modify default location if needed for services.keytab file. keytabFile = System.getProperty("keytabFile", "/space/Jenkins/workspace/services.keytab"); principal = System.getProperty("principal");

System.out.println("Location of key tab file is " + keytabFile);

if (keytabFile == null || !(new File(keytabFile).exists())) {
    fail("Key tab file does not exist or is invalid");
}

if (IsSecurityEnabled()) {
  sslcontext = getSslContext();

  KerberosConfig krbConfig = new DatabaseClientFactory.KerberosConfig().withPrincipal(principal)
        .withUseKeyTab(true)
        .withDoNotPrompt(true)
        .withStoreKey(true)
        .withDebug(true)
        .withKeyTab(keytabFile);
  client = DatabaseClientFactory.newClient(
          appServerHostName, appServerHostPort,
          new DatabaseClientFactory.KerberosAuthContext(krbConfig).withSSLContext(sslcontext));
} else {
    /*Pass this file's location for the gradle (thru Jenkins job) 
      QA functional test project's build.gradle file has << systemProperty "keytabFile", System.getProperty("keytabFile") >>
      On gradle command line use the following syntax to pass in the location of the keytab file:
      ./gradlew marklogic-client-api-functionaltests:test -DkeytabFile=/space/Jenkins/workspace/services.keytab -Dprincipal=user2  .....other options
      */    
    KerberosConfig krbConfig = new DatabaseClientFactory.KerberosConfig().withPrincipal(principal)
            .withUseKeyTab(true)
            .withDoNotPrompt(true)
            .withStoreKey(true)
            .withKeyTab(keytabFile);
    System.out.println("Password of key tab file is " + krbConfig.getStorePass());
    System.out.println("Principle of key tab file is " + krbConfig.getPrincipal());
    client = DatabaseClientFactory.newClient(appServerHostName,
            appServerHostPort, new DatabaseClientFactory.KerberosAuthContext(krbConfig));
}

}

For the time being IsSecurityEnabled() is false. Gradle is used to run the test and parameters are passed from command line such as (from Jenkins Job) ./gradlew -Dkotlin.compiler.execution.strategy="in-process" -Dorg.gradle.daemon=false marklogic-client-api-functionaltests:test -Dprincipal=MLTEST1.LOCAL -DkeytabFile=//space//Jenkins//workspace//services1.keytab --tests "KerberosFromFile"

I tried to pass user2@MLTEST1.LOCAL, machineName@ MLTEST1.LOCAL etc., as the principal, yet the results are the same.

This is the Jenkins Job I am referring to https://jenkins.marklogic.com/view/Java%20API/job/Java-client-api-develop-kerberos-from-file-9.0.7.1/ which throws the error The Jenkins Job that uses cache is at https://jenkins.marklogic.com/view/Java%20API/job/Java-client-api-develop-kerberos-9.0.7.1/. This job runs fine.

The test class in develop branch is called TestDatabaseClientKerberosFromFile @

Maybe I am missing something.

Could you please help?

Thank you. Ajit.

com.marklogic.client.FailedRequestException: Unable to obtain password from user

at com.marklogic.client.impl.HTTPKerberosAuthInterceptor.<init>(HTTPKerberosAuthInterceptor.java:50)
at com.marklogic.client.impl.OkHttpServices.connect(OkHttpServices.java:301)
at com.marklogic.client.impl.OkHttpServices.connect(OkHttpServices.java:270)
at com.marklogic.client.DatabaseClientFactory.newClient(DatabaseClientFactory.java:1082)
at com.marklogic.client.DatabaseClientFactory.newClient(DatabaseClientFactory.java:963)
at com.marklogic.client.functionaltest.TestDatabaseClientKerberosFromFile.setUp(TestDatabaseClientKerberosFromFile.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)

Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at com.marklogic.client.impl.HTTPKerberosAuthInterceptor.buildSubjectCredentials(HTTPKerberosAuthInterceptor.java:94)
at com.marklogic.client.impl.HTTPKerberosAuthInterceptor.<init>(HTTPKerberosAuthInterceptor.java:48
georgeajit commented 5 years ago

Jenkins script to get SPN 

# Steps for Kerberos auth
mkdir -p -m 777 $TMP_DIR/extsec

# Destroy any existing ticket granting ticket using kdestroy
cd $MY_QA_HOME
./destroy-tgt.sh MLTEST1.LOCAL $TMP_DIR/extsec/kdestroy.log $TMP_DIR/extsec/kdestroy.err

# Get ticket granting ticket using kinit
./get-tgt.sh user2 Welcome123 $TMP_DIR/extsec/user2pw $TMP_DIR/extsec/klist.log $TMP_DIR/extsec/klist.err

# Used from SO post : https://stackoverflow.com/questions/37454308/script-kerberos-ktutil-to-make-keytabs
# Copy user principal name into the same keytab file.
export user="builder"
export pass="Welcome123"

printf "%b" "addent -password -p $user -k 1 -e rc4-hmac\n$pass\nwrite_kt /space/Jenkins/workspace/services.keytab" | ktutil

printf "%b" "read_kt /space/Jenkins/workspace/services.keytab\nlist" | ktutil

# Copy keytab file to marklogic data directory
sudo /usr/local/sbin/mladmin copy /space/Jenkins/workspace/services.keytab $MLD_HOME/services.keytab > $TMP_DIR/extsec/copykeytab.log 2> $TMP_DIR/extsec/copykeytab.err

Results of SPN listed.

ktutil:  read_kt services.keytab
ktutil:  list
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    4 HTTP/java-client-api-2-1.marklogic.com@MLTEST1.LOCAL
   2    1                    builder@MLTEST1.LOCAL

The tests are passing once user's PN are in services.keytab file in MarkLogic Server's data directory and the same file is used in client.

...
...
// Modify default location if needed for services.keytab file. Jenkins job passes in the services.keytab file.
    keytabFile = System.getProperty("keytabFile");
    principal = System.getProperty("principal", kdcPrincipalUser);

     System.out.println("Location of key tab file is " + keytabFile);

     if (keytabFile == null) {
        fail("Key tab file value from Gradle is null / incorrect");
     }
     if (!(new File(keytabFile).exists())) {
        fail("Key tab file does not exist or is invalid");
     }

...
...

/*Pass this file's location for the gradle (thru Jenkins job) 
          QA functional test project's build.gradle file has << systemProperty "keytabFile", System.getProperty("keytabFile") >>
          On gradle command line use the following syntax to pass in the location of the keytab file:
          ./gradlew marklogic-client-api-functionaltests:test -DkeytabFile=$WORKSPACE/java-client-api/services.keytab -Dprincipal=builder  .....other options
*/   

KerberosConfig krbConfig = new DatabaseClientFactory.KerberosConfig().withPrincipal(principal)
                .withUseKeyTab(true)
                .withDoNotPrompt(true)
                .withStoreKey(true)
                .withKeyTab(keytabFile);
        System.out.println("Password of key tab file is " + krbConfig.getStorePass());
        System.out.println("Principal of key tab file is " + krbConfig.getPrincipal());
        client = DatabaseClientFactory.newClient(appServerHostName,
                appServerHostPort, new DatabaseClientFactory.KerberosAuthContext(krbConfig));