Open haunted-pancakes opened 1 year ago
Thanks @fhussonnois! I tried the pre-release build and have uncovered a new issue with Jikkou not finding the default krb5.conf file. I tried specifying it as an env variable but got the same results.
$ KRB5_TRACE=/dev/stdout KRB5_CONFIG=/etc/krb5.conf jikkou health get kafkabroker
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at io.streamthoughts.jikkou.kafka.internals.admin.DefaultAdminClientFactory.createAdminClient(DefaultAdminClientFactory.java:60)
at io.streamthoughts.jikkou.kafka.internals.admin.AdminClientContext.getAdminClient(AdminClientContext.java:171)
at io.streamthoughts.jikkou.kafka.health.KafkaBrokerHealthIndicator.getHealth(KafkaBrokerHealthIndicator.java:81)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:100)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:40)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at io.streamthoughts.jikkou.client.Jikkou$1.execute(Jikkou.java:153)
at picocli.CommandLine.execute(CommandLine.java:2170)
at io.streamthoughts.jikkou.client.Jikkou.execute(Jikkou.java:136)
at io.streamthoughts.jikkou.client.Jikkou.main(Jikkou.java:119)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:522)
... 18 more
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:789)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base@17.0.8/java.security.AccessController.executePrivileged(AccessController.java:144)
at java.base@17.0.8/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base@17.0.8/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
... 22 more
Caused by: KrbException: Cannot locate KDC
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1262)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:217)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:199)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:345)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:761)
... 35 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm LOCAL
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCFromDNS(Config.java:1359)
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1235)
... 40 more
As a sanity check, here is the krb5.conf in question, and also an example of doing a kinit without any issue.
# /etc/krb5.conf
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 10h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = LOCAL
default_ccache_name = KEYRING:persistent:%{uid}
udp_preference_limit = 1
[realms]
LOCAL = {
kdc = LOCAL
admin = LOCAL
}
[domain_realm]
.local = LOCAL
local = LOCAL
$ kinit admin@LOCAL -kt /usr/local/jikkou/admin.keytab
$ klist
Ticket cache: KCM:900
Default principal: admin@LOCAL
Valid starting Expires Service principal
10/06/2023 15:20:48 10/07/2023 01:20:48 krbtgt/LOCAL@LOCAL
renew until 10/13/2023 15:20:48
Hi @haunted-pancakes, I'm not sure if this error is coming from Jikkou. Could run Jikkou with system property
-Dsun.security.krb5.debug=true
to enable debug for Kerberos ?
@fhussonnois What's the best way to pass the option in? I tried a few environment variables but am not familiar with Java 17. Thank you!
You can pass this option to your jikkou command: jikkou -Dsun.security.krb5.debug=true <subcommand> ...
I've passed in the option but the result looks the same
$ jikkou -Dsun.security.krb5.debug=true health get kafkabroker
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at io.streamthoughts.jikkou.kafka.internals.admin.DefaultAdminClientFactory.createAdminClient(DefaultAdminClientFactory.java:60)
at io.streamthoughts.jikkou.kafka.internals.admin.AdminClientContext.getAdminClient(AdminClientContext.java:171)
at io.streamthoughts.jikkou.kafka.health.KafkaBrokerHealthIndicator.getHealth(KafkaBrokerHealthIndicator.java:81)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:100)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:40)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at io.streamthoughts.jikkou.client.Jikkou$1.execute(Jikkou.java:153)
at picocli.CommandLine.execute(CommandLine.java:2170)
at io.streamthoughts.jikkou.client.Jikkou.execute(Jikkou.java:136)
at io.streamthoughts.jikkou.client.Jikkou.main(Jikkou.java:119)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:522)
... 18 more
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:789)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base@17.0.8/java.security.AccessController.executePrivileged(AccessController.java:144)
at java.base@17.0.8/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base@17.0.8/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
... 22 more
Caused by: KrbException: Cannot locate KDC
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1262)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:217)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:199)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:345)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:761)
... 35 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm LOCAL
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCFromDNS(Config.java:1359)
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1235)
... 40 more
I then started looking at Oracle's documentation: https://docs.oracle.com/en/java/javase/17/security/troubleshooting.html#GUID-2087ADBA-6C36-43D5-8841-C79FCB4F5FBE
I saw that I can explicitly set the realm and kdc as options as well, so just for a sanity check of what the values should be:
$ KRB5_TRACE=/dev/stdout kinit admin@LOCAL -kt /usr/local/jikkou/admin.keytab
...
[2023703] 1697039506.946350: Getting initial credentials for admin@LOCAL
[2023948] 1697039628.450825: Sending request (289 bytes) to LOCAL
[2023948] 1697039628.450826: Resolving hostname LOCAL
[2023948] 1697039628.450827: Initiating TCP connection to stream 192.168.100.50:88
[2023948] 1697039628.450828: Sending TCP request to stream 192.168.100.50:88
[2023948] 1697039628.450829: Received answer (1693 bytes) from stream 192.168.100.50:88
[2023948] 1697039628.450830: Terminating TCP connection to stream 192.168.100.50:88
[2023948] 1697039628.450831: Sending DNS URI query for _kerberos.LOCAL.
[2023948] 1697039628.450833: Sending DNS SRV query for _kerberos-master._udp.LOCAL.
[2023948] 1697039628.450834: Sending DNS SRV query for _kerberos-master._tcp.LOCAL.
[2023948] 1697039628.450844: Initializing MEMORY:04M5Faa with default princ admin@LOCAL
[2023948] 1697039628.450845: Storing config in MEMORY:04M5Faa for krbtgt/LOCAL@LOCAL: pa_type: 2
[2023948] 1697039628.450851: Storing admin@LOCAL -> krbtgt/LOCAL@LOCAL in KCM:0
But I ended up with the same result when passing it in as well
# Also tried -Djava.security.krb5.conf=/etc/krb5.conf with the same result
$ jikkou -Dsun.security.krb5.debug=true -Djava.security.krb5.realm=LOCAL -Djava.security.krb5.kdc=192.168.100.50 health get kafkabroker
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:144)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at io.streamthoughts.jikkou.kafka.internals.admin.DefaultAdminClientFactory.createAdminClient(DefaultAdminClientFactory.java:60)
at io.streamthoughts.jikkou.kafka.internals.admin.AdminClientContext.getAdminClient(AdminClientContext.java:171)
at io.streamthoughts.jikkou.kafka.health.KafkaBrokerHealthIndicator.getHealth(KafkaBrokerHealthIndicator.java:81)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:100)
at io.streamthoughts.jikkou.client.command.health.GetHealthCommand.call(GetHealthCommand.java:40)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at io.streamthoughts.jikkou.client.Jikkou$1.execute(Jikkou.java:153)
at picocli.CommandLine.execute(CommandLine.java:2170)
at io.streamthoughts.jikkou.client.Jikkou.execute(Jikkou.java:136)
at io.streamthoughts.jikkou.client.Jikkou.main(Jikkou.java:119)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:522)
... 18 more
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:789)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base@17.0.8/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base@17.0.8/java.security.AccessController.executePrivileged(AccessController.java:144)
at java.base@17.0.8/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base@17.0.8/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base@17.0.8/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:103)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
... 22 more
Caused by: KrbException: Cannot locate KDC
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1262)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:217)
at java.security.jgss@17.0.8/sun.security.krb5.KdcComm.send(KdcComm.java:199)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:345)
at java.security.jgss@17.0.8/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
at jdk.security.auth@17.0.8/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:761)
... 35 more
Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm LOCAL
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCFromDNS(Config.java:1359)
at java.security.jgss@17.0.8/sun.security.krb5.Config.getKDCList(Config.java:1235)
... 40 more
Is your feature request related to a problem? Please describe. When using SASL, GSSAPI -> Kerberos authentication, I get the following error:
Describe the solution you'd like To be able to use Kerberos for authentication.
Describe alternatives you've considered NA for my environment.
Additional context Similar to https://github.com/streamthoughts/jikkou/issues/273 it seems.
I looked in reflect-config.json and couldn't find any hints for Kerberos.
I'll attach the application.conf I am using so you can see the Kafka client properties (which should be correct).
Here's a small snippet of that:
application.conf.txt