search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.82k stars 2.69k forks source link

[JDK 23+15] org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback fails with UnsupportedOperationException #39634

Open jerboaa opened 8 months ago

jerboaa commented 8 months ago

Describe the bug

In our Mandrel native integration tests we see new failures since the JDK 23+15 ea build in quarkus-integration-test-kafka-oauth-keycloak-999-SNAPSHOT-runner tests:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
    at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)
    at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
    at java.base@23-beta/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
    at java.base@23-beta/java.lang.ScopedValue$Carrier.runWith(ScopedValue.java:74)
    at java.base@23-beta/java.lang.ScopedValue$Carrier.call(ScopedValue.java:419)
    at java.base@23-beta/java.lang.ScopedValue.callWhere(ScopedValue.java:588)
    at java.base@23-beta/javax.security.auth.Subject.callAs(Subject.java:439)
    at java.base@23-beta/javax.security.auth.Subject.doAs(Subject.java:614)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:535)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:434)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:333)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:274)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:349)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:252)
    at java.base@23-beta/java.lang.Thread.runWith(Thread.java:1588)
    at java.base@23-beta/java.lang.Thread.run(Thread.java:1575)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:836)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:812)

See: https://github.com/graalvm/mandrel/actions/runs/8384081617/job/22961614228#step:12:968

I'm pretty sure, though, this isn't a native-only issue but can happen in JVM mode on JDK 23+15 as well.

Expected behavior

No java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed is being thrown.

Actual behavior

java.lang.UnsupportedOperationException is being thrown.

How to Reproduce?

Build Integration Tests - Kafka OAUTH with Keycloak native test and run test with a 23-beta+15-ea, vendor version: Mandrel-24.1.0-dev13054560 mandrel build. For example from here.

Additional information

This seems to be caused by https://bugs.openjdk.org/browse/JDK-8296244 freshly part of JDK 23+15. See the CSR on how to possibly fix it: https://bugs.openjdk.org/browse/JDK-8327134

quarkus-bot[bot] commented 8 months ago

/cc @Karm (mandrel), @alesj (kafka), @cescoffier (kafka), @galderz (mandrel), @ozangunalp (kafka), @pedroigor (bearer-token), @sberyozkin (bearer-token,security), @zakkak (mandrel)

cescoffier commented 8 months ago

Unfortunately there is not much we can do. It's an issue to report in the Apache Kafka Client.

jerboaa commented 8 months ago

OK. Feel free to bring this to the Apache Kafka Client team.

gsmet commented 7 months ago

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

jerboaa commented 7 months ago

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:

If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.

Hope that helps.

jerboaa commented 7 months ago

@cescoffier Is there an upstream kafka client issue for this? Who would we need to talk to to get it created? I'm hoping to get this on the radar for kafka client folks.

cescoffier commented 7 months ago

Yes, it's an issue for Apache Kafka (the upstream project).

jerboaa commented 7 months ago

Understood. If possible, could somebody create an upstream issue? That tracker seems invite only and I'm not keen on creating one just for a single bug. Could somebody with an (existing) account there file one? Thanks!

https://issues.apache.org/jira/projects/KAFKA/issues

cescoffier commented 7 months ago

@ozangunalp do you have an account?

@jerboaa let me ping Mickael Maison (working in the AMQ Streams team)

jerboaa commented 7 months ago

Thanks! Please let me know once somebody has created an issue.

cescoffier commented 7 months ago

@jerboaa There is an existing ticket: https://issues.apache.org/jira/browse/KAFKA-15862.

However, it's not planned before the Q3.

jerboaa commented 7 months ago

@cescoffier OK, thanks. Whether or not that is a problem for quarkus is something you know better. GraalVM for JDK 23 will get out in September 2024, which is when a fix/known issue for this needs to be in place.

cescoffier commented 7 months ago

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

Note that it's not only Quarkus, it's anything compiling Java to native using GraalVM.

jerboaa commented 7 months ago

Yeah, the timing might be problematic. Basically, if it's not fixed before September 2024, it would not be possible to run Quarkus apps using Kafka authentication in native.

That's my understanding. At least with GraalVM for JDK 23 (the older GraalVM for JDK 21 would still work).

geoand commented 5 months ago

Is this still an issue?

jerboaa commented 5 months ago

Yes. See https://github.com/graalvm/mandrel/issues/742#issuecomment-2151285190

cescoffier commented 5 months ago

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

jerboaa commented 2 months ago

The Kafka team has an issue for it too. It should land 'just in time' (September-ish)

@cescoffier Do you know any update on this? JDK 23 is in RC 2 and there will be a Mandrel for JDK 23 release shortly after JDK 23 GA. Should we start planning on drafting a "known issue" or the like for this, or do you think there will be a fix for it in time still.

cescoffier commented 2 months ago

@jerboaa no news so far - there is a keep and a jira (https://issues.apache.org/jira/browse/KAFKA-15862), but nothing in motion.

jerboaa commented 2 months ago

Thanks. Sounds like it's going to be a known issue for the Mandrel 24.1 for JDK 23 release.

jerboaa commented 2 months ago

@cescoffier Is there a way to advise quarkus users about the issue with the kafka-oauth-keycloak extension when they run on JDK 23+? Could we tell them that this extension isn't working on JDK 23 perhaps pointing here?

cescoffier commented 2 months ago

Except in the documentation, i don't think we can do much.

@ozangunalp do you think we can intercept the exception and log a more meaningful message?

ozangunalp commented 2 months ago

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

cc @cescoffier

zakkak commented 2 months ago

@jerboaa is there anywhere I can get an early access release Mandrel for JDK 23?

@ozangunalp you can get the latest build from https://github.com/graalvm/mandrel/actions/runs/10722382634/artifacts/1896697043

Built in this CI run https://github.com/graalvm/mandrel/actions/runs/10722382634

jerboaa commented 2 months ago

Last time I checked I couldn't reproduce this on JVM mode with 23.ea.29-open (by simply running the kafka-oauth-keycloak test)

You probably need to exercise this code path:

2024-03-22 02:18:58,664 WARN  [org.apa.kaf.com.net.Selector] (kafka-producer-network-thread | kafka-producer-out) [Producer clientId=kafka-producer-out] Unexpected error from localhost/127.0.0.1 (channelId=-1); closing connection: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
    at java.base@23-beta/javax.security.auth.Subject.getSubject(Subject.java:347)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handleCallback(OAuthBearerSaslClientCallbackHandler.java:99)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientCallbackHandler.handle(OAuthBearerSaslClientCallbackHandler.java:83)
    at org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClient.evaluateChallenge(OAuthBearerSaslClient.java:92)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)

Not sure if the test does that in all cases.

ozangunalp commented 2 months ago

That's not about the test code path. I had a -Djava.security.manager=allow on that test. I can certainly reproduce it on jdk 23.

jerboaa commented 2 months ago

I had a -Djava.security.manager=allow on that test.

That would explain it yes.

cescoffier commented 1 week ago

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

jerboaa commented 1 week ago

The root cause should be fixed in Kafka 4. Until that, we would need the flag (as shown by Ozan)

Note that -Djava.security.manager=allow will unconditional throw an exception once JEP 486 integrates for JDK 24. Is there a plan to move to Kafka 4 in time for JDK 25, say?

cescoffier commented 1 week ago

Hard to say, it will be tricky. Kafka 4 is not released yet, hopefully this year. But even with that we would need to do some integration work to continue supporting both Kafka 3 and 4 for a bit.

drakgoku commented 6 days ago

This happens because from JDK 17 onwards, the Security Manager is deprecated and in JDK 23 there are significant changes in how security is handled.

It seems that all of us who are on JDK 23 or updated projects have been stuck because of Kafka and have posted it in the wrong place. It's not Quarkus' fault but Kafka's. But since this problem doesn't exist on the Internet, then we come to post it here. So this post should be redirected to Kafka or Java or even spring.

This is going to be forgotten. You have to link to Java, Kafka and Spring boot.

This mainly has to be linked to Kafka/Java to be fixed.

zakkak commented 2 days ago

So this post should be redirected to Kafka or Java or even spring. ... This mainly has to be linked to Kafka/Java to be fixed.

@drakgoku this is already tracked in Kafka as https://issues.apache.org/jira/browse/KAFKA-15862 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support

As mentioned in https://github.com/spring-projects/spring-kafka/issues/3619#issuecomment-2468614958 it would be best if people affected by it could somehow reach the Kafka team.

I don't have an account on their issue tracker and I can't really see the comments but it looks like the issue was last updated on 30/Sep/24, and according to @cescoffier it will be fixed in version 4.0.0

Update: You can also try reaching out the Kafka team through their mailing lists https://kafka.apache.org/contact.html

anshupitlia commented 1 day ago

Isn't the security manager supposed to go away? I'm surprised they made it mandatory for some operations?

Yes the security manager is going away. The reason this change was done was supporting this move. Please see the CSR of this change which should have all the info needed: https://bugs.openjdk.org/browse/JDK-8327134

It appears a possible work-around is to add -Djava.security.manager=allow. But this comes with the following caveat:

If for some reason, user cannot modify their code in time, they can add
`-Djava.security.manager=allow` on the command line as a workaround.
This allows the old implementation to be used even if a Security Manager is never set.
Please note that in a future version when the Security Manager is completely
removed this workaround would fail.

Hope that helps.

Tried this config, still not working. :(