keycloak / keycloak

Open Source Identity and Access Management For Modern Applications and Services
https://www.keycloak.org
Apache License 2.0
23.13k stars 6.73k forks source link

java.sql.SQLRecoverableException: ORA-17002: I/O error: , Authentication lapse 0 ms. #33827

Open kartikshingala78 opened 2 weeks ago

kartikshingala78 commented 2 weeks ago

Before reporting an issue

Area

dist/quarkus

Describe the bug

During upgrade from keycloak 22.0.5 to keycloak 25.0.6. I am getting errors :-

1) java.sql.SQLRecoverableException: ORA-17002: I/O error: , Authentication lapse 0 ms

2) 2024-10-11 17:23:06,114 ERROR [org.jgroups.protocols.UDP] (mcast-receiver-6,SvistaLStest-15943) JGRP000030: SvistaLStest-15943: failed handling incoming message: java.lang.IllegalArgumentException: invalid magic number -31438; needs to be in range [0..100]

openjdk version "21.0.4" 2024-07-16 ojdbc11-23.5.0.24.07.jar

If I rollback to previous version everything works.

I have attached output of show-config and keycloak logs.

keycloak.log.txt kc.sh show-config.txt

Version

25.0.6

Regression

Expected behavior

Keycloak should be able to connect to Oracle RAC DB without errors.

Actual behavior

Keycloak is not able to connect to Oracle RAC DB.

How to Reproduce?

upgrade from :- openjdk version "17.0.2" 2022-01-18 keycloak 22.0.5 Upgrade to :- openjdk version "21.0.4" 2024-07-16 ojdbc11-23.5.0.24.07.jar keycloak 25.0.6

Anything else?

No response

shawkins commented 1 week ago

java.sql.SQLRecoverableException: ORA-17002: I/O error: , Authentication lapse 0 ms

Ideally this is not related to the Keycloak upgrade. A quick search seems to suggest that there's something wrong in your environment. cc @yrodiere

2024-10-11 17:23:06,114 ERROR [org.jgroups.protocols.UDP] (mcast-receiver-6,SvistaLStest-15943) JGRP000030:

This looks like you are attempting to perform a rolling upgrade and are receiving messages from the older keycloaks. This is not supported. cc @ryanemerson

Please confirm how you are performing this upgrade.

kartikshingala78 commented 1 week ago

@shawkins ,

Ideally this is not related to the Keycloak upgrade. A quick search seems to suggest that there's something wrong in your environment. cc @yrodiere

You are right this was Oracle SCAN listener issue and it is fixed now. Not getting this error anymore.

Please confirm how you are performing this upgrade.

Steps followed are as follows :-

1) Export realm from keycloak 22. kc.sh export --file ./svip_realm_keycloak22.json --realm SmartVista

2) Backup keycloak 22 directory.

3) Extract keycloak 25 binaries.

4) Copy keycloak.conf from keycloak 22 to keycloak 25.

5) Import realm to keycloak exported kc.sh import --file ./svip_realm_keycloak22.json

6) Start keycloak kc.sh start

It was giving errors :- java.sql.SQLRecoverableException: ORA-17002: I/O error: , Authentication lapse 0 ms

I tried to update it manually using below command :- kc.sh start --spi-connections-jpa-default-migration-strategy=update

After fixing listener issue "ORA-17002: I/O error:" is gone

But below error is still there :-

ERROR [org.jgroups.protocols.UDP] (mcast-receiver-6,SvistaLStest-15943) JGRP000030: SvistaLStest-15943: failed handling incoming message: java.lang.IllegalArgumentException: invalid magic number -31438; needs to be in range [0..100]

Seems issue is caused by below command but I am not sure how to fix it.

kc.sh start --spi-connections-jpa-default-migration-strategy=update

shawkins commented 1 week ago

Are you stopping the keycloak 22 instance before starting the new one? I wouldn't expect there to be an invalid message coming over UDP unless both versions were joining the cluster at the same time.

kartikshingala78 commented 1 week ago

Yes, keycloak 22 was stopped before starting keycloak 25. And this is is standalone keycloak it is not a cluster.

We are still getting these messages in logs and only keycloak 25 is running at this time.

shawkins commented 1 week ago

Added core-shared, I'm not what else would cause a jgroups error like that.

ryanemerson commented 1 week ago

Are you stopping the keycloak 22 instance before starting the new one? I wouldn't expect there to be an invalid message coming over UDP unless both versions were joining the cluster at the same time.

This is what I would expect as well

shawkins commented 6 days ago

Based upon the logs, it does not appear there is another member in the cluster:

2024-10-11 16:24:44,277 INFO  [org.jgroups.protocols.pbcast.GMS] (Thread-5) SvistaLStest-5469: no members discovered after 2003 ms: creating cluster as coordinator
2024-10-11 16:24:44,285 INFO  [org.infinispan.CLUSTER] (Thread-5) ISPN000094: Received new cluster view for channel ISPN: [SvistaLStest-5469|0] (1) [SvistaLStest-5469]
2024-10-11 16:24:44,344 INFO  [org.infinispan.CLUSTER] (Thread-5) ISPN000079: Channel `ISPN` local address is `SvistaLStest-5469`, physical addresses are `[172.18.104.116:50166]

Copy keycloak.conf from keycloak 22 to keycloak 25.

@kartikshingala78 Can you provide the show-config output? And to double check is there anything that would alter the dependencies in keycloak, such as having custom providers jars?

keycloak-github-bot[bot] commented 6 days ago

Thanks for reporting this issue, but there is insufficient information or lack of steps to reproduce.

Please provide additional details, otherwise this issue will be automatically closed within 14 days.

kartikshingala78 commented 6 days ago

@shawkins, Yes, it is stand alone instance there is no other member.

I have attached output of show-config to :- https://github.com/user-attachments/files/17343731/kc.sh.show-config.txt

There are no custom jar files in providers.

shawkins commented 6 days ago

@kartikshingala78 thank you for the additonal information.

@ryanemerson what else would be good to have here?

ryanemerson commented 5 days ago

@ryanemerson what else would be good to have here?

Tbh I don't know at this point. As you point out there's no other members joining the cluster, but it still appears that packets of an unknown format are being received over UDP. Could it be from a non-keycloak process? @pruivo any thoughts?

pruivo commented 5 days ago

It could be an external process with some lucky packet or another JGroups process running on the network.

2024-10-11 16:24:45,895 ERROR [org.jgroups.protocols.UDP] (mcast-receiver-6,SvistaLStest-5469) JGRP000030: SvistaLStest-5469: failed handling incoming message: java.lang.IllegalArgumentException: invalid magic number -31438; needs to be in range [0..100]
        at org.jgroups.conf.ClassConfigurator.create(ClassConfigurator.java:136)
        at org.jgroups.util.Util.readSizeStreamable(Util.java:2004)
        at org.jgroups.ObjectMessage.readPayload(ObjectMessage.java:144)
        at org.jgroups.BaseMessage.readFrom(BaseMessage.java:317)
        at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1347)
        at org.jgroups.protocols.TP.receive(TP.java:1303)
        at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:809)
        at java.base/java.lang.Thread.run(Thread.java:1583)

This bothers me because:

A network packet tool may help to figure it out where it comes from. @belaban may have more insights.

belaban commented 1 day ago

This is probably an incompatibility between JGroups versions used by KC 22 and 25. It was mentioned that this is an unclustered deployment, but I guess that's not correct, or else we'd not be seeing these warning messages...?

ahus1 commented 13 hours ago

If this is a standalone instance, then disable the clustering. Otherwise it might interfere with other standalone instances that might be discovered accidentally using multicast.

To disable it, start Keycloak with --cache=local as described in https://www.keycloak.org/server/caching

Then this issue should go away. In Keycloak 26.1 we plan to have a better way of discovering other nodes which should give errors with UDP or multicast (see #34102 which implements JDBC_PING).

ahus1 commented 13 hours ago

Please comment here if this fixes your issue, and close the issue if it does. Thanks!