brettwooldridge / HikariCP

光 HikariCP・A solid, high-performance, JDBC connection pool at last.
Apache License 2.0
19.89k stars 2.92k forks source link

HikariPool-1 - Failed to validate connection oracle.jdbc.driver.T4CConnection@3b904d4f (IO Error: Invalid Operation, NOT Connected). Possibly consider using a shorter maxLifetime value. #1731

Open sanchitgarg-its opened 3 years ago

sanchitgarg-its commented 3 years ago

I am using below maven dependency with below configuration but getting some warn in log.

<dependency>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
        <version>3.3.0</version>
</dependency>

with below config datasource.driver-class-name=oracle.jdbc.OracleDriver datasource.maximum-pool-size=25 datasource.connection-timeout=30000 datasource.connection-test-query=select 1 from dual datasource.minimum-idle=1

Log see:

HikariPool-1 - Failed to validate connection oracle.jdbc.driver.T4CConnection@3b904d4f (IO Error: Invalid Operation, NOT Connected). Possibly consider using a shorter maxLifetime value.

niklasskeppstedt commented 3 years ago

I have the same problem. Spring Boot app using HikariCP-3.4.2.jar. App is working as it should, doing a scheduled task which succeeds. Then 20 minutes after this error starts, and the calling method is PoolBase.isConnectionAlive This is invoked by org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator and is called all the time by Kubernetes readiness check. But suddenly this happens, and after a while the Kubernetes pod is restarted and everything is back to normal. Is there something we should do about it?

FeynmanCheng commented 3 years ago

I have the same problem. The app is working but keeps printing such msg.

ghost commented 3 years ago

Has anyone found an answer as to why this keeps happenig? I have the same issue on an app we've deployed on our platform.

rpgn commented 3 years ago

For some reason the network timeout in PoolBase is set to 0 which throws an exception. If I set a breakpoint in PoolBase::isConnectionAlive and patch the networkTimeout value to 10 then the problem 'goes away'.

raja-anbazhagan commented 2 years ago

It would be worth checking with your network team to see if the firewall between database and the app servers is dropping the connections.

GuilhermeSerra commented 1 year ago

@sanchitgarg-its did you find a solution?

ChanaLii commented 1 year ago

@sanchitgarg-its did you find a solution?

try something like 30minutes、10minutes, or even 2minutes.I set max-lifetime to 2 minutes and the problem is solved.

rpgn commented 1 year ago

That's not solving the problem. That is at most mitigating the problem. This should work without giving any nasty messages or errors out of the box.

DurableHumorGitHub commented 1 year ago

Is anybody found a solution?

jamilxt commented 1 year ago

I'm facing this also. Is the issue related to HikariCP itself?

By default, the Hikari max-lifetime is 1800000 ms (30 mins). I'm giving it a try with a shorter max lifetime. Let's see how it goes. If there is any permanent solution, please let me know.

Updated: Still getting the issue. :(

thanhhaibka commented 1 year ago

@jamilxt did you upgrade to lastest version, im facing same issue at 4.0.3 :(

raja-anbazhagan commented 1 year ago

Guys, If you have any firewall between your Application servers and Database, You may need to check the timeout for your Firewall settings. Usually, some firewalls disconnect any TCP connections that didn't have any packets.

It is also possible that the database has a connection time limit as well.

It's worth checking both of these situations.

jamilxt commented 1 year ago

@jamilxt did you upgrade to lastest version, im facing same issue at 4.0.3 :(

@thanhhaibka latest version of? I'm experiencing it while using Azure SQL DB. When I switched to my local MSSQL, the issue doesn't occur. I think @raja-anbazhagan is right checking other reasons as well. I will check on Azure or our company's firewall, if the issue is causing because of them.

Thank you @raja-anbazhagan for the hint!

thanhhaibka commented 1 year ago

Many thanks, @jamilxt and @raja-anbazhagan , i will try to check my firewall and will update when I have.

PiotrekWojtowicz commented 1 year ago

Having the same issue, has anyone been able to fix this so far?

patric-r commented 1 year ago

We're also facing and still investigating this issue. In our case, it only happens when running within a VM and when a VM snapshot is created (which possibly results in a short-term stall of the VM).

We were able to rule out infrastructure related causes (network, database, VM) and we're now focusing on software side causes (e.g. HikariCP or within our applications which use HikariCP).

Questions to all affected users here:

thanhhaibka commented 1 year ago

Having the same issue, has anyone been able to fix this so far? With my issue, I have checked the Hikari detail log and noticed that the connection was invalid, even though it was created just 2 seconds prior. Therefore, I requested the Infrastructure team to disable all McAfee firewalls, and the issue was resolved.

patric-r commented 1 year ago

There is a fairly recent documentation entry from oracle, 2910414.1. Interestingly, it is referring to exactly this error when using HikariCP!

Recommendation is to ensure that the net.core.somaxconn kernel parameter of the database server (not the server on which the applications are running!) is set to an high enough value, e.g. 32768.

To all affected users: Can you try that and see whether increasing this parameter will solve this issue?

Rejwan-Sharif commented 8 months ago

Is this Problem related with Scheduler? I am facing same problem.

chaitanyaKaushal commented 2 months ago

Any resolutions?