search

awslabs / aws-mysql-jdbc

The Amazon Web Services (AWS) JDBC Driver for MySQL is a driver that enables applications to take full advantage of the features of clustered MySQL databases.
https://awslabs.github.io/aws-mysql-jdbc/
Other
227 stars 49 forks source link

Hikaricp connection aliver check blocked five minutes while rds master failover #612

Closed gavenpeng closed 2 months ago

gavenpeng commented 2 months ago

Support for this project will end on July 25/2024.

No response

Describe the bug

In one of our business applications, we are using HikariCP with the AWS MySQL JDBC driver. After an AWS RDS primary-secondary failover, HikariCP performs a connection validation check when attempting to acquire a connection. According to our logs, after the RDS failover (which occurred around 9:00 AM on August 18th due to an issue with the AWS EBS volumes, with the previous primary taking approximately 5 minutes to recover), HikariCP's connection validation checks were taking an unexpectedly long time—about 5 to 6 minutes. Eventually, an exception was thrown: "The active SQL connection has changed due to a connection failure. Please re-configure session state if required." This entire process, from the start of the validation check to the exception being thrown, took 5 to 6 minutes, until the previous master fully recovered，leading to significant thread exhaustion and blocking，

Expected Behavior

fast failed

Current Behavior

blocked

Reproduction Steps

master failover but origin master is not recovery normal, until the previous master fully recovered

Possible Solution

No response

Additional Information/Context

No response

The AWS JDBC Driver for MySQL version used

1.1.7.2

JDK version used

1.8.0

Operating System and version

ec2 host

gavenpeng commented 2 months ago

jdbc-cost

gavenpeng commented 2 months ago

looked jdbc code, found do failover whiel exec sql failed ,the failover blocked max time five minutes, until the reconnect master success.

karenc-bq commented 2 months ago

Hi @gavenpeng, this is expected behaviour.

The message:

The active SQL connection has changed due to a connection failure. Please re-configure session state if required.

is thrown when the AWS MySQL JDBC Driver has completed a successful driver-sided failover. When the RDS failover happened on the server side and instances became unavailable, the failed connections triggered the driver's failover feature.

You can disable driver sided failover by setting the connection property enableClusterAwareFailover to false if you'd like a fast fail behaviour.

Please note that support for this driver has ended as of July 25/2024. We recommend that you migrate to the new AWS Advanced Wrapper Driver, as all new features will be implemented there.

gavenpeng commented 2 months ago

Hi @gavenpeng, this is expected behaviour.

The message:

The active SQL connection has changed due to a connection failure. Please re-configure session state if required.

is thrown when the AWS MySQL JDBC Driver has completed a successful driver-sided failover. When the RDS failover happened on the server side and instances became unavailable, the failed connections triggered the driver's failover feature.

You can disable driver sided failover by setting the connection property enableClusterAwareFailover to false if you'd like a fast fail behaviour.

Please note that support for this driver has ended as of July 25/2024. We recommend that you migrate to the new AWS Advanced Wrapper Driver, as all new features will be implemented there.

thanks reply, the error message

The active SQL connection has changed due to a connection failure. Please re-configure session state if required. is ok, is thrown while failover successful, the main problem is blocked application thread five minutes while failover can't complete until five minute timeout, the timeout is config by set params failoverTimeoutMs, we config 30 seconds

i will migrate to new aws advanced wrapper driver.

gavenpeng commented 2 months ago

@karenc-bq the params of failoverTimeoutMs default is five minute, is too big, Will it be optimized in aws advanced wrapper driver，thanks