search

helidon-io / helidon

Java libraries for writing microservices
https://helidon.io
Apache License 2.0
3.51k stars 566 forks source link

Helidon DBClient does not trigger an Exception when no sane DB connection can be obtained , post application initialization #4719

Closed msteffi closed 2 years ago

msteffi commented 2 years ago

In case the database server was up during the application initialization and for some reason the server went down ,and then when the application tries to insert a row , we expect a DBClientException to be thrown after the connection timeout , which does not seem to happen .

Environment Details

Problem Description

We are using DBClient in our Helidon SE project with the below configurations url: jdbc:mysql://127.0.0.1:3306/dbName?failOverReadOnly=false&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true username: root password: passwd poolName: mysql initializationFailTimeout: -1 connectionTimeout: 2000

So if the MySql server is up and running , we see no issues as such . But in case the database server was up during the application initialization and for some reason the server went down ,and then when the application tries to insert a row , we expect a DBClientException to be thrown after the connection timeout of 2000 ms , which does not seem to happen .

Snippet of the thread dump of the helidon application

"helidon-jdbc-dbclient-thread-pool-5" #23 daemon prio=5 os_prio=0 cpu=0.74ms elapsed=1453.67s tid=0x00007f90ecd635b0 nid=0x32c2 waiting on condition [0x00007f90b58f7000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@17.0.2/Native Method)

Steps to reproduce

[//]: 1 Database server must be up and running when the application starts . Bring down or make the db server non-reachable . [//]: 2 Make your helidon application to do some operation on the table (insert) and then you will observe that the operation "execute" just hangs . dbClient .execute( exec -> exec.createNamedInsert("insert") .indexedParam(entity ) .execute()) .thenApply(e -> entity ) .exceptionally(throwable -> { logger.atWarning().log( "Failed to insert : %s", throwable); return null; });

tomas-langer commented 2 years ago

Update from customer: createNamedGet is throwing a DBClient Exception whereas other apis like createNamedInsert / update and delete are not .

chandrasrt commented 2 years ago

Hi Helidon Experts, Thanks for clarification on impact. We would like to have this fix/patch in Helidon 2.5.1/2.52. Is there an ETA on this story? Currently we are blocking our story until a fix/workaround is provided, does this sound reasonable? Thanks Chandra

msteffi commented 2 years ago

namedDml api as well has the same issue . It does not seem to throw a DBClientException if a query (in our case it was create table) is executed when the DB server is unavailable .

Tomas-Kraus commented 2 years ago

Tested similar scenario with my old Pokemon example. Running curl -i -X POST -d '{"id":8,"name":"Raticate","idType":2}' http://localhost:8079/db/pokemon caused some warnings to be logged:

2022.08.23 13:45:21 WARNING com.zaxxer.hikari.pool.PoolBase Thread[helidon-jdbc-dbclient-thread-pool-4,5,jdbc-dbclient-thread-pool]: mysql - Failed to validate connection com.mysql.cj.jdbc.ConnectionImpl@4b83f76 (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.
2022.08.23 13:45:21 WARNING com.zaxxer.hikari.pool.PoolBase Thread[helidon-jdbc-dbclient-thread-pool-4,5,jdbc-dbclient-thread-pool]: mysql - Failed to validate connection com.mysql.cj.jdbc.ConnectionImpl@372ace49 (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value.

But example insert code

        dbClient.execute(exec -> exec
                .createNamedInsert("insert-pokemon")
                .indexedParam(pokemon)
                .execute())
                .thenAccept(count -> response.send("Inserted: " + count + " values\n"))
                .exceptionally(throwable -> sendError(throwable, response));

hangs as described in the bug.

Tomas-Kraus commented 2 years ago

Looks like one executorService().submit(...) call is eating exceptions in JdbcStatementDml. :)

Tomas-Kraus commented 2 years ago

@msteffi PR https://github.com/oracle/helidon/pull/4772 should fix this for 2.x. You can verify it using https://github.com/Tomas-Kraus/helidon/tree/issue-4719-2.x branch build if you would like to.