Narayana transaction recovery stops work in DEV mode after hot reload

michalvavrik commented 10 months ago

Describe the bug

When I change my application in DEV mode and the app is reload, Narayana keeps using the old (previous) Datasource connection even though pool has been closed and new one was created. Details in here https://github.com/quarkusio/quarkus/pull/35242#issuecomment-1668239070

Expected behavior

After hot reload, my app still works.

Actual behavior

Warning is logged and periodic recovery doesn't work

2023-08-07 18:41:54,145 WARN  [com.arj.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: This pool is closed and does not handle any more connections!
    at io.agroal.pool.ConnectionPool.beforeAcquire(ConnectionPool.java:209)
    at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:235)
    at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
    at io.quarkus.narayana.jta.runtime.QuarkusDataSource.getConnection(QuarkusDataSource.java:38)
    at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DirectDataSourceJDBCAccess.getConnection(DirectDataSourceJDBCAccess.java:27)
    at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:348)
    at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:169)
    at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:125)
    at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:81)
    at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:171)

How to Reproduce?

Reproducer:

on this branch and inside quickstart added here https://github.com/quarkusio/quarkus-quickstarts/pull/1311
quarkus dev
wait till service is fully started
force restart with s

Output of `uname -a` or `ver`

Fedora

Output of `java -version`

17

GraalVM version (if different from Java)

23

Quarkus version or git rev

999-SNAPSHOT

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.9.3

Additional information

No response

michalvavrik commented 10 months ago

/cc @zhfeng @mmusgrov

michalvavrik commented 2 months ago

Is this solved or on your radar @zhfeng @mmusgrov ?

mmusgrov commented 2 months ago

@michalvavrik I don't recall ever investigating it. You raised the issue so could you not retest it? If not then the transaction team are quite busy: @marcosgopen or @jmfinelli are either of you able to provide Michal with any input on this issue.

michalvavrik commented 2 months ago

@michalvavrik I don't recall ever investigating it. You raised the issue so could you not retest it? If not then the transaction team are quite busy: @marcosgopen or @jmfinelli are either of you able to provide Michal with any input on this issue.

copy that, thanks; I'll re-run the test later this week

marcosgopen commented 2 months ago

Sure, I will have a look at it.

marcosgopen commented 2 months ago

Hi @michalvavrik , I tested your reproducer but I got an error: [INFO] Running org.acme.quickstart.JdbcObjectStoreTest [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 4, Time elapsed: 4.396 s <<< FAILURE! - in org.acme.quickstart.JdbcObjectStoreTest [ERROR] org.acme.quickstart.JdbcObjectStoreTest.testTransactionsLogWithInjectedUserTransaction Time elapsed: 0.015 s <<< ERROR! java.lang.RuntimeException: java.lang.RuntimeException: Failed to load steps from class io.quarkus.resteasy.reactive.server.deployment.ResteasyReactiveProcessor at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:643) at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:727) at java.base/java.util.Optional.orElseGet(Optional.java:364) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) Caused by: java.lang.RuntimeException: Failed to load steps from class io.quarkus.resteasy.reactive.server.deployment.ResteasyReactiveProcessor at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:164) at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:107) at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:333) at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:254) at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:60) at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:224) at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:610) at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:660) ... 1 more Caused by: java.lang.TypeNotPresentException: Type io.quarkus.vertx.http.deployment.EagerSecurityInterceptorBuildItem not present at java.base/java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:298) at java.base/java.lang.reflect.Method.getGenericParameterTypes(Method.java:333) at java.base/java.lang.reflect.Executable.getAllGenericParameterTypes(Executable.java:317) at java.base/java.lang.reflect.Parameter.getParameterizedType(Parameter.java:202) at io.quarkus.deployment.ExtensionLoader.loadStepsFromClass(ExtensionLoader.java:493) at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:162) ... 8 more Caused by: java.lang.ClassNotFoundException: io.quarkus.vertx.http.deployment.EagerSecurityInterceptorBuildItem at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:518) at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:468) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) ... 14 more Are you also experiencing that? I have just built quarkus main branch and there might be 'breaking' changes. If you can still reproduce the issue, do you mind specify the 'git rev' of the quarkus you are using (e.g. the commit sha)? Thanks

michalvavrik commented 2 months ago

sorry @marcosgopen , I had very busy week, sure, there were breaking changes in the meanwhile and I completely forgot about QS PR. Should probably rebase it and resolve conflicts. I'll fix the reproducer next week, thanks so far.

quarkusio / quarkus