Philippvs
commented
4 months ago That would help a lot!
Closed Wallman closed 1 month ago
Philippvs
commented
4 months ago That would help a lot!
Wallman
commented
4 months ago Or does it already have support? Seems to work for us.
Philippvs
commented
4 months ago Yes but with errors
tomix26
commented
4 months ago Thanks for the report. Could you please attach a log with the error for easier investigation?
I'm quite busy right now, but I will take a look at it next week or the week after.
Philippvs
commented
4 months ago The Problem is the following: If a test fails for instance then it will not be stopped (that problem does not exist in Spring boot 3.2)
Full Stack trace: ` 2024-07-07T13:19:10.016+02:00 WARN 10912 --- [pool-3-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08001 2024-07-07T13:19:10.016+02:00 ERROR 10912 --- [pool-3-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper : Connection to localhost:60279 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. 2024-07-07T13:19:10.019+02:00 ERROR 10912 --- [pool-3-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:466) ~[spring-orm-6.1.10.jar:6.1.10]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:532) ~[spring-tx-6.1.10.jar:6.1.10]
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:405) ~[spring-tx-6.1.10.jar:6.1.10]
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:617) ~[spring-tx-6.1.10.jar:6.1.10]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:386) ~[spring-tx-6.1.10.jar:6.1.10]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.1.10.jar:6.1.10]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.10.jar:6.1.10]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.10.jar:6.1.10]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) ~[spring-aop-6.1.10.jar:6.1.10]
at com.bmw.tmx.service.scheduler.SchedulerExecutionJob$$SpringCGLIB$$0.executeScheduledJob(
tomix26
commented
3 months ago @Philippvs I tried it and everything worked as expected. However, based on the log you sent, I would guess that the issue might be caused by database calls from a scheduled background thread (originating from Micrometer) in combination with refresh mode.
If you are using @AutoConfigureEmbeddedDatabase(refresh = XXX) or @FlywayTest annotations, new databases are continuously created and destroyed in the background. If there is a scheduled thread that is not aligned with the test phases, it might call a database that no longer exists, causing the error described above.
Nevertheless, such exceptions should be relatively rare and should not affect the test results. The solution is either to suppress such background threads, as they introduce non-determinism into the tests, or not to use refresh mode, which ensures a common database across all tests.
So, I would like to confirm if you are using refresh mode and how often the described error occurs? Or, if you think the error is caused by something else, I would need a reproducer for further investigation of the issue.
Philippvs
commented
3 months ago Yes I am using the following:
@AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.AFTER_EACH_TEST_METHOD)
Downgrading from Spring 3.3 to 3.2 resolves the problem. When running all the tests of a test class the suit does not terminate and keeps open like in an endless loop. Same for running a single test. I do not use any background jobs in my tests.
tomix26
commented
3 months ago Ok, thank you for the clarification.
While you may not be using any background jobs explicitly, I noticed the following line in the log:
at io.micrometer.observation.Observation.observe(Observation.java:499) ~[micrometer-observation-1.13.1.jar:1.13.1]
This suggests that there's a Micrometer thread running in the background, likely attempting to report some application metrics to the database. It's possible that between Spring 3.2 and 3.3, there was a change in Micrometer configuration resulting in this background thread running. I can't determine if this is intended behavior or a bug. From my perspective, I would recommend disabling such features that are not directly related to the test, as they can lead to non-deterministic behavior, as I mentioned earlier.
I would really like to help you solve this problem, but without a reproducer, I can't accurately assess what's happening in your project, which auto-configurations are active, and how they're interacting with each other.
reneleonhardt
commented
3 months ago @Philippvs Could you provide a minimal example repo for reproducing your problem so Spring Boot 3.3 support could move forward? Otherwise 3.4 will come out faster 😅
tomix26
commented
3 months ago @reneleonhardt Are you also experiencing some issues when upgrading to Spring Boot 3.3?
reneleonhardt
commented
3 months ago Didn't have time yet to upgrade to 3.3, there were some breaking changes (Prometheus sigh). My issue with Zonky in the past was that it was not easy to configure the base template as needed (init.sql first, then flyway migrations, then class BeforeAll, then BeforeEach) and to use the refresh mode between separate test methods so every class and method could run in parallel without long context reload times to reduce integration test time by several minutes). When I have time I will report if those are still obstacles for me or if RTFM helped this time 😅
For me personally it would help massively if there would always be a minimal but complete "right way" example for the latest versions (SB 3.3, Zonky 2.5/2.0, JUnit 5.10, Postgres 16, Java 21 with Virtual Threads and Shenandoah for 1 ms max GC pauses), which would be run as a GitHub workflow inside this repo, so I know it would be working and contain all necessary configuration/annotations needed for the fastest DB tests so I can get rid of super slow and often breaking testcontainers-java at least for Postgres. At least one thing is good about slow Kafka... EmbeddedKafka makes a Docker container unnecessary for most integration tests 😄
tomix26
commented
3 months ago Thank you for your comprehensive comment 🙂
init.sql first, then flyway migrations, then ...
Yes, this is a known issue. I plan to improve and simplify this in the future. Unfortunately, there isn't much time to develop new features right now.
so every class and method could run in parallel
I'm not sure about the current capabilities of the Spring framework, but previously, parallelization was only possible at the JVM process level. So it wasn't possible for one Spring context to be shared by multiple threads from different tests. Perhaps this has changed, I have no idea.
it would help massively if there would always be a minimal but complete "right way" example for the latest versions
In this spirit, I created the project https://github.com/tomix26/embedded-database-demo, where I test new versions of Spring and other libraries. It's true that it's not entirely up-to-date right now, but I've upgraded the versions of necessary libraries locally, and everything still works the same way.
with Virtual Threads and Shenandoah for 1 ms max GC pauses
What's so special about virtual threads and this specific GC in connection with this library that makes it necessary to have it incorporated in the GitHub workflow?
reneleonhardt
commented
3 months ago Thank you for your comprehensive comment 🙂
You're welcome!
Yes, this is a known issue. I plan to improve and simplify this in the future. Unfortunately, there isn't much time to develop new features right now.
I can imagine, it's hard to find time to make all contributions yourself. Maybe you could lay out a rough idea to achieve this, for example in a milestone containing some issues, then the community could contribute the necessary steps more easily.
I'm not sure about the current capabilities of the Spring framework, but previously, parallelization was only possible at the JVM process level. So it wasn't possible for one Spring context to be shared by multiple threads from different tests. Perhaps this has changed, I have no idea.
I'm not sure either, I just see with Cucumber and a Postgres Testcontainer everyday for example, that not even classes are possible to run in parallel, so the integration tests are multiplying the build time several times 😠 As an alternative, could the base template and every derived template (containing the test method specific data) maybe prepared in BeforeAll, so each test method only had to choose the template specific for itself? So the context would not need to be refreshed/replaced before each test method?
| template | test level | structure | data |
|---|---|---|---|
| base | class | init + flyway | common data for all methods in this class |
| method1 | method | derived from base | all data needed for method1 including derived base data |
| method2 | method | derived from base | all data needed for method2 including derived base data |
| methodN | method | derived from base | all data needed for methodN including derived base data |
In this spirit, I created the project https://github.com/tomix26/embedded-database-demo, where I test new versions of Spring and other libraries. It's true that it's not entirely up-to-date right now, but I've upgraded the versions of necessary libraries locally, and everything still works the same way.
Thank you very much, I will look into it this week! 👍
What's so special about virtual threads and this specific GC in connection with this library that makes it necessary to have it incorporated in the GitHub workflow?
It's not really necessary for tests, but it helped me massively in my last project, Shenandoah and ZGC are modern Hotspot GCs which can help reducing stop-the-world pauses / latency dozens or hundreds of times. After my last lessons learned I just want to be sure that frameworks and libraries make proper use of modern JVM features to avoid bad production surprises where Spring endpoints took 2 seconds when their parallel DB queries took only a few milliseconds. Pro tip: Use Jaeger Tracing to search and visualize those blocking pauses, then you can watch GC metrics in Grafana 😉
tomix26
commented
3 months ago I've just upgraded the https://github.com/tomix26/embedded-database-demo project to Spring 3.3.2 and JUnit 5, and everything works as expected. However, the project doesn't include Micrometer, so I couldn't verify the issue reported above.
So the context would not need to be refreshed/replaced before each test method?
If you're talking about the Spring context, it doesn't need to be refreshed/replaced because of the database. The Zonky library tries to avoid refreshing the Spring context under all circumstances. The problem with parallelizing tests in Spring is that if multiple tests share the same context, the beans of this context are also shared among these tests. So you have, for example, 10 tests, and they have a shared DataSource bean. This isn't a problem if the tests run sequentially. Whenever a specific test is running, the database referenced by the DataSource bean is set up and loaded by data for that particular test. However, if the tests run in parallel and the DataSource bean is shared, it's necessary to implement some smart request dispatching so that based on which test is calling the DataSource at a given moment, the call is delegated to the corresponding underlying database. This, besides being problematic to implement, can also significantly complicate the predictability of tests and their debugging.
divStar
commented
1 month ago I have just checked out the embedded-database-demo repository and am having the same issue the OP is having. I also have it on my other project.
The issue solely occurs when using zonky.test.database.provider=docker (default). When using the zonky provider, it works in my other application.
Here's the full stacktrace (I haven't added or changed anything, just ran all the tests using IntelliJ):
I think the reason might be the system I am working on or the Docker engine I am using.
Here are my specs:
If I had to guess the issue probably occurs due to us not using the original Docker Engine.
Reverting to say Spring Boot 3.2.5 does not solve the issue - at least in my case (perhaps my case is a bit different?).
Wallman
commented
1 month ago We are successfully running it with Spring Boot 3.3 nowadays
tomix26
commented
1 month ago I have just checked out the embedded-database-demo repository and am having the same issue the OP is having. I also have it on my other project.
Based on the logs you provided, it's apparent that the testcontainers library can't connect to the Docker containers. As you mentioned, the problem is likely caused by using a non-standard Docker engine. Here's a guide on how to resolve the issue: https://java.testcontainers.org/supported_docker_environment
If problems persist, please open a new separate issue. Because this is a completely different problem and isn't related to Spring Boot 3.3.
We are successfully running it with Spring Boot 3.3 nowadays
Thank you for the confirmation.
tomix26
commented
1 month ago I'm closing this issue for now. I've tested Spring Boot 3.3 and haven't found any issues. The cause of the problem is likely in the integration with the Micrometer library. However, without a reproducer, I'm unable to solve the problem. Feel free to reopen this issue if you can provide more information, ideally a reproducer.
Would be great to have Spring Boot 3.3 support