issue getting this to work with Spring Boot 2.7.5

uli-f commented 1 year ago

Hi,

my project uses Spring Boot 2.7.5 and explicitly includes the following dependencies (written in gradle notation):

implementation 'org.springframework.boot:spring-boot-starter-batch'
implementation 'org.springframework.boot:spring-boot-starter-data-mongodb'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.batch:spring-batch-test'
testImplementation 'de.flapdoodle.embed:de.flapdoodle.embed.mongo.spring27x:4.3.2'

I got an application-test.properties that includes the following line: spring.mongodb.embedded.version=6.0.1

My integration test class is decorated with quite a few Spring annotations:

@SpringBootTest
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
@EnableBatchProcessing
@SpringBatchTest
class MyTest {...}

This works just fine. But it seems to use the autoconfiguration provided by Spring; if I remove the line spring.mongodb.embedded.version=6.0.1 from my application-test.properties the test fails with the following message: Set the spring.mongodb.embedded.version property or define your own MongodConfig bean to use embedded MongoDB

I then implemented the following changes in my application-test.properties:

remove spring.mongodb.embedded.version=6.0.1
add de.flapdoodle.mongodb.embedded.version=6.0.1
add spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.mongo.embedded.EmbeddedMongoAutoConfiguration

The idea is that I explicitly exclude the Spring autoconfiguration for embedded mongo and switch the namespace for the configuration of the embedded mongodb version from spring to flapdoodle. However, this is the error message I get when running my test:

Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused: no further information}}]

Any help is appreciated. Please let me know if you need any more information.

michaelmosmann commented 1 year ago

@uli-f i think you have to include more than the spring adapter.. in an sample project i needed to do this:

testImplementation("de.flapdoodle.embed:de.flapdoodle.embed.mongo.spring27x:4.3.2") testImplementation("de.flapdoodle.embed:de.flapdoodle.embed.mongo:4.2.0")

.. otherwise somehow an old version of de.flapdoodle.embed.mongo is used.. i tried a lot but i am not sure why this happens..

uli-f commented 1 year ago

Thank you @michaelmosmann for your reply.

The old dependency of de.flapdoodle.embed.mongo is used (at least for my gradle configuration) because the Spring gradle plugin io.spring.dependency-management sets a plethora of dependencies to particular versions. One of its dependency management coordinates is de.flapdoodle.embed:de.flapdoodle.embed.mongo 3.4.11. You can list the dependency management coordinates managed by the Spring gradle plugin io.spring.dependency-management with

gradle dependencyManagement

and overwrite a particular dependency management coordindate with

dependencyManagement {
    dependencies {
        dependency group:'de.flapdoodle.embed', name:'de.flapdoodle.embed.mongo', version:'4.3.0'
    }
}

in your build.gradle.

Please let me know if you want me to write something up that you can put in the README.md of this github project.

uli-f commented 1 year ago

To quickly repeat my configuration:

the newest version 4.3.0 of de.flapdoodle.embed:de.flapdoodle.embed.mongo gets pulled in as a dependency
Spring's own EmbeddedMongoAutoConfiguration is excluded by adding spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.mongo.embedded.EmbeddedMongoAutoConfiguration to my application-test.properties
the embedded mongodb version is set to 6.0.1 by adding de.flapdoodle.mongodb.embedded.version=6.0.1 to my application-test.properties

I then ran out of memory with the following stacktrace when starting my test class:

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.nio.file.Files.read(Files.java:3238)
    at java.base/java.nio.file.Files.readAllBytes(Files.java:3295)
    at de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore.lambda$hash$5(ContentHashExtractedFileSetStore.java:115)
    at de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore$$Lambda$927/0x00000008011384a0.get(Unknown Source)
    at de.flapdoodle.types.ThrowingSupplier.lambda$mapException$0(ThrowingSupplier.java:30)
    at de.flapdoodle.types.ThrowingSupplier$$Lambda$893/0x00000008011286a0.get(Unknown Source)
    at de.flapdoodle.types.Try.get(Try.java:42)
    at de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore.hash(ContentHashExtractedFileSetStore.java:115)
    at de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore.extractedFileSet(ContentHashExtractedFileSetStore.java:56)
    at de.flapdoodle.embed.process.transitions.ExtractPackage.extractedFileSet(ExtractPackage.java:108)
    at de.flapdoodle.embed.process.transitions.ExtractPackage.result(ExtractPackage.java:102)
    at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:62)
    at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:50)
    at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:165)
    at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:69)
    at de.flapdoodle.embed.mongo.spring.autoconfigure.MongodWrapper.start(MongodWrapper.java:45)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1930)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1872)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
    at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$587/0x0000000800eaaca0.getObject(Unknown Source)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

This doesn't happen with when using the older version 5.0.6 of the embedded mongodb by setting de.flapdoodle.mongodb.embedded.version=5.0.6. I assume the file size of the downloaded mongodb binary has increased from version 5.0.6 to version 6.0.1.

The error goes away when increasing the maximum heap size to 1024m. However, the default max heap size for recent JVMs is 512m and this is also the max heap size of the JVM that gradle forks to run tests with. So I would assume that people easily run into this OOM.

Is there a way to bring the required memory to a more reasonable size?

michaelmosmann commented 1 year ago

@uli-f ah.. this should not happen... I think I can fix this..

michaelmosmann commented 1 year ago

Thank you @michaelmosmann for your reply.

The old dependency of de.flapdoodle.embed.mongo is used (at least for my gradle configuration) because the Spring gradle plugin io.spring.dependency-management sets a plethora of dependencies to particular versions. One of its dependency management coordinates is de.flapdoodle.embed:de.flapdoodle.embed.mongo 3.4.11. You can list the dependency management coordinates managed by the Spring gradle plugin io.spring.dependency-management with

gradle dependencyManagement

and overwrite a particular dependency management coordindate with
dependencyManagement {
    dependencies {
        dependency group:'de.flapdoodle.embed', name:'de.flapdoodle.embed.mongo', version:'4.3.0'
    }
}
in your build.gradle.

Please let me know if you want me to write something up that you can put in the README.md of this github project.

your welcome:) .. there is a branch for spring 2.7.x if you want to create an pull request:)

michaelmosmann commented 1 year ago

@uli-f .. there is a new release:) de.flapdoodle.embed:de.flapdoodle.embed.mongo:4.3.1

uli-f commented 1 year ago

Just tested de.flapdoodle.embed:de.flapdoodle.embed.mongo:4.3.1. Works fine with default max heap size of 512m. Thank you for this fix, much appreciated.

michaelmosmann commented 1 year ago

@uli-f there is a new version where the archive hash is cached .. it should be a little bit faster now. i use the hash of the archive to ensure that the extracted files are from this archive .. before the last chance which each start the hash of the archive was created reading the whole file .. which is.. IO expensive .. now i store the hash based on the path and last modified file attribute..

so this bug was the root cause for this improvement ..

happy testing:)

uli-f commented 1 year ago

Updated to group:'de.flapdoodle.embed', name:'de.flapdoodle.embed.mongo', version:'4.3.3'.

My tests include quite a few mongodb tests; overall those are the most time-consuming ones for sure as most of them are annotated with @DirtiesContext.

I know it is only a rough figure as I ran the tests once and the JVM's magic might have an impact on the runtime.

group:'de.flapdoodle.embed', name:'de.flapdoodle.embed.mongo', version:'4.3.1' -> 2min 11sec group:'de.flapdoodle.embed', name:'de.flapdoodle.embed.mongo', version:'4.3.3' -> 1min 33sec

Looks like an increase in speed to me 🙂

michaelmosmann commented 1 year ago

@uli-f i am happy that my bugfix does make such impact .. but i should have found this much earlier:)

flapdoodle-oss / de.flapdoodle.embed.mongo.spring

issue getting this to work with Spring Boot 2.7.5 #4