search

flapdoodle-oss / de.flapdoodle.embed.mongo

...will provide a platform neutral way for running mongodb in unittests.
Apache License 2.0
910 stars 160 forks source link

LocalDownloadCache error on multiprocess/multithreaded runs #547

Open Geminist opened 1 week ago

Geminist commented 1 week ago

Version 4.13.1

java.lang.IllegalArgumentException: archive for /mongodb-fastdl/linux/mongodb-linux-x86_64-rhel80-6.0.15.tgz:TGZ already exists with different content (mongodb-fastdllinuxmongodb-linux-x86-64-rhel80-6-0-15-tgz/e89a21cee4ab7eec981d8c6e527e6626dc621ece9e44bd5496e0891d3b8efa8f/archive.tgz)
    at de.flapdoodle.checks.Preconditions.checkArgument(Preconditions.java:32)
    at de.flapdoodle.embed.process.store.LocalDownloadCache.store(LocalDownloadCache.java:77)
    at de.flapdoodle.embed.process.transitions.DownloadPackage.lambda$result$5(DownloadPackage.java:158)

The latest 4.18.0 doesn't fix this. Copy isn't atomic. Files.copy() should be replaced with Files.move(). But this also does not completely solve the problem if the archive and arcFile are in the different file systems.

ms1111 commented 6 days ago

I'm seeing this as well, when running tests with Maven using -T2 -DforkCount=2. Running with Spring Boot v3.3.5, Spring v6.1.14, flapdoodle 4.16.1.

In past releases I would configure CI to pre-download and extract the mongodb tarball, but the structure is a bit more complicated now with fileSets and hashes.

It's similar to https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo.spring/issues/48.

Logs: ``` 2024-11-18 17:36:52.796GMT INFO [main] [-|-@-] de.flapdoodle.embed.process.runtime.ProcessControl: stopOrDestroyProcess: process hasn't exited 2024-11-18 17:36:52.917GMT INFO [main] [-|-@-] d.f.e.mongo.spring.autoconfigure.SyncClientServerFactory: sync server factory 2024-11-18 17:36:53.591GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : starting... 2024-11-18 17:36:53.596GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 0 % 2024-11-18 17:36:53.930GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : starting... 2024-11-18 17:36:53.934GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 0 % 2024-11-18 17:36:53.987GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 10 % 2024-11-18 17:36:54.152GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 20 % 2024-11-18 17:36:54.230GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 30 % 2024-11-18 17:36:54.191GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 10 % 2024-11-18 17:36:54.316GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 40 % 2024-11-18 17:36:54.367GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 20 % 2024-11-18 17:36:54.396GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 50 % 2024-11-18 17:36:54.437GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 60 % 2024-11-18 17:36:54.456GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 30 % 2024-11-18 17:36:54.466GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 70 % 2024-11-18 17:36:54.495GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 80 % 2024-11-18 17:36:54.524GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 90 % 2024-11-18 17:36:54.534GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 40 % 2024-11-18 17:36:54.580GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 50 % 2024-11-18 17:36:54.569GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : finished 2024-11-18 17:36:54.613GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 60 % 2024-11-18 17:36:54.635GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 70 % 2024-11-18 17:36:54.654GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 80 % 2024-11-18 17:36:54.674GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : 90 % 2024-11-18 17:36:54.705GMT INFO [main] [-|-@-] d.flapdoodle.embed.mongo.spring.autoconfigure.EmbeddedMongo: download https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz : finished 2024-11-18 17:36:54.772GMT WARN [main] [-|-@-] o.s.context.annotation.AnnotationConfigApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '...': Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'mongoTemplate' defined in ....DBConfig: Unsatisfied dependency expressed through method 'mongoTemplate' parameter 0: Error creating bean with name 'mongoDbFactory' defined in ...DBConfig: Unsatisfied dependency expressed through method 'mongoDbFactory' parameter 1: Error creating bean with name 'syncClientServerWrapper' defined in class path resource [de/flapdoodle/embed/mongo/spring/autoconfigure/EmbeddedMongoAutoConfiguration$SyncClientServerWrapperConfig.class]: rollback after error on transition to State(Archive), successful reached: State(DownloadCache)=de.flapdoodle.embed.process.store.LocalDownloadCache@459cfed4, State(Package)=Package{archiveType=TGZ, fileSet=FileSet{entries=[Entry{type=Executable, destination=mongod, uncompiledMatchingPattern=UncompiledPattern{regex=.*mongod, flags=2}}]}, url=https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz}, State(ExtractedFileSetStore)=de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore@5ce4b8ef, State(ProcessArguments)=ProcessArguments([--dbpath, /tmp/temp--67247ae2-d27d-4193-ba58-93fdeb535042/mongod-database7912467977103391918, --noauth, --port, 44751, --bind_ip, 127.0.0.1, --syncdelay=0]), State(ProcessWorkingDir)=ProcessWorkingDir(/tmp/temp--67247ae2-d27d-4193-ba58-93fdeb535042/workingDir6955265644063939298), State(Distribution)=GenericFeatureAwareVersion{7.0.12}:Platform{operatingSystem=Linux, architecture=X86_64, distribution=Ubuntu, version=Ubuntu_22_04}, State(SupportConfig)=SupportConfig{name=mongod, supportUrl=https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues, messageOnException=de.flapdoodle.embed.mongo.transitions.ProcessDefaults$$Lambda/0x00007ef0a4798b10@485b39d6}, State(Name)=Name(mongod), State(PersistentDir)=ImmutablePersistentDir(/home/circleci/.embedmongo), State(DatabaseDir)=DatabaseDir(/tmp/temp--67247ae2-d27d-4193-ba58-93fdeb535042/mongod-database7912467977103391918), State(TempDir)=ImmutableTempDir(/tmp/temp--67247ae2-d27d-4193-ba58-93fdeb535042), State(DistributionBaseUrl)=DistributionBaseUrl(https://fastdl.mongodb.org), State(Platform)=Platform{operatingSystem=Linux, architecture=X86_64, distribution=Ubuntu, version=Ubuntu_22_04}, State(ProcessConfig)=ProcessConfig{daemonProcess=false, stopTimeoutInMillis=5000}, State(ProcessEnv)=ProcessEnv({}), State(ProcessOutput)=ProcessOutput{output=de.flapdoodle.embed.process.io.Slf4jStreamProcessor@d0a2a80, error=de.flapdoodle.embed.process.io.Slf4jStreamProcessor@2fc21fab, commands=de.flapdoodle.embed.process.io.NamedOutputStreamProcessor@5618e97c}, State(StartTimeout)=StartTimeout(30000), State(SystemEnv)=SystemEnv({ ... _=/opt/apache-maven/bin/mvn}), State(ProgressListener)=de.flapdoodle.embed.process.io.progress.Slf4jProgressListener@471cffe0, State(Command)=MongoD, State(Version)=GenericFeatureAwareVersion{7.0.12}, State(Net)=Net{bindIp=127.0.0.1, port=44751, isIpv6=false}, State(MongodArguments)=MongodArguments{syncDelay=0, useDefaultSyncDelay=false, isVerbose=false, verbosityLevel=1, isQuiet=false, useNoPrealloc=true, useSmallFiles=true, useNoJournal=true, enableTextSearch=false, auth=false, master=false, isConfigServer=false, isShardServer=false, params={}, args={}} ... 2024-11-18 17:36:54.845GMT ERROR [main] [-|-@-] org.springframework.boot.SpringApplication: Application run failed java.lang.IllegalArgumentException: archive for https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz:TGZ already exists with different content (/home/circleci/.embedmongo/archives/https-fastdl-mongodb-org/56cb3d10404e5b9df5358ac4ee83efcb/linuxmongodb-linux-x86-64-ubuntu2204-7-0-12-tgz/e750990c2772558d2ce48ad7dc444a746e06938085b01eac9c8c3a07cc875e3d/archive.tgz) at de.flapdoodle.checks.Preconditions.checkArgument(Preconditions.java:32) at de.flapdoodle.embed.process.store.LocalDownloadCache.checkArgument(LocalDownloadCache.java:103) ... 156 common frames omitted Wrapped by: java.io.IOException: archive for https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu2204-7.0.12.tgz:TGZ already exists with different content (/home/circleci/.embedmongo/archives/https-fastdl-mongodb-org/56cb3d10404e5b9df5358ac4ee83efcb/linuxmongodb-linux-x86-64-ubuntu2204-7-0-12-tgz/e750990c2772558d2ce48ad7dc444a746e06938085b01eac9c8c3a07cc875e3d/archive.tgz) at de.flapdoodle.embed.process.store.LocalDownloadCache.checkArgument(LocalDownloadCache.java:105) at de.flapdoodle.embed.process.store.LocalDownloadCache.store(LocalDownloadCache.java:85) at de.flapdoodle.embed.process.transitions.DownloadPackage.lambda$result$5(DownloadPackage.java:158) at de.flapdoodle.types.ThrowingSupplier.lambda$mapException$0(ThrowingSupplier.java:30) ... 153 common frames omitted Wrapped by: java.lang.IllegalArgumentException: could not store downloaded artifact at de.flapdoodle.embed.process.transitions.DownloadPackage.lambda$result$6(DownloadPackage.java:159) at de.flapdoodle.types.ThrowingSupplier.lambda$mapException$0(ThrowingSupplier.java:35) at de.flapdoodle.embed.process.transitions.DownloadPackage.result(DownloadPackage.java:160) at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:58) at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:46) at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:180) ... 149 common frames omitted ```
michaelmosmann commented 4 days ago

@Geminist @ms1111 ..hmm.. Files.move does not solve this either.. i have to take a look at this..