Could not run in gitlab-runner

[michaelmosmann]

          The strange thing is, that it works locally (Ubuntu 22 and 24) and locally in a docker container, e.g. with image maven:3.9-eclipse-temurin-17 but as soon as it is run in the gitlab pipeline (even with docker) or the gitlab virtual machine (Debian 12) itself it fails to start

#11 138.4 Caused by: java.lang.RuntimeException: Could not start process: 
#11 138.4 ----------------------
#11 138.4 Hmm.. no failure or success message after 60000ms .. 
#11 138.4 ...the cause must be somewhere in the process output:
#11 138.4 - 8<- - - - - - - - - -
#11 138.4 
#11 138.4 
#11 138.4 
#11 138.4 - >8- - - - - - - - - -
#11 138.4 
#11 138.4   at de.flapdoodle.embed.mongo.transitions.RunningMongoProcess.lambda$null$1(RunningMongoProcess.java:180) ~[de.flapdoodle.embed.mongo-4.1

Originally posted by @F-Node-Karlsruhe in https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues/413#issuecomment-2188813669

[F-Node-Karlsruhe]

gitlab shell runner 16.11.1 on Linux gitlab 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 Docker build with image maven:3.9-eclipse-temurin-17

  <dependency>
    <groupId>de.flapdoodle.embed</groupId>
    <artifactId>de.flapdoodle.embed.mongo.spring3x</artifactId>
    <version>4.14.0</version>
    <scope>test</scope>
    <optional>true</optional>
  </dependency>

[F-Node-Karlsruhe]

It also happens on other debian 12 virtual machines when just building with docker

[michaelmosmann]

@F-Node-Karlsruhe do you have some more logging information?

When you see Hmm.. no failure or success message after 60000ms .. in your log, then the binary did behave different than expected.. i inspect the process log to see if there is some error message or some success message (open port .. bla bla bla). If both did not happen, i kill everything after one minute.

So everthing above these lines may give a hint...

[F-Node-Karlsruhe]

afaiu. the not started process is the root cause. i played a little with the mongodb versions. For 4 and 5 for instance therre is an error where the mongodb binary complains about a missing libcrypto1.1 but for 6 and 7 it fails silently

#11 78.12   State(ProgressListener)=de.flapdoodle.embed.process.io.progress.Slf4jProgressListener@37a0ec3c,
#11 78.12   State(Command)=MongoD,
#11 78.12   State(Version)=GenericFeatureAwareVersion{7.0.11},
#11 78.12   State(Net)=Net{bindIp=127.0.0.1, port=36149, isIpv6=false},
#11 78.12   State(MongodArguments)=MongodArguments{syncDelay=0, useDefaultSyncDelay=false, isVerbose=false, verbosityLevel=1, isQuiet=false, useNoPrealloc=true, useSmallFiles=true, useNoJournal=true, enableTextSearch=false, auth=false, master=false, isConfigServer=false, isShardServer=false, params={}, args={}}
#11 78.12 
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:190) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:69) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:65) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   at de.flapdoodle.embed.mongo.spring.autoconfigure.MongodWrapper.start(MongodWrapper.java:45) ~[de.flapdoodle.embed.mongo.spring3x-4.14.0.jar:na]
#11 78.12   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
#11 78.12   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
#11 78.12   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
#11 78.12   at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
#11 78.12   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1892) ~[spring-beans-6.1.8.jar:6.1.8]
#11 78.12   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1845) ~[spring-beans-6.1.8.jar:6.1.8]
#11 78.12   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1784) ~[spring-beans-6.1.8.jar:6.1.8]
#11 78.12   ... 156 common frames omitted
#11 78.12 Caused by: java.lang.RuntimeException: Could not start process: 
#11 78.12 ----------------------
#11 78.12 Hmm.. no failure or success message after 60000ms .. 
#11 78.12 ...the cause must be somewhere in the process output:
#11 78.12 - 8<- - - - - - - - - -
#11 78.12 
#11 78.12 
#11 78.12 
#11 78.12 - >8- - - - - - - - - -
#11 78.12 
#11 78.12   at de.flapdoodle.embed.mongo.transitions.RunningMongoProcess.lambda$null$1(RunningMongoProcess.java:180) ~[de.flapdoodle.embed.mongo-4.14.0.jar:na]
#11 78.12   at de.flapdoodle.types.ThrowingSupplier.lambda$andFinally$1(ThrowingSupplier.java:47) ~[de.flapdoodle.java8-1.4.2.jar:na]
#11 78.12   at de.flapdoodle.types.ThrowingSupplier.lambda$andFinally$1(ThrowingSupplier.java:47) ~[de.flapdoodle.java8-1.4.2.jar:na]
#11 78.12   at de.flapdoodle.embed.mongo.transitions.RunningMongoProcess.lambda$factory$3(RunningMongoProcess.java:186) ~[de.flapdoodle.embed.mongo-4.14.0.jar:na]
#11 78.12   at de.flapdoodle.embed.process.types.RunningProcess.start(RunningProcess.java:83) ~[de.flapdoodle.embed.process-4.11.0.jar:na]
#11 78.12   at de.flapdoodle.embed.mongo.transitions.MongoServerStarter.result(MongoServerStarter.java:127) ~[de.flapdoodle.embed.mongo-4.14.0.jar:na]
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:58) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.resolve(TransitionWalker.java:46) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   at de.flapdoodle.reverse.TransitionWalker.initState(TransitionWalker.java:180) ~[de.flapdoodle.reverse-1.7.2.jar:na]
#11 78.12   ... 166 common frames omitted

[michaelmosmann]

@F-Node-Karlsruhe can you increase the logging for "de.flapdoodle.embed." .. there should be some output from the starting process ..

[michaelmosmann]

@F-Node-Karlsruhe i do not know your spring app setup.. is there any way to run this on your setup: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo.canary

[F-Node-Karlsruhe]

this is the most specific log above the start failure timeout

#11 79.91 Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'mongoDatabaseFactory' defined in class path resource [org/springframework/boot/autoconfigure/data/mongo/MongoDatabaseFactoryConfiguration.class]: Unsatisfied dependency expressed through method 'mongoDatabaseFactory' parameter 0: Error creating bean with name 'syncClientServerWrapper' defined in class path resource [de/flapdoodle/embed/mongo/spring/autoconfigure/EmbeddedMongoAutoConfiguration$SyncClientServerWrapperConfig.class]: rollback after error on transition to State(RunningMongodProcess), successful reached:

flapdoodle.log

[michaelmosmann]

@F-Node-Karlsruhe is there a chance that you can start the downloaded binary on this instance? you can find the location in the log.. from the last log it is

/root/.embedmongo/fileSets/0795fd5091b94992525658a7f10f8d51c8af2f4588091c13d65abccd532b3c40/mongod

[F-Node-Karlsruhe]

./mongod: error while loading shared libraries: libcrypto.so.3: cannot open shared object file: No such file or directory

but this should not be the cause within the docker container (which runs on all local ubuntu machines)

EDIT: This was in Debian 10

On the debian 12 machine where the gitlab runner lives i get

use@gitlab:~/mongodb-linux-x86_64-ubuntu2204-7.0.11/bin$ sudo chmod +x mongod
use@gitlab:~/mongodb-linux-x86_64-ubuntu2204-7.0.11/bin$ sudo ./mongod
Illegal instruction
user@gitlab:~/mongodb-linux-x86_64-ubuntu2204-7.0.11/bin$ 

https://www.mongodb.com/community/forums/t/debian-11-core-dump-help-please/190146/3

[F-Node-Karlsruhe]

Maxbe related to this issue https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues/467

[F-Node-Karlsruhe]

I think we can close the issue as is appears to be a problem of our architecture and the mongo binaries

[michaelmosmann]

@F-Node-Karlsruhe .. did you execute the binary inside the docker container (i would not expect an 'Illegal instruction') ? If your docker is a debian i am surprised that the os detection thinks thats an ubuntu..

can you show me the content of the os-release file of the docker container?

[F-Node-Karlsruhe]

The container is a maven 3.9 container which is based on ubuntu 24.04

when executing the binary there i get

Illegal instruction (core dumped)

[michaelmosmann]

@F-Node-Karlsruhe Does it work if you use a older mongodb version (4.4.x)?

[michaelmosmann]

@F-Node-Karlsruhe can you provide the output of /proc/cpuinfo inside your docker container?