Could not download embedded mongo from custom download path cause URL mismatch

[nandanibansal]

We were using flapdoodle 2.2.0 before, we used to give the custom path like this -

MongodStarter starter = MongodStarter
        .getInstance(new RuntimeConfigBuilder()
          .defaults(Command.MongoD)
          .artifactStore(new ExtractedArtifactStoreBuilder()
            .defaults(Command.MongoD)
            .download(new DownloadConfigBuilder()
              .defaults()
              .packageResolver(new MongoDownloadPath(Command.MongoD))
              .downloadPath(downLoadPath)))
          .build());
      log.info("mongodb starter config = {}"

Now we need mongo 7.0.5 and ubuntu_22 support, so we migrated to flapdoodle 4.13.0, here we are not able to find hot to give particular download path.

MONGO = Mongod.instance()
              .withMongodArguments(
                      Start.to(MongodArguments.class)
                              .initializedWith(
                                      MongodArguments.defaults()
                                              .withUseNoPrealloc(false)
                                              .withUseSmallFiles(false)
                              ))
         .withNet(Start.to(Net.class).initializedWith(Net.builder().bindIp(bindIp).port(port).isIpv6(Network.localhostIsIPv6()).build()))
              .start(Versions.withFeatures(Version.of("7.0.5")));

we customized the mongod download url with this guide: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/blob/main/docs/Customizations.md#customize-download-url

We are getting this error: {We are giving url without username and password} Caused by: java.lang.IllegalArgumentException: parts missing: 'https://mydomain/osx/mongodb-macos-arm64-7.0.5.tgz' != 'UrlParts{protocol='https', userInfo='null', host='artifacts.com', path='/artifactory/.../osx/mongodb-macos-arm64-7.0.5.tgz'}'

It is adding this extra /osx/mongodb-macos-arm64-7.0.5.tgz in our path.

[michaelmosmann]

@nandanibansal there are two options: the one you chosed is the easiest one: you just changed the base path to the artifacts.. to make this work, the required artifacts must be stored in the same path as in the official mongodb artifact repository ..

If this is not an option, you have to implement your own package resolver.. this is more complicated, here you can find some guidance for that: https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/blob/main/docs/Customizations.md#customize-package-resolver

[nandanibansal]

@michaelmosmann Thanks for the above reference. I tried using this, but getting this error

"Caused by: java.lang.IllegalArgumentException: parts missing: 'https://{host}:{port}/...mongodb-macos-x86_64-7.0.5.RELEASE.tgz' != 'UrlParts{protocol='https', userInfo='null', host='{host}, path='...mongodb-macos-x86_64-7.0.5.RELEASE.tgz'}'",

I checked the code:

@VisibleForTesting
    static UrlParts partsOf(URL url) {
        boolean portIsPartOfTheUrl = url.getPort() != -1 && url.getPort() != url.getDefaultPort();

        return UrlParts.of(
            url.getProtocol(),
            url.getUserInfo(),
            url.getHost().isEmpty() ? null : url.getHost() + (portIsPartOfTheUrl ? ":" + url.getPort() : ""),
            pathPart(url));
    }

the value of portIsPartOfTheUrl is coming false because my port is same as default port.

[nandanibansal]

Solve this by removing port from my url. It takes the default port and is able to download the embedded mongo.

[nandanibansal]

@michaelmosmann I am getting below error while running my test cases in Docker.

Caused by: java.lang.RuntimeException: could not start process
Caused by: java.io.IOException: Cannot run program "/root/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904/mongod" (in directory "/tmp/temp--9ba1a91e-ab50-494f-8c4c-a64649c90cdd/workingDir4540246169661976121"): error=8, Exec format error
Caused by: java.io.IOException: error=8, Exec format error

[michaelmosmann]

@nandanibansal do you use the same binary for docker? can you provide some more logging?

[nandanibansal]

@michaelmosmann

[mongod error]/mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904/mongod: /mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904/mongod: cannot execute binary file

17:58:02 [mongod error] 17:58:21 {"@timestamp":"2024-07-03T12:28:21.457Z", "log.level":"ERROR", "message":" MDMMongoDBStarter initMongDB ", "service.name":"","process.thread.name":"main","log.logger":"MongoConfig","log.origin":{"file.name":"MongoConfig.java","function":"initMongod","file.line":113},"error.type":"java.lang.RuntimeException","error.message":"rollback after error on transition to State(RunningMongodProcess), successful reached:\n State(ExtractedFileSet)=ExtractedFileSet{baseDir=/mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904, executable=/mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904/mongod, libraryFiles=[]},\n State(Archive)=Archive(/mnt/looper/.embedmongo/archives/https-artifacts-linux-x86-64-ubuntu220417-0-5-RELEASEmongodb-linux-x86-64-ubuntu22041-7-0-5-RELEASE-tgz/bf3ebfb3a90a9cf22b5a940f06c311ce18943a01ac5ef8c5a227eefbc57c710c/archive.tgz),\n State(ProcessOutput)=ProcessOutput{output=de.flapdoodle.embed.process.io.NamedOutputStreamProcessor@6fe13392, error=de.flapdoodle.embed.process.io.NamedOutputStreamProcessor@5c31d332, commands=de.flapdoodle.embed.process.io.ConsoleOutputStreamProcessor@6e7e8426},\n State(DownloadCache)=de.flapdoodle.embed.process.store.LocalDownloadCache@32a62e00,\n State(Package)=Package{archiveType=TGZ, fileSet=FileSet{entries=[Entry{type=Executable, destination=mongod, uncompiledMatchingPattern=UncompiledPattern{regex=.mongod, flags=2}}]}, url=https://artifactsmongodb-linux-x86_64-ubuntu22041/7.0.5.RELEASE/mongodb-linux-x86_64-ubuntu22041-7.0.5.RELEASE.tgz},\n State(ExtractedFileSetStore)=de.flapdoodle.embed.process.store.ContentHashExtractedFileSetStore@543de665,\n State(ProcessArguments)=ProcessArguments([--dbpath, /tmp/temp--fd39fb24-f542-4a29-9d87-33b04fc086be/mongod-database5299636114934475042, --noauth, --port, 4705, --bind_ip, 127.0.0.1, --syncdelay=0]),\n State(ProcessWorkingDir)=ProcessWorkingDir(/tmp/temp--fd39fb24-f542-4a29-9d87-33b04fc086be/workingDir5836062452443925821),\n State(Distribution)=GenericFeatureAwareVersion{7.0.5}:Platform{operatingSystem=Linux, architecture=X86_64, distribution=Ubuntu, version=Ubuntu_22_04},\n State(SupportConfig)=SupportConfig{name=mongod, supportUrl=https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/issues, messageOnException=de.flapdoodle.embed.mongo.transitions.ProcessDefaults$$Lambda$910/1263738407@1d0988ca},\n State(Name)=Name(mongod),\n State(PersistentDir)=ImmutablePersistentDir(/mnt/looper/.embedmongo),\n State(DatabaseDir)=DatabaseDir(/tmp/temp--fd39fb24-f542-4a29-9d87-33b04fc086be/mongod-database5299636114934475042),\n State(TempDir)=ImmutableTempDir(/tmp/temp--fd39fb24-f542-4a29-9d87-33b04fc086be),\n State(DistributionBaseUrl)=DistributionBaseUrl(https://artifacts.mongodb/),\n State(Platform)=Platform{operatingSystem=Linux, architecture=X86_64, distribution=Ubuntu, version=Ubuntu_22_04},\n State(ProcessConfig)=ProcessConfig{daemonProcess=false, stopTimeoutInMillis=5000},\n State(ProcessEnv)=ProcessEnv(null),\n State(SystemEnv)=SystemEnv({PATH=/mnt/looper/workspace/tools/nix_64/jdk-1.8.0_161-b12/bin:/mnt/looper/workspace/tools/nix_64/maven-3.6.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin, NO_PROXY=.,cdn.cocoapods.org,-keystone-endpoint.net,slack.com,.blob.core.windows.net,mockDeliveryUrl,ondemand.saucelabs.com,jira.com,.prod.us.net,.googleapis.com,euclid.azurecr.io,xmatters.com,sandbox-cluboperations-claims.azurewebsites.net,accounts.google.com,usgta.wal-mart.com,metadata.google.,cloud.google.com,ossindex.sonatype.org,.azure-api.net,dev-transpo-fresh-pullforward-aggregator.azurewebsites.net/actuator/health,samsclub.riversand.com,sb.scorecardresearch.com,i.imgur.com, LOOPER_FLOW_TYPE=BRANCH, REPOSOLNS_PYPI_REPO=https://pypi.ci.artifacts, agent_name=looperpro-prod-agent131-23, BASEDIR=/mnt/looper/workspace, RUN_DISPLAY_URL=https://runner-1-2175078775.looperpro-prod-runner02.prod-ndc23.com/job/13/display/redirect, HUDSON_HOME=/looper/looper-workspace, RUN_CHANGES_DISPLAY_URL=https://runner-1-2175078775.looperpro-prod-runner02.prod-ndc23.prod.com/job/redirect?page=changes, REPOSOLNS_MVN_RELEASE_REPO=https://mvn.ci.artifacts.com/artifactory/ocal, REPOSOLNS_FQDN_CI=ci.artifacts.com, REPOSOLNS_NUGET_REPO=https://nuget.ci.artifacts.com/artifactory/api/nuget-nuget, WORKDIR=/mnt/looper/workspace/ reposolnsUsername=reposolns, REPOSOLNS_GENERIC_SNAPSHOT_REPO=https://generic.ci.artifacts.com/artifactorylocal, NPM_REGISTRY_PRIMARY=https://npme.com/, JENKINS_SERVER_COOKIE=cad2bf9e97974601, PWD=/mnt/looper/workspace/ws, pipelineCfg={{managedNamespace = {baseName = 'newbets-hnw-cprm', tempName = n2138074129, managedNS = true, managedTTL = 24, servicNowGroup = ', productId = 887, apmId = APM0006865, assignments = [{cluster = 'scus-dev-a5', clusterGroup = {name = '', namespace = ''}, cpu_limit = 24, memory_limit = 48, environment = dev, region = scus, 'namespace-reaper-ttl' = 0, 'namespace-reaper-managed' = false, enableIstio = false, provision_process = {id = '', status = FINISHED}, tenantGroup = ''}, {cluster = 'useast-stage-az-303', clusterGroup = {name = '', namespace = ''}, cpu_limit = 18, memory_limit = 32, environment = stage, region = eus2, 'namespace-reaper-ttl' = 0, 'namespace-reaper-managed' = false, enableIstio = false, provision_process = {id = '', status = FINISHED}, tenantGroup = ''}, {cluster = 'uscentral-stage-az-301', clusterGroup = {name = '', namespace = ''}, cpu_limit = 12, memory_limit = 24, environment = stage, region = scus, 'namespace-reaper-ttl' = 0, 'namespace-reaper-managed' = false, enableIstio = false, provision_process = {id = '', status = FINISHED}, tenantGroup = ''}, {cluster = 'useast-prod-az-330', clusterGroup = {name = '', namespace = ''}, cpu_limit = 32, memory_limit = 128, environment = prod, region = eus2, 'namespace-reaper-ttl' = 0, 'namespace-reaper-managed' = false, enableIstio = false, provision_process = {id = '', status = FINISHED}, tenantGroup = ''}, {cluster = 'uscentral-prod-az-340', clusterGroup = {name = '', namespace = ''}, cpu_limit = 32, memory_limit = 128, environment = prod, region = scus, 'namespace-reaper-ttl' = 0, 'namespace-reaper-managed' = false, enableIstio = false, provision_process = {id = '', status = FINISHED}, tenantGroup = ''}], mlsIndex = 'mls-prosorinos-index2', mlsCluster = wcnp, enableIstio = false, hubGslbManagedNamespace = true, baseNsAdGroup = wmt_hnw_cprm_devs, isBaseNamespacePortalApiManaged = true, complianceLevel = general, prodSecretsOwnerGroup = wmt_hnw_cprm_devs, nonProdSecretsOwnerGroup = wmt_hnw_cprm_devs, enableHelm3 = false}, build = {language = Java, group = version = '0.0.1-28b64ff58', HOME=/mnt/looper}),\n State(ProgressListener)=de.flapdoodle.embed.process.io.progress.StandardConsoleProgressListener@6470793,\n State(Command)=MongoD,\n State(Version)=GenericFeatureAwareVersion{7.0.5},\n State(Net)=Net{bindIp=localhost, port=4705, isIpv6=false},\n State(MongodArguments)=MongodArguments{syncDelay=0, useDefaultSyncDelay=false, isVerbose=false, verbosityLevel=1, isQuiet=false, useNoPrealloc=false, useSmallFiles=false, useNoJournal=true, enableTextSearch=false, auth=false, master=false, isConfigServer=false, isShardServer=false, params={}, args={}}\n","error.stack_trace":[

[michaelmosmann]

@nandanibansal Can you execute the binary /mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904/mongod inside your docker container?

[nandanibansal]

@michaelmosmann I tried looking for the mongod binary inside the /mnt/looper/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904

ARG mavenVersion=3-jdk-8-alpine
ARG zuluJDKVersion=8-jdk-alpine
ARG scannerVersion=0.0.26
ARG baseRepoBranch=main

##Stage 0 certs stage
FROM docker.ci.artifacts.com/wce-docker/ca-roots:latest as roots

##Stage 1 maven build
FROM docker.ci.artifacts.com/hub-docker-release-remote/maven:${mavenVersion} AS build
ARG kittBuildVersion=0.0.1-SNAPSHOT
ARG mvnGoals="clean install -Dtest="ClaimOverrideServiceIntegrationTest" -Dmaven.test.failure.ignore=true"
WORKDIR /app
RUN curl -SLk https://repository.com/content/repositories/thirdparty/settings.xml -o ${MAVEN_HOME}/conf/settings.xml
COPY . .

RUN mvn versions:set -DnewVersion=${kittBuildVersion}
RUN mvn -B -e -f /app/pom.xml ${mvnGoals} || true

RUN ls -l /root/.embedmongo/fileSets
RUN ls -l /root/.embedmongo/fileSets/a2e5309c8465ece8427865fc6a37e49cbfea639352752c9cab799b58c1d62904 

There is no file that exists inside this folder.

[nandanibansal]

Here is my code for reference.

  private TransitionWalker.ReachedState<RunningMongodProcess> MONGO;

  @Bean
  public MongoTemplate mongoTemplate() {
    return new MongoTemplate(MongoClients.create("mongodb://"+bindIp+":"+port), database);
  }

  @PostConstruct
  public void initMongod() {
    try {
      MONGO = Mongod.instance()
              .withDistributionBaseUrl(Start.to(DistributionBaseUrl.class).initializedWith(DistributionBaseUrl.of(downLoadPath+versionPath)))
              .withPackageOfDistribution(PackageOfCommandDistribution.withDefaults()
                .withCommandPackageResolver(command -> distribution -> mongodPackageRules.packageFor(distribution).orElseThrow(() -> new IllegalArgumentException("could not find package for " + distribution))
                ))
              .withMongodArguments(
                      Start.to(MongodArguments.class)
                              .initializedWith(
                                      MongodArguments.defaults()
                                              .withUseNoPrealloc(false)
                                              .withUseSmallFiles(false)
                              ))
              .withNet(Start.to(Net.class).initializedWith(Net.builder().bindIp(bindIp).port(port).isIpv6(Network.localhostIsIPv6()).build()))

              .start(Versions.withFeatures(Version.of("7.0.5")));
    } catch (Exception e) {
      log.error(" MDMMongoDBStarter initMongDB ", e);
      throw new RuntimeException(e);
    }
  }

  PackageFinderRules mongodPackageRules = PackageFinderRules.builder()
          .addRules(PackageFinderRule.builder()
                  .match(PlatformMatch.withOs(CommonOS.Linux)
                          .andThen(DistributionMatch.any(VersionRange.of("5.0.0", "7.1.0"))
                          ))
                  .finder(UrlTemplatePackageFinder.builder()
                          .fileSet(FileSet.builder()
                                  .addEntry(FileType.Executable, "mongod")
                                  .build())
                          .archiveType(ArchiveType.TGZ)
                          .urlTemplate(linuxOSVersion)
                          .build())
                  .build())
          .addRules(PackageFinderRule.builder()
                  .match(PlatformMatch.withOs(CommonOS.OS_X)
                          .andThen(DistributionMatch.any(VersionRange.of("5.0.0", "7.1.0"))
                          )).
                  finder(UrlTemplatePackageFinder.builder()
                          .fileSet(FileSet.builder()
                                  .addEntry(FileType.Executable, "mongod")
                                  .build())
                          .archiveType(ArchiveType.TGZ)
                          .urlTemplate(macOSVersion)
                          .build())
                  .build())
          .build();

[michaelmosmann]

@nandanibansal I think you must login into the running docker container and then search for this file.. but as I saw some details I guess this will not work if you use alpine based images .. because there is no mongodb version for alpine.. you have to use at least some more common.. like Ubuntu, Debian, redhat... or the same like your production environment.

[nandanibansal]

@michaelmosmann Thank, it is working now.