Problem with migration from spring 2.7.1 to 3.1.1

[Dharmik2510]

I was using de.flapdoodle.embed:de.flapdoodle.embed.mongo:3.5.0

I have created EmbeddedMongoServer class, where I am importing MongoProcess, MongodStarter, Defaults, MongodConfig RuntimeConfig, MongodExecutable

However, I want to upgrade my internal mongo library from springboot 2.7 to 3.1.1 and when I replaced flapdoodle to flapdoodle.embed.mongo.spring30x:4.5.2 It can not find the above mentioned classes/interfaces. How can I successfully migrate ?

I am trying with flapdoodle.embed.mongo.spring30x:4.5.2

[michaelmosmann]

@Dharmik2510 with de.flapdoodle.embed.mongo:4.x.x started a heavy rewritten version of this library.. so MongoProcess, MongodStarter etc is gone.. can you show me your EmbeddedMongoServer class to see what you are trying to do? I guess it should be not that hard to change it.

[Dharmik2510]

I have MongoServer.java class where this is what I am doing currently,

import de.flapdoodle.embed.mongo.MongodExecutable; import de.flapdoodle.embed.mongo.MongodProcess; import de.flapdoodle.embed.mongo.MongodStarter; import de.flapdoodle.embed.mongo.config.Defaults; import de.flapdoodle.embed.mongo.config.MongodConfig; import de.flapdoodle.embed.mongo.config.Net; import de.flapdoodle.embed.mongo.distribution.Version; import de.flapdoodle.embed.mongo.packageresolver.Command; import de.flapdoodle.embed.process.config.RuntimeConfig; import de.flapdoodle.embed.process.config.store.DistributionPackage; import de.flapdoodle.embed.process.config.store.FileSet; import de.flapdoodle.embed.process.config.store.FileType; import de.flapdoodle.embed.process.distribution.ArchiveType; import org.apache.commons.lang3.StringUtils;

import java.io.IOException;

public class MongoServer { private MongodProcess mongodProcess; private MongodExecutable mongodExecutable; private Integer port; private String ip;

public MongoServer() throws IOException {
    this("localhost", 27017, null, null);
}

public MongoServer(int port) throws IOException {
    this("localhost", port, null, null);
}

public MongoServer(String ip, Integer port, String url, String archivePath) throws IOException {
    this.port = port;
    this.ip = ip;

    MongodConfig mongodConfig = MongodConfig.builder()
            .version(Version.Main.V4_4)
            .net(new Net(this.ip, this.port, false))
            .build();
    MongodStarter starter;
    if(StringUtils.isNotBlank(url) && StringUtils.isNotBlank(archivePath)) {
        Command command = Command.MongoD;
        RuntimeConfig runtimeConfig = Defaults.runtimeConfigFor(command)
                .artifactStore(Defaults.extractedArtifactStoreFor(command)
                        .withDownloadConfig(Defaults.downloadConfigFor(command)
                                .downloadPath((__) -> url)
                                .packageResolver(distribution ->
                                        DistributionPackage.of(
                                                ArchiveType.TGZ,
                                                FileSet.builder().addEntry(FileType.Executable, command.commandName()).build(),
                                                archivePath
                                                ))
                                .build()))
                .build();
        starter = MongodStarter.getInstance(runtimeConfig);
    } else {
        starter = MongodStarter.getDefaultInstance();
    }

    try {
        this.mongodExecutable = starter.prepare(mongodConfig);
        this.mongodProcess = this.mongodExecutable.start();
    } catch (Throwable t) {
        stop();
        throw t;
    } finally {
    }
}

public void stop() {
    if (this.mongodExecutable != null) {
        this.mongodExecutable.stop();
    }
    if (this.mongodProcess != null) {
        this.mongodProcess.stop();
    }
}

@Override
protected void finalize() throws Throwable {
    super.finalize();
    stop();
}

public int getPort() {
    return port;
}

public String getIp() {
    return ip;
}

}

[michaelmosmann]

@Dharmik2510 where does url and archivePath points to (sample values) .. and why do you do this?

[Dharmik2510]

@michaelmosmann, we have created an internal library for mongo which is used by different services in our project.

url: It represents the download source of the MongoDB distribution. In the code snippet, it is used to set the download path for the artifact. I am specifying a custom location from where the MongoDB distribution will be downloaded.

archivePath: It represents the location where the MongoDB distribution will be stored locally. It specifies the path where the downloaded artifact will be extracted and used to run the MongoDB server.

[michaelmosmann]

@Dharmik2510

This is as close as i think i understand your usecase:

import de.flapdoodle.embed.mongo.config.Net;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.mongo.transitions.Mongod;
import de.flapdoodle.embed.mongo.transitions.RunningMongodProcess;
import de.flapdoodle.embed.mongo.types.DistributionBaseUrl;
import de.flapdoodle.embed.process.io.directories.PersistentDir;
import de.flapdoodle.reverse.TransitionWalker;
import de.flapdoodle.reverse.transitions.Start;

import java.nio.file.Paths;

public class MongoServer {
    private final TransitionWalker.ReachedState<RunningMongodProcess> runningMongod;
    private final Integer port;
    private final String ip;

    public MongoServer() {
        this("localhost", 27017, null, null);
    }

    public MongoServer(int port) {
        this("localhost", port, null, null);
    }

    public MongoServer(String ip, Integer port, String url, String archivePath) {
        this.port = port;
        this.ip = ip;

        ImmutableMongod.Builder builder = Mongod.builder();

        builder
            .net(Start.to(Net.class).initializedWith(Net.builder()
                .bindIp(ip)
                .port(port)
                .build()));
        if (!url.trim().isEmpty() && !archivePath.trim().isEmpty()) {
            builder.distributionBaseUrl(Start.to(DistributionBaseUrl.class)
                    .initializedWith(DistributionBaseUrl.of(url)))
                .persistentBaseDir(Start.to(PersistentDir.class)
                    .initializedWith(PersistentDir.of(Paths.get(archivePath))));
        }
        Mongod mongod = builder.build();

        this.runningMongod = mongod.start(Version.Main.V4_4);

        //ServerAddress serverAddress = runningMongod.current().getServerAddress();
    }

    public void stop() {
        runningMongod.close();
    }

    @Override
    protected void finalize() throws Throwable {
        super.finalize();
        stop();
    }

    public int getPort() {
        return port;
    }

    public String getIp() {
        return ip;
    }
}

I would recommend to change just the base url and leave the rest untouched.. you if want to specify the archive direct it will get complicated.. and the archive stuff is also a little bit more complicated.. the PersistenDir holds the path to the location where the downloads and the extracted artifacts are stored and cached.. hashed are used to avoid any unwanted collisions ..

If you try this just ask if any further questions..

[Dharmik2510]

Thanks for this @michaelmosmann,

I tried this modified code. The library is compiling and building successfully.

I want to just make sure that the behaviour remains intact. Do you think that modifying the existing code with yours will change some behaviour?

[michaelmosmann]

@Dharmik2510 not the behavior of the mongodb..

[Dharmik2510]

Many of my test cases are failing. For your question of what might url and archive path look like:

Here it is: Url: Our Nexus url archivePath == downloadUrlPath == "linux/mongodb-linux-x86_64-ubuntu1804-4.2.0.tgz"

Also while running test cases it throws: one exception: java.lang.IllegalStateExceptions: can not build Net, some of required attributes are not set [isIpv6]

[michaelmosmann]

@Dharmik2510 ah.. ok.. try this:

import de.flapdoodle.embed.mongo.config.Net;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.mongo.transitions.ImmutableMongod;
import de.flapdoodle.embed.mongo.transitions.Mongod;
import de.flapdoodle.embed.mongo.transitions.RunningMongodProcess;
import de.flapdoodle.embed.mongo.types.DistributionBaseUrl;
import de.flapdoodle.os.*;
import de.flapdoodle.os.linux.LinuxDistribution;
import de.flapdoodle.os.linux.UbuntuVersion;
import de.flapdoodle.reverse.TransitionWalker;
import de.flapdoodle.reverse.transitions.Start;

public class MongoServer {
    private final TransitionWalker.ReachedState<RunningMongodProcess> runningMongod;
    private final Integer port;
    private final String ip;

    public MongoServer() {
        this("localhost", 27017, null, null);
    }

    public MongoServer(int port) {
        this("localhost", port, null, null);
    }

    public MongoServer(String ip, Integer port, String url, String archivePath) {
        this.port = port;
        this.ip = ip;

        ImmutableMongod.Builder builder = Mongod.builder();

        builder
            .net(Start.to(Net.class).initializedWith(Net.builder()
                .bindIp(ip)
                .isIpv6(false)
                .port(port)
                .build()));
        if (!url.trim().isEmpty()) {
            builder.distributionBaseUrl(Start.to(DistributionBaseUrl.class)
                    .initializedWith(DistributionBaseUrl.of(url)));
            builder.platform(Start.to(de.flapdoodle.os.Platform.class)
                .initializedWith(ImmutablePlatform.builder()
                    .operatingSystem(CommonOS.Linux)
                    .architecture(CommonArchitecture.X86_64)
                    .distribution(LinuxDistribution.Ubuntu)
                    .version(UbuntuVersion.Ubuntu_18_04)
                    .build()));
        }
        Mongod mongod = builder.build();

        this.runningMongod = mongod.start(Version.Main.V4_4);

        //ServerAddress serverAddress = runningMongod.current().getServerAddress();
    }

    public void stop() {
        runningMongod.close();
    }

    @Override
    protected void finalize() throws Throwable {
        super.finalize();
        stop();
    }

    public int getPort() {
        return port;
    }

    public String getIp() {
        return ip;
    }
}

But maybe you should not force the platform to ubuntu 18.04 .. so you should try it without this code first:

builder.platform(Start.to(de.flapdoodle.os.Platform.class)
                .initializedWith(ImmutablePlatform.builder()
                    .operatingSystem(CommonOS.Linux)
                    .architecture(CommonArchitecture.X86_64)
                    .distribution(LinuxDistribution.Ubuntu)
                    .version(UbuntuVersion.Ubuntu_18_04)
                    .build()));

[Dharmik2510]

Thanks @michaelmosmann , I tried with and without the last code you provided. With builder.platform() there is an issue with initializedWith(ImmutablePlatform.builder() line as my program execution gets finished there and without that .

program execution goes till this.runningMongod = mongod.start(Version.Main.V4_4); and then while debugging I noticed it's starting the server as well.

Getting this as well : java.lang.NoSuchMethodError: 'de.flapdoodle.os.common.OneOf de.flapdoodle.os.common.OneOf.of(de.flapdoodle.os.common.DistinctPeculiarity[])'

Now I am facing another problem. In my test case I am doing: embededServer = new MongoServer(ip, port, downloadUrl, path);

if(mongoTemplate == null){ mongoTemplate = new MongoTemplate(MongoClients.create("mongodb: // %s:%d", embededServer.getIp(), embededServer.getPort()));

But my program return template == null and somehow it returns from below line to a parent call embededServer = new MongoServer(ip, port, downloadUrl, path);

Would it be possible for you to suggest me which flapdoodle dependencies I need to include in my build.gradle file ?

Thanks :)

[michaelmosmann]

@Dharmik2510

java.lang.NoSuchMethodError: 'de.flapdoodle.os.common.OneOf

ok.. i will make a new release of the spring adapter.. with this exeception an instance of MongoServer can not be created.. because the exception is in the constructor..

i think i can make a new release today..

[michaelmosmann]

Did you try the latest release de.flapdoodle.embed:de.flapdoodle.embed.mongo:4.7.1

[Dharmik2510]

I tried version 4.7.1: Now different Exception: IllegalStateException: Could not download "From nexus repository/linux/mongodb-linux-x86_64-ubuntu1804-4.4.18tgz"

[Dharmik2510]

Would it be possible for you to give me a replacement for the below code as issue is in that part only:

MongodStarter starter; if(StringUtils.isNotBlank(url) && StringUtils.isNotBlank(archivePath)) { Command command = Command.MongoD; RuntimeConfig runtimeConfig = Defaults.runtimeConfigFor(command) .artifactStore(Defaults.extractedArtifactStoreFor(command) .withDownloadConfig(Defaults.downloadConfigFor(command) .downloadPath((__) -> url) .packageResolver(distribution -> DistributionPackage.of( ArchiveType.TGZ, FileSet.builder().addEntry(FileType.Executable, command.commandName()).build(), archivePath )) .build())) .build(); starter = MongodStarter.getInstance(runtimeConfig);

Where, url = "https://{to nexus repository}" archivePath = "linux/mongodb-linux-x86_64-ubuntu1804-4.4.18tgz"" (which is again inside nexus repo)

Thanks :)

[michaelmosmann]

@Dharmik2510 can you give me a more detailed error message? is there any cause why the download failed as it looks like that the path for the download is as expected.. ?

[Dharmik2510]

@michaelmosmann , As long as the download URL is concerned it's correct because when I click from my IDE on that URL, it downloads .tgz file.

When I run the program the URL becomes {https://nexus-repo-path/linux/mongodb-linux-x86_64-ubuntu1804-4.4.18tgz/linux/mongodb-linux-x86_64-ubuntu2004-4.4.18tgz}, I don't know why it appends the last .tgz archive

The exception I am getting is: caused by: java.lang.IllegalStateException: could not download from {url} and another one caused by: java.io.FileNotFoundException: {pointing to url}

[Dharmik2510]

I have pasted a problem here as well. It will look more structured: https://stackoverflow.com/questions/76723312/problem-while-downloding-tgz-from-our-nexus-repo-instead-of-local-path-using-em

[michaelmosmann]

@Dharmik2510 replace

builder.distributionBaseUrl(Start.to(DistributionBaseUrl.class)
                    .initializedWith(DistributionBaseUrl.of("https://nexus-repo-path/linux/mongodb-linux-x86_64-ubuntu1804-4.4.18tgz")))

with

builder.distributionBaseUrl(Start.to(DistributionBaseUrl.class).initializedWith(DistributionBaseUrl.of("https://nexus-repo-path/")))

It is the base path where all distributions can be found.. the last part of the path is resolved in a different location..

[Dharmik2510]

@michaelmosmann, I went through internals of library and used,

PackageDistribution packageofDistribution = PackageDistribution.with(dist-> Package.builder()).archiveType(ArchiveType.TGZ).fileSet(FileSet.builder().
addEntry(FileType.Executable, command.commandName().build()))
.url(url+"/"+archivePath).build());

builder.packageOfDistribution(packageofDistribution);

And it worked for me. Is this the correct approach?

[michaelmosmann]

@Dharmik2510 yes.. you replaced the part were the url based on OS, Cpu and Version is created..

[Dharmik2510]

So this means if my code is working locally and in GitLab, this should work in production as well. Right ?. Do you have ay better solution than this one ? I would like to test that as well in my code !!!

[michaelmosmann]

@Dharmik2510 ok.. to be sure.. my library downloads a real mongodb matching your OS, starts it up, tear it down.. so any test you are writing runs on this real mongodb.. if the same code runs in production on the same version of the mongodb database i would expect that it works as expected..

To store the mongodb distributions on your own nexus is total valid.. but i would recommend to change only the distribution base path and not the PackageDistribution implementation because as soon as someone with for instance a different OS comes along you have to change this.. or you need a newer mongodb version..

.. but anything thats works is valid.. it is your project:)

[Dharmik2510]

@michaelmosmann , Thanks for elaboration. I tried DistributionbasePath but it's automatically taking distribution /linux/ubuntu.... which is on my local VM not one present in our nexus repository.

[michaelmosmann]

Yes.. the paths are the one from the official mongodb download server.. you have to put the artifacts in the same location.. (only the needed one).

[Dharmik2510]

Okay thanks a lot :). I will close the issue for now.