Using Docker images of Jenkins for unit tests

[gnl42]

Following some comments in #203 regarding the state of our current Jenkins server used for the unit tests I wondered if Docker images could be used during the Mylyn build.

I've come up with a working PoC using TestContainer to launch Jenkins as docker container and using that container to run a couple of of JUnit tests successfully.

It runs fine under Eclipse but when I try build Mylyn with Maven the build blows up on the modified target file. [ERROR] Failed to resolve target definition file:/E:/workspaces/eclipse-mylyn/mylyn-main/git/org.eclipse.mylyn/org.eclipse.mylyn-target/mylyn-e4.28.target: Artifact MavenArtifactFacade [...], IncludeDependencyScope = [compile, runtime, test], MissingManifestStrategy = GENERATE, IncludeSource = true is not a valid jar file

Also I don't know if it will run using the Mylyn Jenkins instance used for the builds?

Is this something worth exploring more? Or is it just a wild idea?

George Originally posted by @gnl42 in https://github.com/eclipse-mylyn/org.eclipse.mylyn/issues/203#issuecomment-1627643286

[gnl42]

Shoot, so close:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin]) 79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null)As no valid configuration was found, execution cannot continue.

Would it be possible to have docker installed on the server?

[ruspl-afed]

Please have a look at https://wiki.eclipse.org/Jenkins

[gnl42]

:( Doesn't sound promising

docker build has been promised 'soon' since 2018

It had such promise

[ruspl-afed]

@gnl42 , please describe your issue for Helpdesk, I'm sure they can provide a guidance.

[BeckerFrank]

@gnl42 , please describe your issue for Helpdesk, I'm sure they can provide a guidance.

I think we should ask Helpdesk the following thinks.

1. is it possible to have a Docker in Docker image for the build
2. if the answer to 1) is yes we need a network so that the two container can talk to each other

If not possible we should ask if there is an other way to host Test instances for Jenkins and Bugzilla on the Eclipse infastructure and access over an internal url.

[gnl42]

Do the servers on mylyn.org talk to each other? If not, why do we need a network? And if they do, could we not set up the test image to run them all?

How does this sound @BeckerFrank @sddavis @ruspl-afed :

=== The Mylyn project currently uses servers (gerrit, bugzilla jenkin, etc.) running on mylyn.org, a server sponsored by Tasktop.com, for the unit test validation of its code. Tasktop have said that they no longer maintaining the server and has indicated that they will retire the server at some point in the future.

One possible solution to this is run the unit test against services running on Docker images. I've implemented a Proof Of Concept of this using TestContainers that runs on fine my local machine against a Jenkins server docker image, but when the Jenkins server on https://ci.eclipse.org/mylyn ran the code it complained with:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin]) 79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null) As no valid configuration was found, execution cannot continue.

When I look at https://wiki.eclipse.org/Jenkins I see:

I want to build a custom Docker image (with docker build), but it does not work. What should I do? You cannot currently build images on the cluster (i.e. docker build does not work). We plan to address this shortcoming shortly.

This note was added to the page in 2018.

So I'm wondering if the docker engine could be installed on the Jenkins server to be used as a possible replacement for mylyn.org?

If this is not possible, is there any other way a Test instance for the servers could be set up on the Eclipse infrastructure that can be accessed by the Mylyn build?

Thanks George

[wimjongman]

Great.

[BeckerFrank]

Do the servers on mylyn.org talk to each other? If not, why do we need a network? And if they do, could we not set up the test image to run them all?

No they do not talk to each other. But the all register to a list of available services.

How does this sound @BeckerFrank @sddavis @ruspl-afed :

=== The Mylyn project currently uses servers (gerrit, bugzilla jenkin, etc.) running on mylyn.org, a server sponsored by Tasktop.com, for the unit test validation of its code. Tasktop have said that they no longer maintaining the server and has indicated that they will retire the server at some point in the future.

One possible solution to this is run the unit test against services running on Docker images. I've implemented a Proof Of Concept of this using TestContainers that runs on fine my local machine against a Jenkins server docker image, but when the Jenkins server on https://ci.eclipse.org/mylyn ran the code it complained with:

79513 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /opt/tools/java/openjdk/jdk-17/latest/bin, /opt/tools/apache-maven/latest/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin]) 79547 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were: UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock) DockerDesktopClientProviderStrategy: failed with exception NullPointerException (Cannot invoke "java.nio.file.Path.toString()" because the return value of "org.testcontainers.dockerclient.DockerDesktopClientProviderStrategy.getSocketPath()" is null) As no valid configuration was found, execution cannot continue.

When I look at https://wiki.eclipse.org/Jenkins I see:

I want to build a custom Docker image (with docker build), but it does not work. What should I do? You cannot currently build images on the cluster (i.e. docker build does not work). We plan to address this shortcoming shortly.

This note was added to the page in 2018.

So I'm wondering if the docker engine could be installed on the Jenkins server to be used as a possible replacement for mylyn.org?

We need to run the entire build in a Docker image and the container running the unit test must have access to the newly created Jenkins/Bugzilla test instance.

If this is not possible, is there any other way a Test instance for the servers could be set up on the Eclipse infrastructure that can be accessed by the Mylyn build?

I don't know if there is a place for a virtual infrastructure that can be accessed from https://ci.eclipse.org/mylyn where we can have custom Docker images with our Bugzilla/Jenkins instances.

Thanks George

[gnl42]

I've created a helpdesk issue at https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3432

Sounds like docker images are out of the question, but if someone else with some knowledge on how Eclipse infrastructure could jump since I'm now out of my depth as to what would fulfill our needs. Thanks

[gnl42]

@BeckerFrank @ruspl-afed Could someone jump in on the GitLab issue https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/3432?

[BeckerFrank]

@gnl42: What was the solution? Did you try the sidecar-container way?

[gnl42]

@BeckerFrank Sorry, I thought you were working on it

[BeckerFrank]

I have created a POC of a Mylyn Multipass setup and a subdomain but need a tester to verify that we can access this from outside of my network. If this works I can implement this when I have a new Server for hosting this. The POC use my Laptop .

[gnl42]

How can I test it?

[BeckerFrank]

With https://mylyn.local/mylyn_idx/service I can access the services from within my network and get the json String as a result and with https://mylyn.local/mylyn_idx I get a http page with the human more readable output instead of the json String.

In a first step please test if you get a response from the following url's

• https://mylyn.frank-becker.de/mylyn_idx/
• https://mylyn.frank-becker.de/mylyn_idx/service

When this works we know that I can share the content from a multipass vm.

Next you can change org.eclipse.mylyn.commons.sdk.util.TestConfiguration.discover(Class, String, boolean) in /org.eclipse.mylyn.commons.sdk.util/src/org/eclipse/mylyn/commons/sdk/util/TestConfiguration.java and use "/mylyn_idx/service" instead of "/cgi-bin/services" (line 130)

If it not work please ping me so that we can make sure that my MacBook is not in sleep mode.

[gnl42]

Browser issue with the certificate:

https://mylyn.frank-becker.de/mylyn_idx/service

Peer’s Certificate issuer is not recognized.

Once I accept the risk, I'm in.

Needed to change URL_SERVICES_DEFAULT as well in TestConfiguration.

9 Errors, 2 Failures. Basically two types of errors:

• org.eclipse.core.runtime.CoreException: I/O Error occurred while communicating with https://mylyn.frank-becker.de/bugzilla-4_4. UnknownHostException: mylyn.local at org.eclipse.mylyn.internal.bugzilla.core.BugzillaClient.connectInternal(BugzillaClient.java:342)
• junit.framework.ComparisonFailure: expected:<[Unable to login to https://mylyn.frank-becker.de/bugzilla-4_4. The username or password you entered is not valid. Please validate credentials via Task Repositories view.]> but was:<[I/O Error occurred while communicating with https://mylyn.frank-becker.de/bugzilla-4_4. UnknownHostException: mylyn.local]> at junit.framework.Assert.assertEquals(Assert.java:100) at junit.framework.Assert.assertEquals(Assert.java:107) at junit.framework.TestCase.assertEquals(TestCase.java:260) at org.eclipse.mylyn.bugzilla.tests.core.BugzillaClientTest.testValidateUserPlusHTTP(BugzillaClientTest.java:183)

[BeckerFrank]

@gnl42: Thanks for testing.

I think the problems you are experiencing are due to the multipass setup being designed for the mylyn.local domain. This domain is the one I can access from my home network and used in the the bugzilla.conf.

From within my home network I can not access the public domain because of the config in my router. I have changed the bugzilla setup so please try it again

Do you think we need to change the Certificate?

[BeckerFrank]

@gnl42: Did you have found time to test this again? I have changed the bugzilla.conf and hope that this fixed the login error.

[gnl42]

@BeckerFrank Sorry, battling guava.

Not being able to connect to the server at the moment.

[BeckerFrank]

OK, maybe my Notebook was in sleep mode. Can you try again?

[gnl42]

Much better. Just one error now. Looks like missing data?:

org.eclipse.core.runtime.CoreException: Repository error from :

Bug id not found. The requested bug id does not exist.
    at org.eclipse.mylyn.internal.bugzilla.core.MultiBugReportFactory.populateReport(MultiBugReportFactory.java:63)
    at org.eclipse.mylyn.internal.bugzilla.core.BugzillaClient.getTaskData(BugzillaClient.java:2338)
    at org.eclipse.mylyn.bugzilla.tests.support.BugzillaFixture.getTask(BugzillaFixture.java:258)
    at org.eclipse.mylyn.bugzilla.tests.core.BugzillaClientTest.testLeadingZeros(BugzillaClientTest.java:277)

The test is looking for String taskNumber = "0002";

[BeckerFrank]

Thanks, this looks like a setup error but not a connection error.

Now we know that I can set up a server with a multipass instance for all others, but not for an access from my internal network . So I will plan when and how I replace my old internal with a new one that can host her needed Multipass Instance.