How to make sure there's no stray container not removed when using withDockerfileFromBuilder?

[vmassol]

Hi guys,

On XWiki, and since the upgrade to TC 1.15.0 (might not be related but didn't notice it before), I've noticed on our CI agents, that there are some TC-created containers that are not always removed on agents. For example:

These containers are created with the following:

                LOGGER.info("(*) Build a dedicated image embedding LibreOffice...");
                // The second argument of the ImageFromDockerfile is here to indicate we won't delete the image
                // at the end of the test container execution.
                container = new XWikiLocalGenericContainer(new ImageFromDockerfile(imageName, false)
                    .withDockerfileFromBuilder(builder -> {
                        builder
                            .from(baseImageName)
                            .user("root")
                            .env("LIBREOFFICE_VERSION", officeVersion)
                            .env("LIBREOFFICE_DOWNLOAD_URL", "https://downloadarchive.documentfoundation.org/"
                                + "libreoffice/old/$LIBREOFFICE_VERSION/deb/x86_64/"
                                + "LibreOffice_${LIBREOFFICE_VERSION}_Linux_x86-64_deb.tar.gz")
                            // Note that we expose libreoffice /usr/local/libreoffice so that it can be found by
                            // JODConverter: https://bit.ly/2w8B82Q
                            .run("apt-get update && "
                                + "apt-get --no-install-recommends -y install curl unzip procps libxinerama1 "
                                    + "libdbus-glib-1-2 libcairo2 libcups2 libsm6 && "
                                + "rm -rf /var/lib/apt/lists/* /var/cache/apt/* && "
                                + "wget --no-verbose -O /tmp/libreoffice.tar.gz $LIBREOFFICE_DOWNLOAD_URL && "
                                + "mkdir /tmp/libreoffice && "
                                + "tar -C /tmp/ -xvf /tmp/libreoffice.tar.gz && "
                                + "cd /tmp/LibreOffice_${LIBREOFFICE_VERSION}_Linux_x86-64_deb/DEBS && "
                                + "dpkg -i *.deb && "
                                + "ln -fs `ls -d /opt/libreoffice*` /opt/libreoffice")
                            // Increment the image version whenever a change is brought to the image so that it can
                            // reconstructed on all machines needing it.
                            .label(OFFICE_IMAGE_VERSION_LABEL, imageVersion);
                        if (this.testConfiguration.getServletEngine() == ServletEngine.JETTY) {
                            // Create the right jetty user directory since it doesn't exist
                            builder.run("mkdir -p /home/jetty && chown jetty:jetty /home/jetty")
                                // Put back the user as jetty since it's a best practice to not execute the container as
                                // root.
                                .user("jetty");
                        }

                        builder.build();
                    }));

See https://github.com/xwiki/xwiki-platform/blob/eaced75f5a08b4131e36504fb5f6c71d6b392fa8/xwiki-platform-core/xwiki-platform-test/xwiki-platform-test-docker/src/main/java/org/xwiki/test/docker/internal/junit5/servletengine/ServletContainerExecutor.java#L286 for more details.

Would you have any idea what could be happening and what we could do?

Thanks a lot

[vmassol]

This is still happening (we upgraded to 1.15.1). The days when the container exited suggest that the container is removed most of the time but sometimes it's not. Since it only happens for the container we dynamically build with withDockerfileFromBuilder, there must be something special with it I guess.

@rnorth @bsideup would you have any idea or direction for me to look at to debug this problem?

Thanks a lot

[bsideup]

@vmassol could you please docker inspect one of such containers and share it? (e.g. as a Gist)

I'd assume that, while building the image, Docker creates a temporary container that we cannot control, and somehow fails to remove it afterwards. Although in such case there isn't much we can do and it requires a help from the Docker team (unless you can verify that it worked with 1.14.x, for example)

[vmassol]

@vmassol could you please docker inspect one of such containers and share it? (e.g. as a Gist)

https://gist.github.com/vmassol/fbe1c8190313eb5b5e3be7a7040e262e

(unless you can verify that it worked with 1.14.x, for example)

What's sure is that it wasn't happening in the past. But I don't know if it's related to a TC version or a Docker version or some other factor.

Thanks for the help!

[bsideup]

Oh yeah, the container is missing org.testcontainers.* labels, meaning that we have no control over it.

We do add the labels to buildImageCmd to control it, so maybe Docker introduced a regression and build containers no longer have them? Although I would still expect Docker to take care of the temporary containers that it creates.

I am afraid there isn't much we can do on our side. Consider reporting this regression to Docker...

[vmassol]

Note that, seen the pattern, most of the containers from that custom image are removed when the tests stop. It's only under some conditions that they're not removed (jvm crash, low memory, etc?).

Just checked now and I have on one agent:

CONTAINER ID   IMAGE                         COMMAND                   CREATED          STATUS                      PORTS                                       NAMES
33746d529429   xwiki/build:latest            "setup-xwiki-ssh /bi…"    16 minutes ago   Up 16 minutes               22/tcp                                      frosty_joliot
b1cf5ca28954   zenika/alpine-chrome:latest   "chromium-browser --…"    10 hours ago     Up 10 hours                 0.0.0.0:9222->9222/tcp, :::9222->9222/tcp   headless-chrome-pdf-printer
559f799a9493   3164a570eeda                  "/bin/sh -c 'apt-get…"    6 weeks ago      Exited (127) 6 weeks ago                                                cranky_johnson
e1bdd2a24a61   b90919f94b34                  "/bin/sh -c 'apt-get…"    6 weeks ago      Exited (127) 6 weeks ago                                                elastic_lewin
a26ca27ee938   3164a570eeda                  "/bin/sh -c 'apt-get…"    7 weeks ago      Exited (127) 7 weeks ago                                                infallible_panini
0ad07d8be43c   97239b5261ad                  "/bin/sh -c 'apt-get…"    8 weeks ago      Exited (127) 8 weeks ago                                                infallible_keldysh
08b40212f071   97239b5261ad                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               relaxed_clarke
f8689d670a70   db4dc2c95281                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               naughty_moser
1ab73e2dd7d5   db4dc2c95281                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               trusting_beaver
fe7336414a6e   c4be6f55f475                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               musing_antonelli
f0f9e9dc13e5   db4dc2c95281                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               nostalgic_chatelet
e3238a68bc86   95b12cdf8c88                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               eager_jackson
a4a1b30fbab7   db4dc2c95281                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               quirky_kirch
04fc608fd010   db4dc2c95281                  "/bin/sh -c 'apt-get…"    2 months ago     Exited (127) 2 months ago                                               unruffled_swirles
01f33815fab1   db4dc2c95281                  "/bin/sh -c 'apt-get…"    3 months ago     Exited (127) 3 months ago                                               tender_sammet
f2178e641cad   db4dc2c95281                  "/bin/sh -c 'apt-get…"    3 months ago     Exited (127) 3 months ago                                               fervent_liskov
0a5de21cf39d   e02cd5ad11c2                  "/bin/sh -c 'apt-get…"    6 months ago     Exited (4) 6 months ago                                                 dreamy_feynman
fe481ef87177   e02cd5ad11c2                  "/bin/sh -c 'apt-get…"    6 months ago     Exited (4) 6 months ago                                                 ecstatic_montalcini
9adc2b921d1d   eff629089685                  "docker-entrypoint.s…"    13 months ago    Exited (0) 13 months ago                                                jolly_swirles
fc376cf6e641   testcontainers/sshd:1.0.0     "sh -c 'echo \"root:$…"   13 months ago    Exited (137) 5 weeks ago                                                serene_euler

(it's all the "apt-get" ones)

@bsideup I wanted to work around this but I don't see how to do it. If I do an explicit remove at the end of the test (in a JUnit5 extension for ex), it probably won't help since those are probably removed correctly already.

Is there any way I could manually register this container against ryuk so that it removes it after the test is done?

Thanks

[vmassol]

Actually I know how to do it! :) I'll put a "docker prune --filter" in our jenkins pipeline before the build starts. Just need to find the tags to identify these containers.

[vmassol]

Just need to find the tags to identify these containers.

Done in:

• https://github.com/xwiki/xwiki-platform/commit/5b750f342f5459fcb67c472c56fe622d9c11b364
• https://github.com/xwiki/xwiki-jenkins-pipeline/commit/5643e3f1aad27f1b89b9dd77f31c203b573fc9a6

Let's see if it works now...