Use java client for docker access

[markusheiden]

"jib" does not work well with all docker registries, e.g. Portus does not show images pushed with "jib". "jibDockerBuild" requires a docker image containing a docker binary. That is not appropriate for modern CI processes which require "plain" docker images, e.g. a plain java image for executing gradle.

So it would be nice if "jibDockerBuild" would use a java based docker client instead of expecting docker to be installed. This way a lean "docker in docker" approach can be used to provide the docker unix socket to the "plain" build container. This approach avoids any incompatibilities like e.g. the Portus problem or extremely slow upload speeds (compared to docker push) into GitLab repositories.

E.g. fabric8s docker maven plugin already does it this way. Another way would be using https://github.com/amihaiemil/docker-java-api.

[chanseokoh]

Thanks for the feedback.

For now, you need a Docker client CLI (docker), which you can simply download and unzip to use it; no need for a Docker engine (dockerd) installation.

Calling Docker Engine API without going through the docker client CLI was something we had discussed a long time ago. We started with simple, and so far, on one has asked for this, so this has been a very low priority. In any case, if we go through this route, I think we will just do it ourselves with no dependency. We've been beaten two many times from relying on external dependencies.

That said, I think one of the main reasons you are asking for this change is because the registry build (jib) does not work for you. We are actually more interested in fixing this as a high priority. If there is a blocking problem for the main Jib feature, we want to fix it. Jib should just work for conforming container registries. Could you file another issue for this?

extremely slow upload speeds (compared to docker push) into GitLab repositories

This may be unrelated, but 1.5.0 and 1.5.1 had a serious performance problem due to the issue in our google-http-client dependency. This has been fixed in 1.6.0. In any case, if Jib is slower on GitLab, we are very eager to investigate and fix it. Could you file another issue for this?

[markusheiden]

I created another issue for the Portus problems. The performance problems are gone. Because the direct interaction with the docker demon is low priority, I close this issue.

[chanseokoh]

I'll leave this open, as I hope we will move to a Docker library eventually. However, other issues have more priority, so I'm unsure when we'll get to this.

[bsideup]

Hi! FYI I am the current maintainer of https://github.com/docker-java/docker-java

We now have an api package, as well as very lean transports like OkHttp (from 3.2.0) or Apache HttpClient5 (to be released this week).

docker-java is used by http://github.com/testcontainers/testcontainers-java and we were happy with the performance of operations, but, if you find any performance issues, please do let me know!

[chanseokoh]

@bsideup, to use docker-java, which dependencies are required minimally? We'd like to minimize transitive dependencies as much as possible, and I was thinking perhaps we should implement this ourselves. The API we use is only three: docker load, docker save, and docker inspect.

And I see that docker-java-api pulls in SLF4J, which I am not familiar with. How can a Gradle/Maven Jib plugin can direct docker-java to log to Gradle/Maven console output?

[INFO] \- com.github.docker-java:docker-java-api:jar:3.2.1:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.30:compile

[bsideup]

@chanseokoh per Testcontainers' experience, you only need docker-java-api's dependencies and JNA (for native transports that you cannot avoid), the rest can be shaded.

And I see that docker-java-api pulls in SLF4J, which I am not familiar with. How can a Gradle/Maven Jib plugin can direct docker-java to log to Gradle/Maven console output?

Slf4j is a facade (it is important to mention that, for Java, it is an industry standard). You can optionally plug an implementation (like slf4j-simple, or log4j bridge, or logback bridge, or any other) that will typically print it to standard output, so that you get it in Gradle/Maven console output.

I was thinking perhaps we should implement this ourselves

Famous last words 😂 okay, jokes aside, you could, but I'd advice against. Also, I came here from #2130, and, if you implement your own client, the problem of "I want to provide my own" will stay.

[loosebazooka]

Yeah I could see the interface as being the right solution here (perhaps the serviceloader mechanic that we use for extensions)

[chanseokoh]

the rest can be shaded.

You mean embedding other dependencies after renaming packages, right? That option doesn't help to us. What we're really trying is to just not use external dependencies as much as possible, as we've beaten by bugs/constraints in external libraries (Apache HttpClient and Google HTTP Client, for example). Often we needed to fix their bugs ourselves, and it has been painful to push them to release the fixes timely.

if you implement your own client, the problem of "I want to provide my own" will stay.

Maybe I misunderstood, but I don't get how implementing this ourselves could be different; it's like I'm using library A (my own implementation) vs library B (docker-java).

[bsideup]

@loosebazooka

Yeah I could see the interface as being the right solution here (perhaps the serviceloader mechanic that we use for extensions)

I hope there will be an option to provide an instance, too. ServiceLoader mechanic is nice, but global, and sometimes one needs to ensure that the right types are used (thinking about http://github.com/testcontainers/testcontainers-java here)

@chanseokoh

Maybe I misunderstood, but I don't get how implementing this ourselves could be different; it's like I'm using library A (my own implementation) vs library B (docker-java).

docker-java provides an API and an implementation (plus, various transport options). If you implement it yourself, you will either need #2130 to allow custom implementations or a separate abstraction for at least a transport (what docker-java does). Eventually, you may end up providing a subset of what docker-java gives you already, but with much smaller community and amount of existing code/implementation.

[loosebazooka]

Yeah if designed as an interface, I believe service loading would happen only at the plugin level. jib-core should still allow providing implementations.

Having said that, our concern with jib-core is always minimizing dependencies (even through shading). So the interface helps solve our problem there, as long as an implementation can be provided.

We're open to accepting design proposals here. As it is not blocking any of our core functionality, it's rather low priority for us.

[bric3]

Just hit this as we are using gradle docker image to build app, the Docker tooling is not installed.

[chanseokoh]

Ref: https://github.com/GoogleContainerTools/jib/issues/3571#issuecomment-1037107437