search

GoogleContainerTools / jib

🏗 Build container images for your Java applications.
Apache License 2.0
13.5k stars 1.42k forks source link

Jib 3.4.2 build fails #4235

Open artemptushkin opened 3 months ago

artemptushkin commented 3 months ago

Environment:

When I run 

./gradlew jibDockerBuild

It fails with

Caused by: java.lang.NoSuchMethodError: 'void org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(org.apache.commons.compress.archivers.tar.TarArchiveEntry)'
        at com.google.cloud.tools.jib.tar.TarStreamBuilder.writeAsTarArchiveTo(TarStreamBuilder.java:52)
        at com.google.cloud.tools.jib.image.ImageTarball.dockerWriteTo(ImageTarball.java:166)
        at com.google.cloud.tools.jib.image.ImageTarball.writeTo(ImageTarball.java:84)
        at com.google.cloud.tools.jib.docker.CliDockerClient.load(CliDockerClient.java:195)

Workaround. It works when I set this in both root and gradle submodule build.gradle.kts

buildscript {
    dependencies {
        classpath("commons-codec:commons-codec:1.16.1")
    }
    configurations.all {
        resolutionStrategy {
            force("org.apache.commons:commons-compress:1.26.0")
            force("commons-codec:commons-codec:1.16.1")
        }
    }
}

Please make it work w/o workarounds

chaouki-reply commented 3 months ago

Facing the same issue also! I hope that will be fixed soon

mpeddada1 commented 2 months ago

Hey @artemptushkin, thanks for filing the issue and appreciate you sharing the workaround! As we try to learn more, would it be possible to see how your project's dependency tree looks without the explicit inclusion of org.apache.commons:commons-compress:1.26.0? Particularly curious if the dependency is being transitively brought in from another place.

chaouki-reply commented 2 months ago

Are there any updates?

adenhartog commented 2 months ago

We have the same issue in the jib-gradle-plugin (without the workaround). A gradlew buildEnvironment shows org.apache.commons:commons-compress:1.26.0 is requested by jib-gradle-plugin itself:

\--- com.google.cloud.tools:jib-gradle-plugin:3.4.2
     +--- com.google.http-client:google-http-client:1.42.2
     |    ...
     +--- com.google.http-client:google-http-client-apache-v2:1.42.2
     |    ...
     +--- com.google.auth:google-auth-library-oauth2-http:1.10.0
     |    ...
     +--- org.apache.commons:commons-compress:1.26.0

Other transitive dependencies request 1.21 but Gradle resolves the conflict with the newer version requested by jib-gradle-plugin. But the stacktrace shows it is the jib-gradle-plugin itself which breaks on this dependency, as can be seen in the first post.

artemptushkin commented 2 months ago

@adenhartog thanks for the trace

My

|         |    +--- org.apache.commons:commons-compress:1.21

comes from

+--- org.springframework.boot:org.springframework.boot.gradle.plugin:3.2.4

Though no "commons-codec" in the trace for ./gradlew buildEnvironment

adenhartog commented 2 months ago

For comparison, we also have the spring-boot-gradle plugin but in our project. It requests 1.21 but gradle resolves it to 1.26.0 in our project:

          |    \--- org.springframework.boot:spring-boot-gradle-plugin:3.2.4
          |         +--- org.springframework.boot:spring-boot-buildpack-platform:3.2.4
          |         |    ...
          |         |    +--- org.apache.commons:commons-compress:1.21 -> 1.26.0 (*)
chanseokoh commented 2 months ago

Other transitive dependencies request 1.21 but Gradle resolves the conflict with the newer version requested by jib-gradle-plugin.

For comparison, we also have the spring-boot-gradle plugin but in our project. It requests 1.21 but gradle resolves it to 1.26.0 in our project:

If your project is a multi-module one, then it's pretty common that ./gradlew buildEnvironment lies about how it resolved build environment dependencies; it's a long-standing Gradle bug where build dependencies declared in the parent project shamelessly and silently overrides dependencies of sub-projects, as opposed to what you see from ./gradlew buildEnv.

So, if you haven't done so, I recommend first that you strictly follow these instructions for setting up your multi-module project. That is, declare all your plugins in the project root while not applying them and then selective apply plugins in sub-projects.

UPDATE: another case of Gradle lying with ./gradlew buildEnv was https://github.com/GoogleContainerTools/jib/issues/4235#issuecomment-2066685245 where an old version of the Apache library was silently pulled in from buildSrc/build.gradle.

mihalyr-prospect commented 2 months ago

Here is a non multi-module project buildEnvironment (also spring-boot but not using the spring-boot gradle plugin), our compress comes from Jib and it also broke our jibBuildTar step after 3.4.2

+--- com.google.cloud.tools.jib:com.google.cloud.tools.jib.gradle.plugin:3.4.2
|    \--- com.google.cloud.tools:jib-gradle-plugin:3.4.2
|         +--- org.apache.commons:commons-compress:1.26.0

This is on Gradle (wrapper) 8.6, Java 21, and Linux (Fedora 38, but also failing in GitHub CI on ubuntu-latest).

It seems to be the same issue as reported above:

> Task :jibBuildTar FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jibBuildTar'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: 'void org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(org.apache.commons.compress.archivers.tar.TarArchiveEntry)'

Stacktrace:

Caused by: java.lang.NoSuchMethodError: 'void org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(org.apache.commons.compress.archivers.tar.TarArchiveEntry)'
        at com.google.cloud.tools.jib.tar.TarStreamBuilder.writeAsTarArchiveTo(TarStreamBuilder.java:52)
        at com.google.cloud.tools.jib.hash.Digests.computeDigest(Digests.java:104)
        at com.google.cloud.tools.jib.blob.WritableContentsBlob.writeTo(WritableContentsBlob.java:37)
        at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeUncompressedLayerBlobToDirectory(CacheStorageWriter.java:430)
        at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeUncompressed(CacheStorageWriter.java:282)
        at com.google.cloud.tools.jib.cache.Cache.writeUncompressedLayer(Cache.java:145)
        at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call(BuildAndCacheApplicationLayerStep.java:108)
        at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call(BuildAndCacheApplicationLayerStep.java:38)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
mihalyr-prospect commented 2 months ago

Another thing, I saw the workaround is to enforce commons-codec version 1.16, in our project is at version 1.11 currently, which is also pulled in by jib

+--- com.google.cloud.tools.jib:com.google.cloud.tools.jib.gradle.plugin:3.4.2
|    \--- com.google.cloud.tools:jib-gradle-plugin:3.4.2
|         +--- com.google.http-client:google-http-client:1.42.2
|         |    +--- org.apache.httpcomponents:httpclient:4.5.13
|         |    |    +--- org.apache.httpcomponents:httpcore:4.4.13 -> 4.4.15
|         |    |    +--- commons-logging:commons-logging:1.2
|         |    |    \--- commons-codec:commons-codec:1.11
adenhartog commented 2 months ago

@chanseokoh Aligning the versions in de root project plugins block fixes the problem in our multi-module project. Good to know about this obscure Gradle issue/bug, thanks!

chanseokoh commented 2 months ago

@adenhartog glad you fixed your project.

@mihalyr-prospect I think it must be that for one reason or another, your project is set up to "downgrade" the Apache commons compress 1.26.0, which is the version Jib requires. You should ensure that your build environment doesn't downgrade it. If you have to force it, do so. I can take a look if you have a reproducible sample, but I don't think Jib is doing anything wrong about it.

mihalyr-prospect commented 2 months ago

@chanseokoh I shared above our buildEnvironment that shows that we use commons compress 1.26.0 that is only pulled in by jib and nothing should depend on it or "downgrade" it as far as I can see. Unfortunately, I don't have a sample as it is a private project and right now this isn't that important for us to debug this further. For now I just skip this update, it was supposed to be only a patch bump that broke our CI, we don't really need anything from the new version right now.

chaouki-reply commented 2 months ago

Is there a fix in the next release? or we should force the usage of commons-compress & commons-codec? Anyway at the moment, we downgraded also to the previous version.

chanseokoh commented 2 months ago

FWIW, there is nothing to fix in Jib. Jib asks to use the latest version of the libraries, but somehow your project is pulling an old version. Basically, it is your responsibility to correctly set up your project so that it doesn't happen. If your project is a multi-module one, make sure you follow https://github.com/GoogleContainerTools/jib/issues/4235#issuecomment-2057081017. Sometimes, you will have to manually resolve this kind of diamond dependency conflicts.

artemptushkin commented 2 months ago

@chanseokoh no, that's not fully right, there is no "commons-codec:commons-codec" at all as you see I have to define it by myself. There are plenty of comments here, you could take a look if you can fix it, usually it's a case that we have to enforce a libraries version in the build classpath - it doesn't happen for other plugins

chanseokoh commented 2 months ago

If anyone can provide a reproducible sample, I can help.

artemptushkin commented 2 months ago

you can take any base spring boot project on 3.2.4

chanseokoh commented 2 months ago

I am not the maintainer of this repo. I am contributing my own time to help the community. Please provide a reproducible sample if it is that easy to create.

mihalyr-prospect commented 2 months ago

I had some free time to look into why my build fails and in my case buildEnvironment wasn't telling me the truth, because it did not include the dependencies from buildSrc (I have a custom plugin for jOOQ that spins up a testcontainer, to build the DB and generate the DSL).

demo/buildSrc$ ../gradlew -q dependencies

Revealed this:

\--- org.testcontainers:postgresql -> 1.19.7
     \--- org.testcontainers:jdbc:1.19.7
          \--- org.testcontainers:database-commons:1.19.7
               \--- org.testcontainers:testcontainers:1.19.7
                    +--- junit:junit:4.13.2
                    |    \--- org.hamcrest:hamcrest-core:1.3 -> 2.2
                    |         \--- org.hamcrest:hamcrest:2.2
                    +--- org.slf4j:slf4j-api:1.7.36 -> 2.0.12
                    +--- org.apache.commons:commons-compress:1.24.0

While running ./gradlew buildEnvironment was telling me that nothing uses commons-compress except jib and it is on version 1.26.0 which wasn't the case.

And in this particular case testcontainers already tracks the update, so it's just a matter of time until the order is restored again... https://github.com/testcontainers/testcontainers-java/issues/8338

The fix was also simple once I figured out what was the problem, I've added the following into my buildSrc/build.gradle where I had testcontainers defined:

// buildSrc/build.gradle
dependencies {
  // other plugin deps
  implementation 'org.testcontainers:postgresql:1.19.7'
  implementation 'org.apache.commons:commons-compress:1.26.1'
}

And now both, testcontainers and jib, seem to be working for me.

artemptushkin commented 2 months ago

Okay, two dependencies:

chanseokoh commented 2 months ago

@mihalyr-prospect there you go! As you saw, the cause is that Gradle pulls in an old version of some library in your build environment for one reason or another where ./gradlew buildEnv lies, which is a known Gradle issue. All you have to do is to resolve your diamond dependency conflict in your project in one way or another. Glad you were able to fix it.

danieladams456 commented 2 months ago

Moving the jib plugin version definition to the root of my multi-module project with apply false and using it at the subproject solved the issue for me as well.