search

zncdatadev / containers

Container for images of the Kubedoop Data Platform
Apache License 2.0
2 stars 4 forks source link

fix(kafka): fix gradle lock error when multip-arch build #60

Closed whg517 closed 3 months ago

whg517 commented 3 months ago

According to the gradle documentation, when gradle is built, a daemon process is started in the background and a.lock lock file is generated. Starting multiple gradle instances will cause concurrency conflicts.

The current workaround is to cache gradle's cache directory at build time and mark it in lock mode so that concurrency is prohibited when building at that layer. This avoids file conflicts caused by sharing file systems between build environments when building multiple architectures.

This problem was raised very early in the gradle project and has not been solved well so far.

context

Dockerfile


ARG JAVA_BASE_VERSION
ARG JAVA_DEVEL_VERSION

# stage: java-builder
FROM quay.io/zncdatadev/java-devel:${JAVA_DEVEL_VERSION}-kubedoop0.0.0-dev as java-builder

ARG PRODUCT_VERSION
ARG JMX_EXPORTER_VERSION
ARG SCALA_VERSION

# build kafka
RUN <<EOF
    set -ex

    mkdir -p /build/kafka-src
    pushd /build/kafka-src
    curl -sSfL \
        https://archive.apache.org/dist/kafka/${PRODUCT_VERSION}/kafka-${PRODUCT_VERSION}-src.tgz \
        | tar xzf - --strip-components=1
    ./gradlew clean releaseTarGz -x test
    tar -xzf core/build/distributions/kafka_${SCALA_VERSION}-${PRODUCT_VERSION}.tgz -C /kubedoop
    ln -s /kubedoop/kafka_${SCALA_VERSION}-${PRODUCT_VERSION} /kubedoop/kafka
    popd
EOF

# smoke test
RUN /kubedoop/kafka/bin/kafka-server-start.sh --version

......

Command

 buildah build --manifest=quay.io/zncdatadev/kafka:3.8.0-kubedoop0.0.0-dev \
    --platform=linux/amd64,linux/arm64 \
    --jobs=4 \
    --build-arg=PRODUCT_VERSION=3.8.0 \
    --build-arg=JAVA_BASE_VERSION=21 \
    --build-arg=JAVA_DEVEL_VERSION=21 \
    --build-arg=JMX_EXPORTER_VERSION=1.0.1 \
    --build-arg=SCALA_VERSION=2.13 \
    kafka

In the kafka 3.8.0, it use gradle 8.7.

When i run command, i got error:

0.209 + mkdir -p /build/kafka-src
0.244 + pushd /build/kafka-src
0.246 /build/kafka-src /build
0.249 + tar xzf - --strip-components=1
0.252 + curl -sSfL https://archive.apache.org/dist/kafka/3.8.0/kafka-3.8.0-src.tgz
19.53 + ./gradlew clean releaseTarGz -x test
22.32 Downloading https://services.gradle.org/distributions/gradle-8.7-all.zip
26.88 .....................10%.....................20%......................30%.....................40%......................50%.....................60%.....................70%......................80%.....................90%......................100%
77.41
77.41 Welcome to Gradle 8.7!
77.41
77.41 Here are the highlights of this release:
77.41  - Compiling and testing with Java 22
77.41  - Cacheable Groovy script compilation
77.41  - New methods in lazy collection properties
77.41
77.42 For more details see https://docs.gradle.org/8.7/release-notes.html
77.42
78.90 Starting a Gradle Daemon (subsequent builds will be faster)
350.3
350.3 > Configure project :
350.3 Starting build with version 3.8.0 (commit id unknown) using Gradle 8.7, Java 21 and Scala 2.13.14
350.3 Build properties: maxParallelForks=8, maxScalacThreads=8, maxTestRetries=0
400.9
400.9 > Task :core:clean UP-TO-DATE
400.9 > Task :clean UP-TO-DATE
400.9 > Task :examples:clean UP-TO-DATE
400.9 > Task :jmh-benchmarks:clean UP-TO-DATE
400.9 > Task :group-coordinator:clean UP-TO-DATE
400.9 > Task :generator:clean UP-TO-DATE

......

820.5 > Task :streams:streams-scala:compileScala
908.0 > Task :core:compileScala FAILED
1019.7
1019.7 FAILURE: Build failed with an exception.
1019.7
1019.7 * What went wrong:
1019.7 Execution failed for task ':core:compileScala'.
1019.7 > Timeout waiting to lock zinc-1.9.2_2.13.14_21 compiler cache (/root/.gradle/caches/8.7/zinc-1.9.2_2.13.14_21). It is currently in use by another Gradle instance.
1019.7   Owner PID: 691
1019.7   Our PID: 695
1019.7   Owner Operation:
1019.7   Our operation:
1019.7   Lock file: /root/.gradle/caches/8.7/zinc-1.9.2_2.13.14_21/zinc-1.9.2_2.13.14_21.lock
1019.7
1019.7 * Try:
1019.7 > Run with --stacktrace option to get the stack trace.
1019.7 > Run with --info or --debug option to get more log output.
1019.7 > Get more help at https://help.gradle.org.
1019.8
1019.8 Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
1019.8
1019.8 You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
1019.8
1019.8 For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
1019.8
1019.8 BUILD FAILED in 16m 20s
1019.8 148 actionable tasks: 92 executed, 56 up-to-date

Note: I also try to use docker buildx and podman build.

❌ Try the same directory as the arch or random directory

# build kafka
RUN <<EOF
    set -ex

    export GRADLE_USER_HOME=/build/.gradle-$(uname -m)
    # export GRADLE_USER_HOME=/build/.gradle-$(uuidgen)

    mkdir -p /build/kafka-src
    pushd /build/kafka-src
    curl -sSfL \
        https://archive.apache.org/dist/kafka/${PRODUCT_VERSION}/kafka-${PRODUCT_VERSION}-src.tgz \
        | tar xzf - --strip-components=1
    ./gradlew clean releaseTarGz -x test
    tar -xzf core/build/distributions/kafka_${SCALA_VERSION}-${PRODUCT_VERSION}.tgz -C /kubedoop
    ln -s /kubedoop/kafka_${SCALA_VERSION}-${PRODUCT_VERSION} /kubedoop/kafka
    popd
EOF

The problem is still .

843.2 > Task :streams:test-utils:jar
851.9 > Task :streams:streams-scala:compileScala
934.3 > Task :core:compileScala FAILED
1034.5
1034.5 FAILURE: Build failed with an exception.
1034.5
1034.5 * What went wrong:
1034.5 Execution failed for task ':core:compileScala'.
1034.5 > Timeout waiting to lock zinc-1.9.2_2.13.14_21 compiler cache (/build/.gradle-aarch64/caches/8.7/zinc-1.9.2_2.13.14_21). It is currently in use by another Gradle instance.
1034.5   Owner PID: 733
1034.5   Our PID: 729
1034.5   Owner Operation:
1034.5   Our operation:
1034.5   Lock file: /build/.gradle-aarch64/caches/8.7/zinc-1.9.2_2.13.14_21/zinc-1.9.2_2.13.14_21.lock
1034.5
1034.5 * Try:
1034.5 > Run with --stacktrace option to get the stack trace.
1034.5 > Run with --info or --debug option to get more log output.
1034.5 > Get more help at https://help.gradle.org.
1034.6
1034.6 Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
1034.6
1034.6 You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
1034.6
1034.6 For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
1034.6
1034.6 BUILD FAILED in 16m 53s

❌ Try using the tmpfs cache

# build kafka
RUN --mount=type=tmpfs,target=/root/.gradle <<EOF
    set -ex

    mkdir -p /build/kafka-src
    pushd /build/kafka-src
    curl -sSfL \
        https://archive.apache.org/dist/kafka/${PRODUCT_VERSION}/kafka-${PRODUCT_VERSION}-src.tgz \
        | tar xzf - --strip-components=1
    ./gradlew clean releaseTarGz -x test
    tar -xzf core/build/distributions/kafka_${SCALA_VERSION}-${PRODUCT_VERSION}.tgz -C /kubedoop
    ln -s /kubedoop/kafka_${SCALA_VERSION}-${PRODUCT_VERSION} /kubedoop/kafka
    popd
EOF

The problem is still .

802.9 > Task :core:compileScala
803.0 > Task :streams:streams-scala:compileScala
805.9 > Task :clients:compileTestJava
893.2 > Task :core:compileScala FAILED
1000.7
1000.7 FAILURE: Build failed with an exception.
1000.7
1000.7 * What went wrong:
1000.7 Execution failed for task ':core:compileScala'.
1000.7 > Timeout waiting to lock zinc-1.9.2_2.13.14_21 compiler cache (/root/.gradle/caches/8.7/zinc-1.9.2_2.13.14_21). It is currently in use by another Gradle instance.
1000.7   Owner PID: 737
1000.7   Our PID: 741
1000.7   Owner Operation:
1000.7   Our operation:
1000.7   Lock file: /root/.gradle/caches/8.7/zinc-1.9.2_2.13.14_21/zinc-1.9.2_2.13.14_21.lock
1000.7
1000.7 * Try:
1000.7 > Run with --stacktrace option to get the stack trace.
1000.7 > Run with --info or --debug option to get more log output.
1000.7 > Get more help at https://help.gradle.org.
1000.8
1000.8 Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
1000.8
1000.8 You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
1000.8
1000.8 For more on this, please refer to https://docs.gradle.org/8.7/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
1000.8
1000.8 BUILD FAILED in 16m 22s
1000.8 148 actionable tasks: 92 executed, 56 up-to-date

Try cache and (shared or private)

Note: buildah/podman not support sharing=private

The problem is still .

✅ Try cache and locked

# build kafka
RUN --mount=type=cache,sharing=locked,target=/root/.gradle <<EOF
    set -ex

    mkdir -p /build/kafka-src
    pushd /build/kafka-src
    curl -sSfL \
        https://archive.apache.org/dist/kafka/${PRODUCT_VERSION}/kafka-${PRODUCT_VERSION}-src.tgz \
        | tar xzf - --strip-components=1
    ./gradlew clean releaseTarGz
    tar -xzf core/build/distributions/kafka_${SCALA_VERSION}-${PRODUCT_VERSION}.tgz -C /kubedoop
    ln -s /kubedoop/kafka_${SCALA_VERSION}-${PRODUCT_VERSION} /kubedoop/kafka
    popd
EOF

Using this configuration, gradle's concurrent locking problem is solved.

conclusion

However, I think that using locked mode may have some performance impact on parallel builds, but in order to eliminate this problem, we have to do this.

As a final note, the -no-daemon parameter is useless