Closed ezYakaEagle442 closed 1 year ago
The Dockerfile is at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/docker/petclinic-admin-server/Dockerfile
I can locally successfully build the image :
docker build --build-arg --no-cache -t "petclinic-admin-server" -f "./docker/petclinic-admin-server/Dockerfile" .
docker run -p 8080:8080 -p 8081:8081 -p 8888:8888 -p 9090:9090 petclinic-admin-server
I do not have the same error when using CLI 2.40.0 in the workflow :
az acr build --registry ${{ env.AZURE_CONTAINER_REGISTRY }} -g ${{ env.RG_APP }} \
-t ${{ env.REPOSITORY }}/${{ env.ADMIN_SERVER }}:{{.Run.ID}} \
--file "./docker/petclinic-admin-server/Dockerfile" \
${{ github.workspace }}
Run acr_usr=$(az deployment group show -g rg-iac-aca-petclinic-mic-srv -n acrpetcliaca --query properties.outputs.acrRegistryUsr.value | tr -d '"')
WARNING: Packing source code into tar to upload...
WARNING: Excluding '.gitignore' based on default ignore rules
WARNING: Excluding '.git' based on default ignore rules
WARNING: Uploading archived source code from '/tmp/build_archive_4f5ebbdb5430456890cb403d23118da8.tar.gz'...
WARNING: Sending context (362.578 MiB) to registry: acrpetcliaca...
WARNING: Queued a build with ID: cb7
WARNING: Waiting for an agent...
2022/10/05 21:06:41 Downloading source code...
2022/10/05 21:06:47 Finished downloading source code
2022/10/05 21:06:48 Using acb_vol_75c4b778-fa03-4499-9288-cf2e8a3c5416 as the home volume
2022/10/05 21:06:48 Setting up Docker configuration...
2022/10/05 21:06:49 Successfully set up Docker configuration
2022/10/05 21:06:49 Logging in to registry: acrpetcliaca.azurecr.io
2022/10/05 21:06:50 Successfully logged into acrpetcliaca.azurecr.io
2022/10/05 21:06:50 Executing step ID: build. Timeout(sec): 28800, Working directory: '', Network: ''
2022/10/05 21:06:50 Scanning for dependencies...
2022/10/05 21:06:50 Successfully scanned dependencies
2022/10/05 21:06:50 Launching container with name: build
Sending build context to Docker daemon 435MB
Step 1/24 : FROM mcr.microsoft.com/openjdk/jdk:11-mariner as builder
11-mariner: Pulling from openjdk/jdk
7f0517a4d0b4: Pulling fs layer
89d1ee0edfd7: Pulling fs layer
7f0517a4d0b4: Verifying Checksum
7f0517a4d0b4: Download complete
7f0517a4d0b4: Pull complete
89d1ee0edfd7: Verifying Checksum
89d1ee0edfd7: Download complete
89d1ee0edfd7: Pull complete
Digest: sha256:e01f4e2b5cdca94b0e2b6c446ac8ef811823d82380284bd88e73a4c33df0b8c9
Status: Downloaded newer image for mcr.microsoft.com/openjdk/jdk:11-mariner
---> 3e383202dc94
Step 2/24 : ENV APPI_VERSION="3.4.1"
---> Running in 8591c58b01e8
Removing intermediate container 8591c58b01e8
---> 81d72436f7db
Step 3/24 : ENV APP_INSIGHTS_AGENT_JAR_FILE_PATH="/tmp/app/applicationinsights-agent-$***APPI_VERSION***.jar"
---> Running in 139969f5b37b
Removing intermediate container 139969f5b37b
---> 9d346709ee21
Step 4/24 : LABEL Maintainer="pinpin <noname@microsoft.com>"
---> Running in 4e79024d8fda
Removing intermediate container 4e79024d8fda
---> 95800f62de61
Step 5/24 : LABEL Description="PetClinic admin-server: Java Spring Boot microservice built from MS OpenJDK 11-Mariner"
---> Running in 2ce1a3d8e0e7
Removing intermediate container 2ce1a3d8e0e7
---> 517d2febd55e
Step 6/24 : RUN java -version
---> Running in 7bd2803dc43f
openjdk version "11.0.16.1" 2022-08-12 LTS
OpenJDK Runtime Environment Microsoft-40648 (build 11.0.16.1+1-LTS)
OpenJDK 64-Bit Server VM Microsoft-40648 (build 11.0.16.1+1-LTS, mixed mode, sharing)
Removing intermediate container 7bd2803dc43f
---> d3adeb20a8e3
Step 7/24 : RUN ls -al /usr/lib/jvm/msopenjdk-11
---> Running in 28d12fb3e87d
total 40
drwxr-xr-x 9 root root 4096 Oct 3 09:04 .
drwxr-xr-x 3 root root 4096 Oct 3 09:04 ..
drwxr-xr-x 2 root root 4096 Oct 3 09:04 bin
drwxr-xr-x 4 root root 4096 Oct 3 09:04 conf
drwxr-xr-x 3 root root 4096 Oct 3 09:04 include
drwxr-xr-x 2 root root 4096 Oct 3 09:04 jmods
drwxr-xr-x 73 root root 4096 Aug 24 21:23 legal
drwxr-xr-x 6 root root 4096 Oct 3 09:04 lib
drwxr-xr-x 4 root root 4096 Aug 24 21:23 man
-rw-rw-r-- 1 root root 1232 Aug 24 21:23 release
Removing intermediate container 28d12fb3e87d
---> 2f150b4d5535
Step 8/24 : RUN mkdir /tmp/app
---> Running in 8fd3f7061[98](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:99)2
Removing intermediate container 8fd3f7061982
---> 917812ab34c4
Step 9/24 : WORKDIR /tmp/app
---> Running in 1cf32a6705d4
Removing intermediate container 1cf32a6705d4
---> 29427d6cad0b
Step 10/24 : COPY "spring-petclinic-admin-server/target/*.jar" /tmp/app/app.jar
---> 82e1eb83509a
Step 11/24 : RUN curl -SL --output $***APP_INSIGHTS_AGENT_JAR_FILE_PATH*** https://github.com/microsoft/ApplicationInsights-Java/releases/download/$***APPI_VERSION***/applicationinsights-agent-$***APPI_VERSION***.jar
---> Running in 6f9d1fb7dae4
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
3 36.4M 3 1135k 0 0 1072k 0 0:00:34 0:00:01 0:00:33 1072k
40 36.4M 40 14.8M 0 0 7401k 0 0:00:05 0:00:02 0:00:03 13.8M
84 36.4M 84 30.8M 0 0 10.0M 0 0:00:03 0:00:03 --:--:-- 14.8M
100 36.4M 100 36.4M 0 0 10.7M 0 0:00:03 0:00:03 --:--:-- 15.1M
Removing intermediate container 6f9d1fb7dae4
---> e46dd8572f08
Step 12/24 : RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract
---> Running in 5e7847a82b8f
Removing intermediate container 5e7847a82b8f
---> 4b758dfb2c5d
Step 13/24 : FROM mcr.microsoft.com/openjdk/jdk:11-mariner
11-mariner: Pulling from openjdk/jdk
Digest: sha256:e01f4e2b5cdca94b0e2b6c446ac8ef811823d82380284bd88e73a4c33df0b8c9
Status: Image is up to date for mcr.microsoft.com/openjdk/jdk:11-mariner
---> 3e383202dc94
Step 14/24 : WORKDIR /tmp/app
---> Running in 1d3[99](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:100)750b0a0
Removing intermediate container 1d399750b0a0
---> b8243eff63b3
Step 15/24 : ENV APPI_VERSION="3.4.1"
---> Running in 6ee529833133
Removing intermediate container 6ee529833133
---> 788d2c0e8dae
Step 16/24 : ENV APP_INSIGHTS_AGENT_JAR_FILE_PATH="/tmp/app/applicationinsights-agent-$***APPI_VERSION***.jar"
---> Running in c787d2eab2ac
Removing intermediate container c787d2eab2ac
---> da6131d215d8
Step 17/24 : ENV SPRING_PROFILES_ACTIVE="docker,mysql"
---> Running in b827f9aa80ad
Removing intermediate container b827f9aa80ad
---> a5f95f70f3ba
Step 18/24 : COPY --from=builder $***APP_INSIGHTS_AGENT_JAR_FILE_PATH*** ./
---> d225cc4f8866
Step 19/24 : COPY --from=builder /tmp/app/dependencies/ ./
---> be3050560a49
Step 20/24 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
---> 64de4376ec11
Step 21/24 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:7f92e557[119](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:120)03c43c92210c264f20871f1ad724fc4373ed2383be03e908e[121](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:122)0: layer does not exist
[202](https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3192596067/jobs/5210403362#step:11:203)2/10/05 21:07:27 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1
Run ID: cb7 failed after 47s. Error: failed during run, err: exit status 1
ERROR: Run failed
Error: Process completed with exit code 1.
Hi @ezYakaEagle442 ,
Looking into the logs, it seems that there's an issue at Step 21/24. It looks like steps 18/19/20 succeeded when copying from the expected directories, but for some reason the directory /tmp/app/spring-boot-loader/
fails.
tmp/app/sprint-boot-loader
? tmp/app/sprint-boot-loader
in your system?
2.2 Could you make sure the other directories (/tmp/app/dependencies/, tmp/app/snapshot-dependencies/, etc) are created as expected?@AntonioVT thank you for your quick reply.
The step is
RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract
I do not think there is anythingthere as the final stage copies the artifacts to the folder /tmp/app :
COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
COPY --from=builder /tmp/app/dependencies/ ./
COPY --from=builder /tmp/app/snapshot-dependencies/ ./
COPY --from=builder /tmp/app/spring-boot-loader/ ./
COPY --from=builder /tmp/app/application/ ./
I als odid a quick test locally in VS, ACR CLI fails with the same error
az acr build --registry acrpetcliaca -g rg-iac-aca-petclinic-mic-srv -t petclinic:adm-test --file "./docker/petclinic-admin-server/Dockerfile" .
Step 21/24 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:c37bbfdfd933d6d6b3fafc0df403c8a79824968a020966b2821da6f358a5df60: layer does not exist
2022/10/06 09:11:38 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1
Run ID: cb8 failed after 45s. Error: failed during run, err: exit status 1
confirmed , I have added a debug trace with :
RUN ls -al /tmp/app
the folder is empty:
---> Running in 28f2333e7851
Removing intermediate container 28f2333e7851
---> 99125d2d9ad3
Step 18/25 : RUN ls -al /tmp/app
---> Running in 30ec769fe2c4
total 8
drwxr-xr-x 2 root root 4096 Oct 6 09:22 .
drwxrwxrwt 1 root root 4096 Oct 6 09:22 ..
Removing intermediate container 30ec769fe2c4
---> 21078588665e
Step 19/25 : COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
---> 4f49097c12d6
Step 20/25 : COPY --from=builder /tmp/app/dependencies/ ./
---> 26220f3c83e0
Step 21/25 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
---> 6da4249d8e60
Step 22/25 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:498b96bfc7390a186bae3f682265d00b334865a11d75e42eb7a4449aa6e2e6cd: layer does not exist
2022/10/06 09:22:32 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1
Run ID: cb9 failed after 43s. Error: failed during run, err: exit status 1
@ezYakaEagle442 Thanks for your response.
I see the logs from the cloud build (github workflow) and noticed the directory is empty and the build fails. But I was wondering if in your local environment some of these directories exist already. Could you run docker build locally and verify the /tmp/app directory is empty as well?
Could you share the steps for building the docker image locally?
I have cloned the project in order to reproduce the issue, but when running the command:
docker build --build-arg --no-cache -t "petclinic-admin-server" -f "./docker/petclinic-admin-server/Dockerfile" .
I get the following error:
[builder 7/7] RUN java -Djarmode=layertools -jar "/tmp/app/app.jar" extract:
11 0.340 Error: Unable to access jarfile /tmp/app/app.jar
Thanks
@ezYakaEagle442 I tried to repro it using az acr build. What I did is: git clone https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/ cd to aca-java-petclinic-mic-srv Queue a build: C:\src\aca-java-petclinic-mic-srv>az acr build --registry huanglireg -t mytest123 --file ./docker/petclinic-admin-server/Dockerfile .
I got a different error: Step 10/25 : COPY "spring-petclinic-admin-server/target/*.jar" /tmp/app/app.jar COPY failed: no source files were specified
I cannot find target folder under https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/tree/main/spring-petclinic-admin-server
Could you please confirm if the above repro steps are correct or not?
@huanwu @AntonioVT this is because you need to build the Java project with maven (from the project root aca-java-petclinic-mic-srv) :
mvn -B clean package --file pom.xml -DskipTests -Denv=cloud
Then you will see the Artifacts at : aca-java-petclinic-mic-srv\spring-petclinic-admin-server\target\aca-spring-petclinic-admin-server-2.6.6.jar aca-java-petclinic-mic-srv\spring-petclinic-api-gateway\target\aca-spring-petclinic-api-gateway-2.6.6.jar aca-java-petclinic-mic-srv\spring-petclinic-config-server\target\aca-spring-petclinic-config-server-2.6.6.jar aca-java-petclinic-mic-srv\spring-petclinic-customers-service\target\aca-spring-petclinic-customers-service-2.6.6.jar aca-java-petclinic-mic-srv\spring-petclinic-vets-service\target\aca-spring-petclinic-vets-service-2.6.6.jar aca-java-petclinic-mic-srv\spring-petclinic-visits-service\targetaca-spring-petclinic-visits-service-2.6.6.jar
here are the ref docs :
/!\ ACR is set with Managed Identity: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication-managed-identity?tabs=azure-cli
A few steps might be missing before running az acr build command:
# /!\ ACR is set with Managed Identity: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication-managed-identity?tabs=azure-cli
# az identity create -g rg-iac-aca-petclinic-mic-srv --name myTestIdentity
# identitySpnID=$(az identity show -g rg-iac-aca-petclinic-mic-srv --name myTestIdentity --query principalId --output tsv)
# identityResourceID=$(az identity show -g rg-iac-aca-petclinic-mic-srv --name myTestIdentity --query id --output tsv)
# acrID=$(az acr show -g rg-iac-aca-petclinic-mic-srv --name acrpetcliaca --query id --output tsv)
# az role assignment create --assignee $identitySpnID --scope $acrID --role acrpull
# az login --identity --username $identityResourceID
# az acr login --name acrpetcliaca
The az login fails ()from VS/WSL, I am not connecting to a VM with :
Error detail: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01&msi_res_id=%2Fsubscriptions%2F7b5f97dc-3c4d-424d-8288-bdde3891f242%2FresourceGroups%2Frg-iac-aca-petclinic-mic-srv%2Fproviders%2FMicrosoft.ContainerRegistry%2Fregistries%2Facrpetcliaca (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fae066d8dc0>, 'Connection to 169.254.169.254 timed out. (connect timeout=None)'))
then az acr login --name acrpetcliaca runs fine
Login Succeeded
az acr build fails with the same error message
Step 18/25 : RUN ls -al /tmp/app
---> Running in b9ffd5ce4bc7
total 8
drwxr-xr-x 2 root root 4096 Oct 8 09:14 .
drwxrwxrwt 1 root root 4096 Oct 8 09:14 ..
Removing intermediate container b9ffd5ce4bc7
---> b2e86569b20f
Step 19/25 : COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
---> f3b46f513c3e
Step 20/25 : COPY --from=builder /tmp/app/dependencies/ ./
---> 7f12bfbccbb5
Step 21/25 : COPY --from=builder /tmp/app/snapshot-dependencies/ ./
---> abd632f985c8
Step 22/25 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
failed to export image: failed to create image: failed to get layer sha256:20912aadb176614fb0b1f865eabe101834b5ad73b0fb2c796159ee6ceb32161c: layer does not exist
2022/10/08 09:14:20 Container failed during run: build. No retries remaining.
failed to run step ID: build: exit status 1
What should be implemented in the GH Workflow to sucessfully ACR build
- name: Azure Login
uses: azure/login@v1.4.6 # https://github.com/marketplace/actions/azure-login
with:
creds: ${{ env.credentials }} // This is a SPN NOT an Identity
- name: Build image and push it to ACR
run: |
az acr login --name acrpetcliaca
az acr build --registry acrpetcliaca -g rg-iac-aca-petclinic-mic-srv -t petclinic:adm-test --file "./docker/petclinic-admin-server/Dockerfile" .
@huanwu @AntonioVT you can very also tha the files are correctly extracted incuding tmp/app/sprint-boot-loader running locally :
java -Djarmode=layertools -jar aca-java-petclinic-mic-srv\spring-petclinic-admin-server\target\aca-spring-petclinic-admin-server-2.6.6.jar extract
the issue looks similar to https://github.com/Azure/azure-cli/issues/20413
Hi @ezYakaEagle442 , thanks for providing the information, I have been able to reproduce the issue.
After some investigation, we have found that this is a known issue with Moby and we have mitigation steps. There seems to be a bug when there's a sequence of COPY commands in a multi stage build. There is 1 possible workaround:
RUN true
)
COPY --from=builder ${APP_INSIGHTS_AGENT_JAR_FILE_PATH} ./
COPY --from=builder /tmp/app/dependencies/ ./
COPY --from=builder /tmp/app/snapshot-dependencies/ ./
RUN true
COPY --from=builder /tmp/app/spring-boot-loader/ ./
RUN true
COPY --from=builder /tmp/app/application/ ./
The run will show logs similar to:
// Successful run from az acr build
Step 24/28 : COPY --from=builder /tmp/app/spring-boot-loader/ ./
---> 75ae647c2b16
Step 25/28 : RUN true
---> Running in d30661d783cc
Removing intermediate container d30661d783cc
---> 604e692d6d3e
Step 26/28 : COPY --from=builder /tmp/app/application/ ./
---> 12425a941a63
Step 27/28 : EXPOSE 80 8080 8081 8082 8083 8084 8888 9090
---> Running in 7b0e8c1a3c00
Removing intermediate container 7b0e8c1a3c00
---> a553dff3cca8
Step 28/28 : ENTRYPOINT ["java", "-javaagent:/tmp/app/applicationinsights-agent-3.4.1.jar",
...
Run ID: cc2y was successful after 1m4s
Moby issue references:
https://github.com/moby/moby/issues/37965
https://stackoverflow.com/questions/51115856/docker-failed-to-export-image-failed-to-create-image-failed-to-get-layer
Please let us know if this mitigation step resolves your issue.
- Thanks @huanwu @northtyphoon for helping troubleshoot the issue offline
@AntonioVT @huanwu @northtyphoon Thank you ! I have tested locally and now the ACR build runs fine, I will test later in the GH Workflow.
FYI I did not hit this issue on ARO, which uses CRI-O , not Moby.
See Runner logs at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/actions/runs/3191241482/jobs/5207516396
Workflow at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/.github/workflows/maven-build.yml#L167
see also https://github.com/Azure/azure-cli/issues/20761
@ams0