cache unavailable when rebuild application

[TianMing2018]

I'm trying to build image from a spring-boot application, when I rebuild image ,it loss thc plugin cause it unavailable when rebuild app

Expected Behavior

buildpack cache should be effective when reactive command

Current Behavior

don't hit cache when rebuild app

Possible Solution

Steps to Reproduce

1. create a spring-boot app by this link
2. change pom.xml

<plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <executions>
                            <execution>
                                <id>build-image</id>
                                <goals>
                                    <goal>build-image</goal>
                                </goals>
                                <configuration>
                                    <docker>
                                        <host>tcp://localhost:2375</host>
                                        <tlsVerify>false</tlsVerify>
                                        <publishRegistry>
                                            <url>xxx.com:5000</url>
                                            <username>xxx</username>
                                            <password>xxx</password>
                                        </publishRegistry>
                                    </docker>
                                    <image>
                                        <!--  download proxy,eg jdk and syft-->
                                        <bindings>
                                            <binding>proxy-volume:/platform/bindings/dependency-mapping:ro</binding>
                                        </bindings>
                                        <!--<verboseLogging>true</verboseLogging> -->
                                        <!--                                buildpack -->
                                        <buildpacks>
                                            <buildpack>paketo-buildpacks/java</buildpack>
                                            <buildpack>docker://gcr.io/paketo-buildpacks/health-checker</buildpack>
                                        </buildpacks>
                                        <publish>true</publish>
                                        <name>
                                            docker.myserver.com:5000/userme/${project.artifactId}:${project.version}
                                        </name>
                                        <pullPolicy>IF_NOT_PRESENT</pullPolicy>
                                        <!--                        <verboseLogging>true</verboseLogging>-->
                                        <buildCache>
                                            <volume>
                                                <name>cache-${project.artifactId}.build</name>
                                            </volume>
                                        </buildCache>
                                        <launchCache>
                                            <volume>
                                                <name>cache-${project.artifactId}.launch</name>
                                            </volume>
                                        </launchCache>
                                        <env>
                                            <BP_JVM_TYPE>JDK</BP_JVM_TYPE>
                                            <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                                            <BP_HEALTH_CHECKER_ENABLED>true</BP_HEALTH_CHECKER_ENABLED>
                                            <THC_PORT>8080</THC_PORT>
                                            <THC_PATH>/actuator/health</THC_PATH>
                                        </env>
                                    </image>
                                </configuration>
                            </execution>
                            <execution>
                                <id>build-info</id>
                                <goals>
                                    <goal>build-info</goal>
                                </goals>
                                <configuration>
                                    <additionalProperties>
                                        <encoding.source>${project.build.sourceEncoding}</encoding.source>
                                        <encoding.reporting>${project.reporting.outputEncoding}</encoding.reporting>
                                        <java.source>${maven.compiler.source}</java.source>
                                        <java.target>${maven.compiler.target}</java.target>
                                    </additionalProperties>
                                </configuration>
                            </execution>
                        </executions>
                        <configuration>
                            <excludes>
                                <exclude>
                                    <groupId>org.projectlombok</groupId>
                                    <artifactId>lombok</artifactId>
                                </exclude>
                            </excludes>
                        </configuration>
                    </plugin>

4. Motivations

thc still be unpacked and I can use health check in my docker container after rebuilded app

by the way,it would be better if you push this image to docker too. cause is unavalibale in china

[dmikusa]

Can you provide the build output from a.) the first build without any cache that works & b.) the subsequent build that fails?

Also, when you run the build, set BP_LOG_LEVEL=debug in the <env>..</env> block in your pom.xml. That'll generate a lot more detailed output from the buildpacks.

Lastly, if you could try running the same build using pack that would be helpful too. It would let us know if there is a difference in behavior between pack and the Spring Boot tools.

by the way,it would be better if you push this image to docker too. cause is unavalibale in china

I'm hoping to have this done shortly for all of the Java-related buildpacks. We'll be publishing to both places. Check back in a couple of weeks.

Thanks

[TianMing2018]

I'have push code(include upload build image log) to https://github.com/TianMing2018/paketo-demo ,you can read README.md to get more detail info

[dmikusa]

In log 3, you can see that it's reinstalling thc.

[INFO] [creator] Expected metadata: map[cpes:[cpe:2.3:a:dmikusa-pivotal:tiny-health-checker:0.7.0:::::::] id:thc licenses:[map[type:Apache-2 uri:https://github.com/dmikusa-pivotal/tiny-health-checker/blob/main/LICENSE]] name:Tiny Health Checker purl:pkg:generic/thc@0.7.0?arch=amd64 sha256:d3940b0f347744f9c0ebdc827f46014964a9de45b0d60b20116f6a60bb849a8a stacks:[io.buildpacks.stacks.bionic io.paketo.stacks.tiny ] uri:https://github.com/dmikusa/tiny-health-checker/releases/download/v0.7.0/thc-x86_64-unknown-linux-musl version:0.7.0] [INFO] [creator] Actual metadata: map[]

This is compared to log2 where it has previous metadata:

[INFO] [creator] Expected metadata: map[buildpackInfo:map[clear-env:false description:A Cloud Native Buildpack that adds custom CA certificates to a build and a created image homepage:https://github.com/paketo-buildpacks/ca-certificates id:paketo-buildpacks/ca-certificates keywords:[ca-certificates trust certificates] licenses:[map[type:Apache-2.0 uri:https://github.com/paketo-buildpacks/ca-certificates/blob/main/LICENSE]] name:Paketo Buildpack for CA Certificates sbom-formats:[application/vnd.cyclonedx+json application/vnd.syft+json] version:3.5.1] helperNames:[ca-certificates-helper]] [INFO] [creator] Actual metadata: map[buildpackInfo:map[clear-env:false description:A Cloud Native Buildpack that adds custom CA certificates to a build and a created image homepage:https://github.com/paketo-buildpacks/ca-certificates id:paketo-buildpacks/ca-certificates keywords:[ca-certificates trust certificates] licenses:[map[type:Apache-2.0 uri:https://github.com/paketo-buildpacks/ca-certificates/blob/main/LICENSE]] name:Paketo Buildpack for CA Certificates sbom-formats:[application/vnd.cyclonedx+json application/vnd.syft+json] version:3.5.1] helperNames:[ca-certificates-helper]]

It's checking the expected metadata against the actual and the actual is empty, so it doesn't match. That's why it is reinstalling it.

You should see the lifecycle write:

[INFO] [creator] ===> RESTORING

When it's restoring metadata. In log 1 and 3, it's not restoring any metadata which is why there is no metadata to compare in the buildpack.

Log4 with pack, it has nothing to restore (I'm guessing cause it's the first run) and then in log5 it restores the metadata. I don't see a Log6, so I'm not sure if pack can continue to restore the metadata or if it suffers similar issues when trying to rebuild.

I don't think this is an issue with the health-checker buildpack as it's losing the metadata and reinstalling across all of the buildpacks. It may be that you're losing the volume where the metadata is stored. If you're running this in CI, that can sometimes happen if your CI provider doesn't make volumes available to all workers, so for example, if you run a build on Worker-A the volume is visible there, but when you rebuild on Worker-B the volume isn't present. Pack has an option to store cache in a registry instead of a volume. If you are having issues in CI, you might give that a try and see if it helps.

Other things that might help. Try running with pack build -v that'll give some more verbose output from pack and the lifecycle. Also, there is a line:

[INFO] > Using build cache volume 'cache-paketo-demo.build'

and that tells you the volume name that's being used for the build. You could perhaps try to run something like docker volume ls prior to your builds and see if that volume exists. If it exists, pack & Spring Boot build tools should be able to find it and reuse it so I'm guessing it doesn't exist, but that would allow you to probe a little more. If you run docker volume ls and it doesn't exist, then you know it's not a buildpacks issue and something with your environment.

Hope that helps!

[TianMing2018]

Thanks for you guide, I had redo test by using pack ,the log6 you wished do work like log5,so health-checker work in pack build mode. As for spring-boot, when I clean all caches and re-run test, like this mvn clean install -P paketo -X -l health1.log I got this when I type docker run --rm --volume cache-paketo-demo.build:/tmp/my/build --volume cache-paketo-demo.launch:/tmp/my/launch bash cat /tmp/my/build/committed/io.buildpacks.lifecycle.cache.metadata

{"sbom":{"sha":"sha256:76c60e36e9d13b269410d6047f086ceb024e0def4d90b296ee25d7b2e85d2589"},"buildpacks":[{"key":"paketo-buildpacks/ca-certificates","version":"3.5.1","layers":{}},{"key":"paketo-buildpacks/bellsoft-liberica","version":"9.10.1","layers":{"jdk":{"sha":"sha256:29be8c13e0e711b5c1b8c615bd7fbd242a3bbdb4b105b5117bdb11ac89eda86a","data":{"cert-file":"322a2d01b2bc265427978eb2e0fc96294e9ba4f1b6234a41124068ac125d7593","dependency":{"cpes":["cpe:2.3:a:oracle:jdk:11.0.17:*:*:*:*:*:*:*"],"id":"jdk","licenses":[{"type":"GPL-2.0 WITH Classpath-exception-2.0","uri":"https://openjdk.java.net/legal/gplv2+ce.html"}],"name":"BellSoft Liberica JDK","purl":"pkg:generic/bellsoft-jdk@11.0.17?arch=amd64","sha256":"86df083bd817970404d56d360926d440eb895041ab8d7f580d79b34b19e621e8","stacks":["io.buildpacks.stacks.bionic","io.paketo.stacks.tiny","*"],"uri":"https://github.com/bell-sw/Liberica/releases/download/11.0.17+7/bellsoft-jdk11.0.17+7-linux-amd64.tar.gz","version":"11.0.17"}},"build":true,"launch":true,"cache":true}}},{"key":"paketo-buildpacks/syft","version":"1.23.0","layers":{"syft":{"sha":"sha256:0a610f41820b93ac8b0d2162a229daaf99f8a0ba2d180cd8277fea3e16679388","data":{"cpes":["cpe:2.3:a:anchore:syft:0.62.1:*:*:*:*:*:*:*"],"id":"syft","licenses":[{"type":"Apache-2.0","uri":"https://github.com/anchore/syft/blob/main/LICENSE"}],"name":"Syft","purl":"pkg:generic/anchore-syft@0.62.1","sha256":"89860504694a05a75688991ac24281cb84cfa61d48c973ddee7559fa7fc0a60e","stacks":["io.buildpacks.stacks.bionic","io.paketo.stacks.tiny","*"],"uri":"https://github.com/anchore/syft/releases/download/v0.62.1/syft_0.62.1_linux_amd64.tar.gz","version":"0.62.1"},"build":true,"launch":false,"cache":true}}},{"key":"paketo-buildpacks/executable-jar","version":"6.5.0","layers":{}},{"key":"paketo-buildpacks/dist-zip","version":"5.4.0","layers":{}},{"key":"paketo-buildpacks/spring-boot","version":"5.22.0","layers":{}},{"key":"paketo-buildpacks/environment-variables","version":"4.4.0","layers":{}},{"key":"paketo-buildpacks/health-checker","version":"1.1.0","layers":{}}]}

It seem that no layers left after build, maybe it's the real reason why re-build in spring-boot will fail.However, It's wired the actual metamap still get the health-checker meta info. I agree with you about this issue is not health-checker's . Could you tell me where sholud I report this issue ?

[dmikusa]

Quick question. What is the environment where you running this? It sounds like it's a CI, but what CI are you using?

I'm asking cause I just talked with another user that was having some issues on Github Enterprise (running Github Actions). Their workers had some unique configurations and it was causing problems with their builds.

The cached data needs to persist from build-to-build in the Docker daemon (that's the expected behavior from build tools), but with a CI system the system might not persist the cache from run-to-run, or perhaps it may persist the cache when convenient, so if you land on the same worker then the Docker daemon will still have the cache but if you land on a different worker then it won't have the cache. At any rate, you may want to talk with your CI administrators (if feasible) and see if there could be some environment specific cause for this behavior.

You might also try this suggestion.

[TianMing2018]

First,I'm using Windows and WSL( Ubuntu 22.04.1 LTS (GNU/Linux 5.15.79.1-microsoft-standard-WSL2 x86_64)), spring-boot-maven plugin (is it you asked cli), I had keep using same machine and run whole test in Ubuntu or Windows in a 'lifecycle'.

Docker(installed in Windows Host) running all days and without restart, I just clean all volumes or images when a re-start run test

here is the maven information in ubuntu and windows

ubuntu@DESKTOP-T0EV4J4:~/test/paketo-demo$ mvn -v
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /mnt/c/Software/apache-maven-3.8.6
Java version: 11.0.17, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.15.79.1-microsoft-standard-wsl2", arch: "amd64", family: "unix"

PS D:\code\myconfig\docker\portainer> mvn -v
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: C:\Software\apache-maven-3.8.6
Java version: 11.0.16.1, vendor: BellSoft, runtime: C:\Software\Java\jdk-11.0.16.1
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 11", version: "10.0", arch: "amd64", family: "windows"

I had already named build cache and launch cache,but there was two other volume caches name prefix with pack- , I didn't find any infomation about that. I'll try to report issue on here github.com/spring-projects/spring-boot/tree/main/spring-boot-project/spring-boot-tools/spring-boot-maven-plugin

[scottfrederick]

I can re-create the problem of the disappearing /thc/bin/thc with both the Spring Boot plugin and the pack CLI. The trigger seems to be setting the THC_PORT and THC_PATH environment variables.

I'm using dive to inspect the image built with this command:

pack build pack-demo \
  --builder paketobuildpacks/builder:base \
  --buildpack paketo-buildpacks/java \
  --buildpack docker://gcr.io/paketo-buildpacks/health-checker \
  --volume proxy-volume:/platform/bindings/dependency-mapping:ro \
  --env BP_HEALTH_CHECKER_ENABLED=true \
  --env BP_LOG_LEVEL=DEBUG \
  --env THC_PORT=8180 \
  --env THC_PATH="/actuator/health" \
  --path target/paketo-demo-0.0.1-SNAPSHOT.war

The first time I run this command, I see a layer with these contents:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        ├── bin                    
        │   └── thc                
        └── env.launch             
            └── health-check
                ├── THC_PATH.default
                └── THC_PORT.default

The second time I run the command, I see a layer with these contents:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        └── env.launch             
            └── health-check
                ├── THC_PATH.default
                └── THC_PORT.default

If the THC_PORT and THC_PATH environment variables are removed from the Spring Boot configuration or the pack command, the layer contains only the binary after every build:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        └── bin                    
            └── thc                

[dmikusa]

Thanks for the information you've provided. To this point it is not clear what's causing this. I was talking to @scottfrederick who maintains the Boot plugins, and we're not sure what's happening.

We would like to try this. Please capture the full output of each step and label the step.

1.) Run the first build 2.) Run `docker volume ls' this will list the volumes. 3.) Repeat 1.) and 2.) a second time. It's my understanding that this should succeed. Tell me if that's incorrect. 4.) Repeat 1.) and 2.) again. It's my understanding this will fail. Tell me if that's incorrect. 5.) Repeat 1.) and 2.) one more time. Tell me is this works or if it fails. Trying to see if it goes back to working or if it continues to fail.

Lastly, please let me confirm my understanding of the issue:

The Buildpacks run and produce functional images every time. There are no build errors or run error. You are concerned about caching and the fact that Buildpacks will unexpectedly reinstall all binaries. Not just the health checker binary, everything is reinstalled.

Please correct anything that I may have misunderstood, thanks.

[TianMing2018]

Thanks for the information you've provided. To this point it is not clear what's causing this. I was talking to @scottfrederick who maintains the Boot plugins, and we're not sure what's happening.

We would like to try this. Please capture the full output of each step and label the step.

1.) Run the first build 2.) Run `docker volume ls' this will list the volumes. 3.) Repeat 1.) and 2.) a second time. It's my understanding that this should succeed. Tell me if that's incorrect. 4.) Repeat 1.) and 2.) again. It's my understanding this will fail. Tell me if that's incorrect. 5.) Repeat 1.) and 2.) one more time. Tell me is this works or if it fails. Trying to see if it goes back to working or if it continues to fail.

Lastly, please let me confirm my understanding of the issue:

The Buildpacks run and produce functional images every time. There are no build errors or run error. You are concerned about caching and the fact that Buildpacks will unexpectedly reinstall all binaries. Not just the health checker binary, everything is reinstalled.

Please correct anything that I may have misunderstood, thanks.

Only the first test write bin/thc and failed on last test. It seems that volume have been recreated everytime ? you can find detial description in my repo TianMing2018/paketo-demo

[TianMing2018]

I can re-create the problem of the disappearing /thc/bin/thc with both the Spring Boot plugin and the pack CLI. The trigger seems to be setting the THC_PORT and THC_PATH environment variables.

I'm using dive to inspect the image built with this command:

pack build pack-demo \
  --builder paketobuildpacks/builder:base \
  --buildpack paketo-buildpacks/java \
  --buildpack docker://gcr.io/paketo-buildpacks/health-checker \
  --volume proxy-volume:/platform/bindings/dependency-mapping:ro \
  --env BP_HEALTH_CHECKER_ENABLED=true \
  --env BP_LOG_LEVEL=DEBUG \
  --env THC_PORT=8180 \
  --env THC_PATH="/actuator/health" \
  --path target/paketo-demo-0.0.1-SNAPSHOT.war

The first time I run this command, I see a layer with these contents:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        ├── bin                    
        │   └── thc                
        └── env.launch             
            └── health-check
                ├── THC_PATH.default
                └── THC_PORT.default

The second time I run the command, I see a layer with these contents:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        └── env.launch             
            └── health-check
                ├── THC_PATH.default
                └── THC_PORT.default

If the THC_PORT and THC_PATH environment variables are removed from the Spring Boot configuration or the pack command, the layer contains only the binary after every build:

layers
└── paketo-buildpacks_health-checker
    └── thc                     
        └── bin                    
            └── thc                

My fault,should transform command fully as spring-boot-maven, I had re-test both in spring-boot way and pack way. It seems that volume have been recreated everytime ? you can find detial description in my repo TianMing2018/paketo-demo

[dmikusa]

OK, my apologies. I misunderstood the issue here. I see what you're saying now.

I think this is because of this code happening outside the layer contributor here the env variables are supposed to be attached to the layer, so it's probably recreating the layer.

Testing this now.

[dmikusa]

OK, I did a quick test and it seems to correct the problem. I've got #28 to fix this.

If you can access dmikusa/paketo-buildpacks-health-checker on Docker Hub, I've published the code from #28 there.

https://hub.docker.com/r/dmikusa/paketo-buildpacks-health-checker

Let me know if you can try it & if it helps. Thanks

[TianMing2018]

OK, I did a quick test and it seems to correct the problem. I've got #28 to fix this.

If you can access dmikusa/paketo-buildpacks-health-checker on Docker Hub, I've published the code from #28 there.

https://hub.docker.com/r/dmikusa/paketo-buildpacks-health-checker

Let me know if you can try it & if it helps. Thanks

Thanks. here are some information you maybe interested

1. I can access dmikusa/paketo-buildpacks-health-checker on Docker Hub
2. How to use THC_PATH.default correctly? I didn't see any docs about it, more detail description lay at bottome of this reply
3. It works when I use spring-boot way to rebuild three times,and they all have thc client ,releated logs prefix with sb-health in build order

But when I want to test on pack mode , I try to load the buildpack through pack buildpack pull dmikusa/paketo-buildpacks-health-checker , there was always timeout ,handshake , unexpected EOF ,It took my whole day but I can't make it. So it's hard to definitely say that this bug was fixed in pack mode( It not so easy to use github and google cloud registry (gcr.io?) smoothly). I would be appreciate if you can try it by using my repo TianMing2018/paketo-demo.

By the way, when I attach container and run /layers/paketo-buildpacks_health-checker/thc/bin/thc ,the default setting eg THC_PATH.default did't works , is there any misunderstooding?, what am I supposed to do more ?

└── paketo-buildpacks_health-checker
    └── thc                     
        ├── bin                    
        │   └── thc                
        └── env.launch             
           └── health-check
                ├── THC_PATH.default
                └── THC_PORT.default

[dmikusa]

By the way, when I attach container and run /layers/paketo-buildpacks_health-checker/thc/bin/thc ,the default setting eg THC_PATH.default did't works , is there any misunderstooding?, what am I supposed to do more ?

That is usually because of the way you attach to the container. You need to invoke the launcher so that it will set all of the buildpack specified env variables for you. See the docs here for an example.

[dmikusa]

It works when I use spring-boot way to rebuild three times,and they all have thc client ,releated logs prefix with sb-health in build order

OK, that's a good sign. I went and cut a release of the buildpack. This should be ready shortly if you want to try the official build.

[dmikusa]

I'm going to mark this as solved in 1.2.0. If anyone still has issues, please reopen this issue.