search

paketo-buildpacks / bellsoft-liberica

A Cloud Native Buildpack that provides the Bellsoft Liberica implementations of JREs and JDKs
Apache License 2.0
74 stars 18 forks source link

Use local downloaded version of bellsoft-liberica #323

Closed bitbaggi closed 2 years ago

bitbaggi commented 2 years ago

Hey guys,

my internet connection is very slow. Can I download your jdk once and the Buildpack uses my downloaded package?

Thank you Pascal

dmikusa commented 2 years ago

Yes! You can use dependency mappings and either specify a location on your LAN from which you can download the dependencies, or you can specify a file:// URL and volume mount the binaries into the container at the expected path. So if you specify file:///binaries/... you can pack build --volume=$PWD/binaries:/binaries ....

Dependency mappings can be tedious, as is downloading the binaries. I created a tool to make this easier, see the binding-tool. The README has some more instructions on how you can use it, but basically bt dm -b paketo-buildpacks/bellsoft-liberica and it'll download all the JDK/JVM binaries and make the bindings for you.

Let me know if you have issues with it or questions.

bitbaggi commented 2 years ago

Hey,

i use the Spring Boot plugin with gradle. Do you know how i can add the Bindings there?

Thank you

scottfrederick commented 2 years ago

The equivalent of the pack build --volume=$PWD/binaries:/binaries ... example that @dmikusa mentioned using the Spring Boot Gradle plugin would be this: 

bootBuildImage {
    bindings = [ "${projectDir}/binaries:/binaries" ]
}
dmikusa commented 2 years ago

I'm going to close this out. If you have any more follow up questions, feel free to reach out. Thanks

ongiant commented 10 months ago

The equivalent of the pack build --volume=$PWD/binaries:/binaries ... example that @dmikusa mentioned using the Spring Boot Gradle plugin would be this:

bootBuildImage {
  bindings = [ "${projectDir}/binaries:/binaries" ]
}

It doesn't work for me. When I run ./gradlew bootBuildImage command, it still tried to download from github repository. And I use network proxy.

  1. my `build.gradle` file: 
    bootBuildImage {
    builder = "docker.io/paketobuildpacks/builder-jammy-base"
    imageName = "${project.name}"
    environment = ["BP_JVM_VERSION": "17"]
    bindings = [ "${project.projectDir}/paketo-buildpacks/binaries:/binaries" ]

    docker {
        publishRegistry {
            username = project.findProperty("registryUsername")
            password = project.findProperty("registryToken")
            url = project.findProperty("registryUrl")
        }
    }
}
  2. std output error message(click here) 
    ➜  catalog-service git:(master) ✗ ./gradlew clean bootBuildImage

> Task :bootBuildImage
Building image 'docker.io/library/catalog-service:latest'

 > Pulling builder image 'docker.io/paketobuildpacks/builder-jammy-base:latest' ..................................................
 > Pulled builder image 'paketobuildpacks/builder-jammy-base@sha256:403cfca1bc740b7663d461f1ec515033b7aba5ecc3955ae951bade382e1a84b6'
 > Pulling run image 'docker.io/paketobuildpacks/run-jammy-base:latest' ..................................................
 > Pulled run image 'paketobuildpacks/run-jammy-base@sha256:18c92f9d53d1b3b941624cb823df3c802782f8eb337ad3a229d14372df4cd27d'
 > Executing lifecycle version v0.18.3
 > Using build cache volume 'pack-cache-c67fa1d5fcb6.build'

 > Running creator
    [creator]     ===> ANALYZING
    [creator]     Restoring data for SBOM from previous image
    [creator]     ===> DETECTING
    [creator]     6 of 26 buildpacks participating
    [creator]     paketo-buildpacks/ca-certificates   3.6.7
    [creator]     paketo-buildpacks/bellsoft-liberica 10.4.4
    [creator]     paketo-buildpacks/syft              1.42.0
    [creator]     paketo-buildpacks/executable-jar    6.8.3
    [creator]     paketo-buildpacks/dist-zip          5.6.8
    [creator]     paketo-buildpacks/spring-boot       5.27.8
    [creator]     ===> RESTORING
    [creator]     Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:java-security-properties" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/bellsoft-liberica:jre" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/syft:syft" from cache
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:helper" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:spring-cloud-bindings" from app image
    [creator]     Restoring metadata for "paketo-buildpacks/spring-boot:web-application-type" from app image
    [creator]     Restoring data for "paketo-buildpacks/syft:syft" from cache
    [creator]     Restoring data for SBOM from cache
    [creator]     ===> BUILDING
    [creator]     
    [creator]     Paketo Buildpack for CA Certificates 3.6.7
    [creator]       https://github.com/paketo-buildpacks/ca-certificates
    [creator]       Launch Helper: Reusing cached layer
    [creator]     
    [creator]     Paketo Buildpack for BellSoft Liberica 10.4.4
    [creator]       https://github.com/paketo-buildpacks/bellsoft-liberica
    [creator]       Build Configuration:
    [creator]         $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
    [creator]         $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
    [creator]         $BP_JVM_TYPE                 JRE                                                          the JVM type - JDK or JRE
    [creator]         $BP_JVM_VERSION              17                                                           the Java version
    [creator]       Launch Configuration:
    [creator]         $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
    [creator]         $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
    [creator]         $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
    [creator]         $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
    [creator]         $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
    [creator]         $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
    [creator]         $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
    [creator]         $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
    [creator]         $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
    [creator]         $BPL_JMX_PORT                5000                                                         configure the JMX port
    [creator]         $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
    [creator]         $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
    [creator]         $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
    [creator]         $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
    [creator]         Using Java version 17 from BP_JVM_VERSION
    [creator]       BellSoft Liberica JRE 17.0.9: Reusing cached layer
    [creator]       Launch Helper: Reusing cached layer
    [creator]       Java Security Properties: Reusing cached layer
    [creator]     
    [creator]     Paketo Buildpack for Syft 1.42.0
    [creator]       https://github.com/paketo-buildpacks/syft
    [creator]         Downloading from https://github.com/anchore/syft/releases/download/v0.99.0/syft_0.99.0_linux_amd64.tar.gz
    [creator]     unable to invoke layer creator
    [creator]     unable to get dependency syft
    [creator]     unable to download https://github.com/anchore/syft/releases/download/v0.99.0/syft_0.99.0_linux_amd64.tar.gz
    [creator]     unable to request https://github.com/anchore/syft/releases/download/v0.99.0/syft_0.99.0_linux_amd64.tar.gz
    [creator]     Get "https://objects.githubusercontent.com/github-production-release-asset-2e65be/262126497/fd542442-6e36-4947-b6fa-a8bccd067902?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240115%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240115T213923Z&X-Amz-Expires=300&X-Amz-Signature=e980b0e630b0cfdb6e602815a57f3220cbc9cbb1d6da641b9d42004346a78b86&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=262126497&response-content-disposition=attachment%3B%20filename%3Dsyft_0.99.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream": net/http: TLS handshake timeout
    [creator]     ERROR: failed to build: exit status 1

> Task :bootBuildImage FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootBuildImage'.
> Builder lifecycle 'creator' failed with status code 51

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD FAILED in 14s
6 actionable tasks: 6 executed

The screenshot: Screenshot_20240116_1

dmikusa commented 10 months ago

@ongiant You need to have a dependency mapping binding which consists of some text files that provide metadata about the binary & where to find the binary. I don't see that in your screenshot. I just see the binaries. The name might be a little confusing in the example above, but the directory for which you add a binding needs to have more than just the binaries.

If you use and run the binding-tool I linked to above, it will pull down everything and set up the binding metadata for you. You then just add your binding to the root directory that binding-tool created.

bootBuildImage {
    bindings = [ "${projectDir}/created-by-binding-tool:/bindings" ]
        environment["SERVICE_BINDING_ROOT"] = "/bindings"
}

If you use binding-tool, then the path inside the container should always be /bindings. In that case, you also need to add an env variable for SERVICE_BINDING_ROOT=/bindings, because the default path is different (see above).

If you have any other questions, please post over on our discussions forum here.

ongiant commented 10 months ago

@dmikusa Thanks for your answer! But I still have some problem and write in this discussion.

gresockj commented 1 month ago

Yes! You can use dependency mappings and either specify a location on your LAN from which you can download the dependencies, or you can specify a file:// URL and volume mount the binaries into the container at the expected path. So if you specify file:///binaries/... you can pack build --volume=$PWD/binaries:/binaries ....

Let me know if you have issues with it or questions. @dmikusa

This looks promising, thanks for the tool! I'm trying bt dm -b paketo-buildpacks/bellsoft-liberica@v10.8.1, but it's taking forever with my slow connection. I really only need one of these tarballs (https://github.com/bell-sw/Liberica/releases/download/21.0.3+12/bellsoft-jre21.0.3+12-linux-amd64.tar.gz). Is there a way to tell it to just grab one dependency? Or is there a way to create a binding for just that one?

dmikusa commented 1 month ago

Is there a way to tell it to just grab one dependency? Or is there a way to create a binding for just that one?

No, not at the moment. The tool doesn't support filters. If you only need one or two, you can download them manually. The links for the downloads are in the buildpack.toml it's the url and sha256 properties of the dependencies you want.

The sha256 hash is the file name, the contents of that file are the file:// path relative to inside your container. The tool uses /binaries/... which is why --volume=$PWD/binaries:/binaries is required with the tool. Then make your local binaries/ directory and put the actual download in that.

It's perfectly fine to do this manually, there's no magic happening. As long as everything lines up, it'll work. The tool is really just an automation of the process, meant for bulk downloading the dependencies and reducing user error.

Hope that helps!

gresockj commented 1 month ago

The sha256 hash is the file name, the contents of that file are the file:// path relative to inside your container.

Great! And where does that file go? In the same directory as the binary itself?

I'm using the spring-boot-maven-plugin, so I'll be specifying the binding as seen here, I think: https://github.com/spring-projects/spring-boot/issues/23518

dmikusa commented 1 month ago

The tool will create a folder structure like this:

bindings/
└── dependency-mapping
    ├── 04d5a8540a5ed57af493137be85ce996ebb3afc86390f685c0f922a9c6fa1d62
    ├── 0d82ad4821b55d276d25486132f708f4cc747e242a9ce256fdf1c7f21979a94c
    ├── 138f8bd3801445391ff19d683ea7f3c4662dcd0e2cd0b8ab2a5d3966b05124f8
    ├── 17843f92dc9de74b161a63c52cc2a4597e5472cf3f6f6d71930fb655b35f9208
    ├── 1f5b781f8bd0d6b85ab2462e4b98d36782a2227fef5b168db174b3959a0ebebe
    ├── 21a18375ab2692b0c5b563e25bc7f58b625705aff5a32e7595bbed27f00772cc
    ├── 2b5c94fe51c7a9094b44a9cdb9e79109b92051793d0530266fc2e7b2f41463aa
    ├── 2bda48088925a5cead341ff9ae8b12c4075937f444827946a268ba773cf0339b
    ├── 2fcb0b438dec18cede304aa645d24b211dd245760b5bb358b4f597eea3721ae5
    ├── 302036befbcb382e1babe404aa952882589e7c72adb5f63b5471863bd84b2928
    ├── 3473d8b3b1bc682e95adfb3ac1d9a59b51b0f43e2b752f2a5b550e4ebfa2fd17
    ├── 3dea0f7a9312c738d22b5e399b6ce9abe13b45b2bc2c04346beb941a94e8a932
    ├── 412e73b5f0ece55fc859059e5f215373eb07ea2fff567e798a1f08b59ad2a82a
    ├── 4ec8826c5fc56f93f1e511d87147aaf8701ef0517c4fae72f24cd656ba866e2a
    ├── 5459d924a420877f171832fc082dec2e9c87647b433810f2661001b9db070621
    ├── 55b341ca1bca5219aafa8ed7c8a2273b81d184dd600d8261c837fc32d2dedae5
    ├── 59543f28f392b41fe7d03230c8ebb918a441213c5e1d3f84bd6e2e8d18e36c71
    ├── 6155654311ca0f7dbdba0571aec0633ab9de2a7a20941f2c959d8413df5b2cdc
    ├── 6fa5aa8a18538bf6c27d422a3a7d811dcd69bd2cc9d61d57e6e044cd31a8e334
    ├── 763d0b1959f52dc5aaddd21b4692d4bd4c4ce61b64a75666b463ed281653a0dd
    ├── 7a066dec17670d846551a2cb7e65b4d49fd94187d13a68f2bc6681dcabfbbb1b
    ├── 8499c0b034dd881cd9a880c44021632422a28dc23d7a81ca0a97b04652245982
    ├── 86df083bd817970404d56d360926d440eb895041ab8d7f580d79b34b19e621e8
    ├── 88227671e7f886aef012066f15e70e19ad36c95972a22ee80b8f88ecc33ad835
    ├── a3d6b774afd9cd02a52a4c2d369422f24228f923d8ba58ecb72503e70ee96df6
    ├── a6d6f945b637386e5bf456c204c1f2d2b7b503a830f118b3a861eb85d7ef6356
    ├── ae9cb1afe327d49a8c049ab588090838e622d9d832b9a1c0523821a6f38d6b4d
    ├── af692dd2a8d343446187df36058f1d27a176672c6ea2465b0c42c95db0e9ce4e
    ├── bbd7b92e5754bef8c7eca585cdce52e6edf015bb7e1c0bfe1bdd9d1837529cb7
    ├── binaries
    │   ├── bellsoft-jdk11.0.16.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jdk11.0.17+7-linux-amd64.tar.gz
    │   ├── bellsoft-jdk11.0.19+7-linux-amd64.tar.gz
    │   ├── bellsoft-jdk17.0.4.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jdk17.0.5+8-linux-amd64.tar.gz
    │   ├── bellsoft-jdk17.0.7+7-linux-amd64.tar.gz
    │   ├── bellsoft-jdk18.0.2.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jdk19.0.1+11-linux-amd64.tar.gz
    │   ├── bellsoft-jdk20.0.1+10-linux-amd64.tar.gz
    │   ├── bellsoft-jdk8u345+1-linux-amd64.tar.gz
    │   ├── bellsoft-jdk8u352+8-linux-amd64.tar.gz
    │   ├── bellsoft-jdk8u372+7-linux-amd64.tar.gz
    │   ├── bellsoft-jre11.0.16.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jre11.0.17+7-linux-amd64.tar.gz
    │   ├── bellsoft-jre11.0.19+7-linux-amd64.tar.gz
    │   ├── bellsoft-jre17.0.4.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jre17.0.5+8-linux-amd64.tar.gz
    │   ├── bellsoft-jre17.0.7+7-linux-amd64.tar.gz
    │   ├── bellsoft-jre18.0.2.1+1-linux-amd64.tar.gz
    │   ├── bellsoft-jre19.0.1+11-linux-amd64.tar.gz
    │   ├── bellsoft-jre20.0.1+10-linux-amd64.tar.gz
    │   ├── bellsoft-jre8u345+1-linux-amd64.tar.gz
    │   ├── bellsoft-jre8u352+8-linux-amd64.tar.gz
    │   ├── bellsoft-jre8u372+7-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk11.0.16.1+1-22.2.0+3-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk11.0.17+7-22.3.0+2-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk11.0.19+7-22.3.2+1-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk17.0.4.1+1-22.2.0+3-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk17.0.5+8-22.3.0+2-linux-amd64.tar.gz
    │   ├── bellsoft-liberica-vm-core-openjdk17.0.7+7-22.3.2+1-linux-amd64.tar.gz
    │   ├── graalvm-ce-java11-linux-amd64-22.3.0.tar.gz
    │   ├── graalvm-ce-java17-linux-amd64-22.3.0.tar.gz
    │   ├── graalvm-ce-java19-linux-amd64-22.3.0.tar.gz
    │   ├── native-image-installable-svm-java11-linux-amd64-22.3.0.jar
    │   ├── native-image-installable-svm-java17-linux-amd64-22.3.0.jar
    │   ├── native-image-installable-svm-java19-linux-amd64-22.3.0.jar
    │   ├── rust-1.70.0-x86_64-unknown-linux-gnu.tar.gz
    │   ├── spring-cloud-bindings-1.10.0.jar
    │   ├── syft_0.60.3_linux_amd64.tar.gz
    │   ├── syft_0.71.0_linux_amd64.tar.gz
    │   └── wlp-webProfile7-22.0.0.12.zip
    ├── c02c90aac559319987a41a07f21b32578bc17b4eee78842f46be8069da9a0299
    ├── c1afaea35fae3c921851c301bf68795c9adbb24575eb64f7194cdae48afb3f56
    ├── c6922752d93191845941a64efcf0c07430f5210977a227897a5c1fe6220e49af
    ├── d1f5c58b65c57ad8a0c7da0c4569ce815ebf1ae503b0741ba1facf27b816d398
    ├── d4200bcc43e5ad4e6949c1b1edc1e59f63066e3a2280d5bd82d0c9b1d67c3f2c
    ├── d98941ef6b79de54e4a4918d3c11a6ea2f6bb4c10a69c75ed8440aded7d313fc
    ├── dc79126c929f9e4dccacbdb8a680cf3600fbe582cb9b1ea3dfb94b2166a59fbd
    ├── e3eeaf929a03a58b61cb400ccdfdf966189233a85911e07e8ca66446319dce4c
    ├── e80d5884fb25840243f231b01c17364fe0f6ef2d68d090d3aa5bf795d1458261
    ├── ebe94c2747015de6a2a427f1c761bc6c808fbc8c92b5829c1b5751ed26fb40e5
    ├── ee56d911dd187d4965fe2d5280e17b76253eb40eb4e5c8582a17cd46ea0168b1
    └── type

The type file needs to be dependency-mapping. The other files are as I mentioned in the previous post.

I like this structure because you only need one volume mount, this directory. You could put your binaries somewhere else and use two volume mounts. Ultimately, the path in the files that have a sha256 hash for their names just needs to resolve inside the container, so file:///to/something/file.tgz needs to point to the binary you mount in.