Buildpacks produce quite big image

[andrejusc]

Hi,

I'm comparing image I'm building directly with Docker file like such with Rust static executable inside it:

FROM scratch
COPY target/x86_64-unknown-linux-gnu/release/function /function
CMD [ "/function" ]

where I'm getting as output - image of about 14MB in size:

$ podman image ls | grep sample
localhost/sample                                                   latest         9f76d50aaa01  8 minutes ago  13.9 MB

and layers:

$ podman history sample
ID            CREATED        CREATED BY                                     SIZE        COMMENT
8144b031a549  8 minutes ago  /bin/sh -c #(nop) CMD [ "/function" ]          0 B         FROM 8144b031a549
<missing>     8 minutes ago  /bin/sh -c #(nop) COPY file:3b28c741a95a71...  13.9 MB   

And then I have image for same Rust function produced via buildpacks - about 107MB:

$ podman image ls | grep buildpack-sample
localhost/buildpack-sample                                         latest         181d2f0215c3  43 years ago    107 MB

and layers in it:

$ podman history buildpack-sample
ID            CREATED       CREATED BY  SIZE        COMMENT
181d2f0215c3  43 years ago              3.07 kB     
<missing>     43 years ago              4.1 kB      
<missing>     43 years ago              2.31 MB     
<missing>     43 years ago              2.56 kB     
<missing>     43 years ago              362 kB      
<missing>     43 years ago              27.6 kB     
<missing>     43 years ago              12.8 MB     
401511bdd9f6  43 years ago              2.56 kB     
<missing>     43 years ago              4.61 kB     
<missing>     43 years ago              4.1 kB      
<missing>     43 years ago              25.9 MB     
<missing>     43 years ago              3.07 kB     
<missing>     43 years ago              3.07 kB     
<missing>     43 years ago              65.5 MB     

What is the cause of such big difference? Any specific setting/config to tweak somewhere to make it smaller via buildpacks as well?

[andrejusc]

ok, Readme recommends usage of tiny. So tried it for function building this way:

func_linux_amd64 build --image buildpack-smaple --verbose --builder-image=gcr.io/paketo-buildpacks/builder:tiny

but such faced then this issue:

...
Adding buildpack paketo-community/rust version 0.27.0 to builder
Adding buildpack paketo-community/rust-dist version 1.16.1 to builder
Adding buildpack paketo-community/rustup version 1.8.1 to builder
Adding buildpack paketo-buildpacks/procfile version 5.6.0 to builder
Adding buildpack paketo-buildpacks/syft version 1.26.0 to builder
Adding buildpack paketo-community/cargo version 0.9.0 to builder
Setting custom order
Creating builder with the following buildpacks:
-> paketo-buildpacks/go@0.2.4
-> paketo-buildpacks/dep@0.0.172
-> paketo-buildpacks/dep-ensure@0.0.33
-> paketo-buildpacks/go-build@0.1.0
-> paketo-buildpacks/go-dist@0.2.3
-> paketo-buildpacks/go-mod-vendor@0.0.169
-> paketo-buildpacks/java-native-image@4.3.0
-> paketo-buildpacks/maven@3.2.1
-> paketo-buildpacks/spring-boot@3.4.0
-> paketo-buildpacks/leiningen@1.2.1
-> paketo-buildpacks/procfile@3.0.0
-> paketo-buildpacks/environment-variables@2.1.2
-> paketo-buildpacks/executable-jar@3.1.3
-> paketo-buildpacks/graalvm@3.3.1
-> paketo-buildpacks/gradle@3.3.1
-> paketo-buildpacks/image-labels@2.0.6
-> paketo-buildpacks/sbt@3.4.1
-> paketo-buildpacks/spring-boot-native-image@1.6.1
-> paketo-community/rust@0.27.0
-> paketo-community/rust-dist@1.16.1
-> paketo-community/rustup@1.8.1
-> paketo-buildpacks/procfile@5.6.0
-> paketo-buildpacks/syft@1.26.0
-> paketo-community/cargo@0.9.0
Error: validating buildpacks: buildpack paketo-community/rust@0.27.0 (Buildpack API 0.7) is incompatible with lifecycle 0.9.3 (Buildpack API(s) 0.2, 0.3, 0.4)

Not sure how to overcome this one yet.

[andrejusc]

I'm using such version of func cli:

$ ./func_linux_amd64 --version
v1.9.2

which to my understanding is coming from here: https://github.com/knative/func/blob/knative-v1.9.2/go.mod

which references this: github.com/buildpacks/pack v0.28.0

which in turn references this: github.com/buildpacks/lifecycle v0.15.1

which has this: https://github.com/buildpacks/lifecycle/blob/v0.15.1/api/apis.go

and such block:

var (
    Platform  = newApisMustParse([]string{"0.3", "0.4", "0.5", "0.6", "0.7", "0.8", "0.9", "0.10"}, nil)
    Buildpack = newApisMustParse([]string{"0.2", "0.3", "0.4", "0.5", "0.6", "0.7", "0.8", "0.9"}, nil)
)

But I'm not familiar with such and how it leads to error in above comment

[andrejusc]

Further analysis reveals such:

$ podman inspect gcr.io/paketo-buildpacks/builder:tiny | grep -oP "lifecycle.*}"
lifecycle\":{\"version\":\"0.9.3\",\"api\":{\"buildpack\":\"0.2\",\"platform\":\"0.3\"},\"apis\":{\"buildpack\":{\"deprecated\":[],\"supported\":[\"0.2\",\"0.3\",\"0.4\"]},\"platform\":{\"deprecated\":[],\"supported\":[\"0.3\",\"0.4\"]}}},\"createdBy\":{\"name\":\"Pack CLI\",\"version\":\"0.15.0+git-49ad805.build-1613\"}}

so at least such explains where very old lifecycle 0.9.3 is coming from then

[andrejusc]

And for previously used base I have lifecycle 0.16.1:

$ podman inspect gcr.io/paketo-buildpacks/builder:base | grep -oP "lifecycle.*}"
lifecycle\":{\"version\":\"0.16.1\",\"api\":{\"buildpack\":\"0.2\",\"platform\":\"0.3\"},\"apis\":{\"buildpack\":{\"deprecated\":[\"0.2\",\"0.3\",\"0.4\",\"0.5\",\"0.6\"],\"supported\":[\"0.2\",\"0.3\",\"0.4\",\"0.5\",\"0.6\",\"0.7\",\"0.8\",\"0.9\"]},\"platform\":{\"deprecated\":[\"0.3\",\"0.4\",\"0.5\",\"0.6\"],\"supported\":[\"0.3\",\"0.4\",\"0.5\",\"0.6\",\"0.7\",\"0.8\",\"0.9\",\"0.10\",\"0.11\"]}}},\"createdBy\":{\"name\":\"Pack CLI\",\"version\":\"0.29.0+git-95c8060.build-4209\"}}

[andrejusc]

@dmikusa - may you know if my observations here are correct and that gcr.io/paketo-buildpacks/builder:tiny image is outdated and of no use by now? Would be still nice to have some common up-to-date common tiny image somewhere.

[ryanmoran]

The GCR registry is a legacy location for these builder images. We no longer post new images there. You can see a listing of all the the available builder images which are now hosted at DockerHub here: https://paketo.io/docs/reference/builders-reference/.

[andrejusc]

@ryanmoran - I see, confirming that taken from DockerHub tiny image has appropriate lifecycle 0.16.1version in it:

$ podman inspect docker.io/paketobuildpacks/builder:tiny | grep -oP "lifecycle.*}"
lifecycle\":{\"version\":\"0.16.1\",\"api\":{\"buildpack\":\"0.2\",\"platform\":\"0.3\"},\"apis\":{\"buildpack\":{\"deprecated\":[\"0.2\",\"0.3\",\"0.4\",\"0.5\",\"0.6\"],\"supported\":[\"0.2\",\"0.3\",\"0.4\",\"0.5\",\"0.6\",\"0.7\",\"0.8\",\"0.9\"]},\"platform\":{\"deprecated\":[\"0.3\",\"0.4\",\"0.5\",\"0.6\"],\"supported\":[\"0.3\",\"0.4\",\"0.5\",\"0.6\",\"0.7\",\"0.8\",\"0.9\",\"0.10\",\"0.11\"]}}},\"createdBy\":{\"name\":\"Pack CLI\",\"version\":\"0.29.0+git-95c8060.build-4209\"}}

Would it be possible to update this repo's main Readme and add your note above about CGR registry as a legacy?

[dmikusa]

Sorry about the bad docs there. Updated in this PR: https://github.com/paketo-community/rust/pull/289

[andrejusc]

@dmikusa - based on all above discussion - I've switched now to use such builder image: docker.io/paketobuildpacks/builder:tiny

and explicitly set BP_RUSTUP_INIT_LIBCto musl, so initially in output getting such:

Paketo Buildpack for Rustup 1.8.1
  https://github.com/paketo-community/rustup
  Build Configuration:
    $BP_RUSTUP_ENABLED       true     use rustup to install Rust
    $BP_RUSTUP_INIT_LIBC     musl     libc implementation: gnu or musl
    $BP_RUSTUP_INIT_VERSION  1        the rustup version
    $BP_RUST_PROFILE         minimal  the Rust profile to install
    $BP_RUST_TARGET                   an additional Rust target to install
    $BP_RUST_TOOLCHAIN       stable   the Rust toolchain or version number to install
  Rustup (musl libc) 1.25.2: Contributing to layer
    Downloading from https://static.rust-lang.org/rustup/archive/1.25.2/x86_64-unknown-linux-musl/rustup-init
    Verifying checksum
    Copying to /layers/paketo-community_rustup/rustup-init-musl/bin
  Cargo: Contributing to layer
  Rustup: Reusing cached layer
  Rust: Contributing to layer
    Installing Rust
      info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'

        stable-x86_64-unknown-linux-gnu unchanged - rustc 1.68.2 (9eb3afe9e 2023-03-27)

      info: checking for self-updates
      info: downloading component 'rust-std' for 'x86_64-unknown-linux-musl'
      info: installing component 'rust-std' for 'x86_64-unknown-linux-musl'

but then hitting such error:

      error: failed to run custom build command for `zstd-sys v2.0.7+zstd.1.5.4`

      Caused by:
        process didn't exit successfully: `/workspace/target/release/build/zstd-sys-4c214eeedc489111/build-script-build` (exit status: 1)
        --- stdout
        cargo:rerun-if-env-changed=ZSTD_SYS_USE_PKG_CONFIG
        TARGET = Some("x86_64-unknown-linux-musl")
        OPT_LEVEL = Some("3")
        HOST = Some("x86_64-unknown-linux-gnu")
        cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
        CC_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
        CC_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CC
        TARGET_CC = None
        cargo:rerun-if-env-changed=CC
        CC = None
 ...
        CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
        running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-o" "/workspace/target/x86_64-unknown-linux-musl/release/build/zstd-sys-b4af3a3ac6cc6c6e/out/zstd/lib/common/debug.o" "-c" "zstd/lib/common/debug.c"
...
        --- stderr  

        error occurred: Failed to find tool. Is `musl-gcc` installed?

Anything to check in such case? Based on current Readme - my understanding is that tiny builder will install if needed muslspecific parts, but maybe that is incorrect understanding?

[dmikusa]

Rustup (musl libc) 1.25.2: Contributing to layer Downloading from https://static.rust-lang.org/rustup/archive/1.25.2/x86_64-unknown-linux-musl/rustup-init Verifying checksum Copying to /layers/paketo-community_rustup/rustup-init-musl/bin Cargo: Contributing to layer Rustup: Reusing cached layer Rust: Contributing to layer Installing Rust info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'

I believe this is a caching issue in the buildpack. You've gone from using the base stack to the tiny stack, which for Rust means you need to use musl (see the note in the README for why). That part is working. The buildpack is installing Rustup which is good, but then Rustup doesn't run because the layer is cached. That's a bug. I'll open an issue about this.

https://github.com/paketo-community/rustup/issues/132

What you can do to work around this is to clear the buildpack cache. If you run your next pack build with the --clear-cache flag, that should clear it out and you should see this line go away Rustup: Reusing cached layer. Don't forget to remove that argument on subsequent builds.

Thanks for all your feedback, btw!

[andrejusc]

@dmikusa - as in my use case I'm executing not pack directly - is there some analogy of --clear-cache functionality to be activated via something like BP_RUSTUP_CACHE_CLEAR: true (which by default could be false) flag/setting? That would suit my needs then and I think would be awesome from other third-parties utilizing buildpacks functionality. What do you think?

[andrejusc]

So I found that caches are basically presented in my case under here ~/.local/share/containers/storage/volumes/:

drwx------ 3 user user 4096 Mar 31 20:37 pack-my-sample_latest-4f438fa4aa91.build
drwx------ 3 user user 4096 Mar 31 20:37 pack-my-sample_latest-4f438fa4aa91.launch

and after trying to play with cleanup - now I'm facing new issue with tiny builder usage:

...
===> BUILDING
Starting build
Running build for buildpack paketo-community/rustup@1.8.1
Looking up buildpack
Finding plan
Creating plan directory
Preparing paths
Running build command

Paketo Buildpack for Rustup 1.8.1
  https://github.com/paketo-community/rustup
  Build Configuration:
    $BP_RUSTUP_ENABLED       true     use rustup to install Rust
    $BP_RUSTUP_INIT_LIBC     musl     libc implementation: gnu or musl
    $BP_RUSTUP_INIT_VERSION  1        the rustup version
    $BP_RUST_PROFILE         minimal  the Rust profile to install
    $BP_RUST_TARGET                   an additional Rust target to install
    $BP_RUST_TOOLCHAIN       stable   the Rust toolchain or version number to install
  Rustup (musl libc) 1.25.2: Contributing to layer
    Downloading from https://static.rust-lang.org/rustup/archive/1.25.2/x86_64-unknown-linux-musl/rustup-init
    Verifying checksum
    Copying to /layers/paketo-community_rustup/rustup-init-musl/bin
  Cargo: Contributing to layer
  Rustup: Contributing to layer
    Installing Rustup
  Rust: Contributing to layer
    Installing Rust
      info: syncing channel updates for 'stable-x86_64-unknown-linux-musl'
      info: latest update on 2023-03-28, rust version 1.68.2 (9eb3afe9e 2023-03-27)
      info: downloading component 'cargo'
      info: downloading component 'rust-std'
      info: downloading component 'rustc'
      info: installing component 'cargo'
      info: installing component 'rust-std'
      info: installing component 'rustc'

        stable-x86_64-unknown-linux-musl installed - (error reading rustc version)

      info: default toolchain set to 'stable-x86_64-unknown-linux-musl'
      info: checking for self-updates
      info: component 'rust-std' for target 'x86_64-unknown-linux-musl' is up to date
unable to invoke layer creator
unable to contribute Rust layer
error executing 'rustc --version':
 Combined Output: error: command failed: 'rustc': No such file or directory (os error 2)
: 
exit status 1
ERROR: failed to build: exit status 1
Error: executing lifecycle: failed with status code: 51

why it's not happy now about rustc?

[dmikusa]

By default, it pack will store cached build information in volumes. In Docker you can docker volume ls and see them. You can also delete the volumes that way, it's like clearing the cache. I assume Podman has the same commands but I don't have it installed to check.

You can also just change the name of your application/image. The cache is scoped to the application name, so if the name changes then it should be treated like a new fresh build.

[dmikusa]

why it's not happy now about rustc?

That is strange. The buildpack runs rustc --version so that we can include that in the cached information. It enables us to automatically invalidate the cache when a new version of rust is available. It seems that it's unable to run that for some reason.

It's weird, even rustup is having trouble. It should print the version as well.

stable-x86_64-unknown-linux-musl installed - (error reading rustc version)

I'm not sure off-hand what might be causing that and it's hard to say without knowing more about your setup.

I will say that builds are a little trickier on tiny because of using musl. You can try switching to the base stack and builder, which will work with gnu libc.

[andrejusc]

I'm not sure off-hand what might be causing that and it's hard to say without knowing more about your setup.

I don't see any tiny image usage under .github/workflows in this repo and if it mentioned as supported in Readme - I think we need at least 1 evaluation flow definition there. Just to prove it works somewhere else as expected and then it's down to someone's setup's specifics.

I could work on adding that if you could point me where baseimage is used for some example build in workflows.

[dmikusa]

I could work on adding that if you could point me where base image is used for some example build in workflows.

Not a base image, sorry. I didn't explain that well. There are three variants of the stack and builder that Paketo publishes: tiny, base, and full. I was suggesting that you give the base image a try instead. Try switching to paketobuildpacks/builder-jammy-base as your builder.

Another possibility is that you could be hitting an issue with the Jammy stack. There are some differences with the Jammy stack, security has been tightened down some. You may be hitting something there. If you want to try using paketobuildpacks/builder:tiny or paketobuildpacks/builder:base that is the older stack. If that works and Jammy fails, let me know and I can look into that more. I did try running builds on both and they both worked for me.

[andrejusc]

Ok, culprit here seems in my attempt to set explicitly builder's setting BP_RUSTUP_INIT_LIBC to musl. When it set - both builder-jammy-base and builder-jammy-tiny are failing.

And when it unset - this part of tiny builder output is interesting then in a sense of it using gnu toolchain and not musl:

      info: installing component 'rustc'

        stable-x86_64-unknown-linux-gnu installed - rustc 1.68.2 (9eb3afe9e 2023-03-27)

      info: default toolchain set to 'stable-x86_64-unknown-linux-gnu'
      info: checking for self-updates
      info: component 'rust-std' for target 'x86_64-unknown-linux-gnu' is up to date

and that Readme says I need to use x86_64-unknown-linux-musl as a target. Seems some disconnect in that statement then.

And when I have BP_RUSTUP_INIT_LIBC set to musl again - then tiny fails on such:

      info: installing component 'rustc'

        stable-x86_64-unknown-linux-musl installed - (error reading rustc version)

      info: default toolchain set to 'stable-x86_64-unknown-linux-musl'
      info: checking for self-updates
      info: downloading component 'rust-std' for 'x86_64-unknown-linux-gnu'
      info: installing component 'rust-std' for 'x86_64-unknown-linux-gnu'

And when now builder-jammy-tiny works - I'm getting not that big 107MB image as before, but now smaller of about 40MB:

$ podman image ls | grep buildpack-sample
localhost/buildpack-sample                                         latest         d99a6ccc68e9  43 years ago  39.5 MB

Such size is near comparable to my direct Dockerfile approach, so probably could live for now with such, unless some other optimization possible.

[andrejusc]

Additionally now tried with builder paketobuildpacks/builder-jammy-buildpackless-tiny to see if any difference in produced image size - so far same as for builder-jammy-tiny before, i.e. 39.5MB.

[dmikusa]

And when I have BP_RUSTUP_INIT_LIBC set to musl again

Ok, yes. I think that would cause the issue. This setting is for something different. It installs the underlying Rust tools compiled using musl libc, but that would require a base image that is also compatible with musl. The Paketo builder images use gnu libc, so that needs to stay as the default for Paketo builder images. That's why it can't run the binary because there's an incompatible libc (my understanding at least).

What you want is to have the tools compiled against gnu libc so it's compatible with the Paketo builders but have a Rust target of musl installed. That will compile your application against musl libc.

and that Readme says I need to use x86_64-unknown-linux-musl as a target. Seems some disconnect in that statement then.

OK, I know what's happening here. This is a Jammy thing. I've logged some issues to get this fixed.

I think you can make this work if you do two things:

1. Add a rust-toolchain.toml and set targets = ["x86_64-unknown-linux-musl"].

2. Set BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-musl'

I think that will make cargo build the right target.

Such size is near comparable to my direct Dockerfile approach, so probably could live for now with such, unless some other optimization possible.

👍 Excellent. I don't think you'll end up with a smaller image using musl. That would statically compile in more code, so it would probably make the image a bit larger.

The tiny image is about 22M, so you can see the rest is your app binary. Down the road, I want to add support for upx which can compress binaries. I've seen good results with it compressing binaries up to 50%. If you got that kind of compression ratio upx could take down the image size to maybe 30M.

[andrejusc]

The tiny image is about 22M, so you can see the rest is your app binary.

So if looking into this:

$ podman history docker.io/paketobuildpacks/run-jammy-tiny:latest
ID            CREATED        CREATED BY                SIZE        COMMENT
61bab0020545  292 years ago                            2.56 kB     
<missing>     292 years ago                            4.1 kB      
<missing>     3 days ago     COPY /tiny/ / # buildkit  24 MB       buildkit.dockerfile.v0

i.e. some buildkit is inside that image. Then last question I have here - why such is part of my final image? Could this buildkit layer be excluded by some means while using some flag/option passed to builder? Hoping that my question makes sense, but please correct me otherwise.

[dmikusa]

That is just the base layer from the run image. It looks a little weird but that is just because of the way the packages get installed and then copied into the final run image at build time. It happens like that because there are no tools to install packages in the run image, so you can't just run apt to install the packages. We install them to a prefixed location /tiny running on an image that has the package tools, then copy that prefix location over to the final run image.

[andrejusc]

Let me rephrase - what is the purpose of that run layer within my final image if all I have is built static executable, which could run without anything in addition as shown at the beginning of this thread inside my Dockerfile?

[dmikusa]

There are a few things that the buildpack spec requires, like the presence of user/groups so if you had a scratch image I don't think that would comply (probably would fail, I haven't tried) with the buildpacks launcher. That said, most of that is glibc, openssl, trusted tls certs, and timezone data. You may or may not need those for your app, but even for statically compiled Rust apps you might need TLS certs or timezone data.

It all boils down to Paketo trying to provide a shared base image that many different apps can use. We don't add much to the tiny stack, but these things are pretty broadly applicable so they are in the tiny stack.

There is a static stack that is even more trimmed down, but it's pretty new. I'm not 100% sure the state of it. See https://github.com/paketo-buildpacks/jammy-static-stack. This only has the files required by buildpacks, trusted TLS cert and timezone data. I don't think there is a builder, but you should be able to try it if you add --run-image docker.io/paketobuildpacks/run-jammy-static:latest to your pack build command. That should swap in the static run image.

[robdimsdale]

Just to add to what @dmikusa said, the goal of the static stack is to enable Buildpacks-compliant minimal images for statically-linked binaries. It's experimental at the moment, and we're looking for feedback. I would expect that a statically-linked, self-contained rust application should work on the static stack. If it doesn't, that's useful feedback and we'd appreciate you filing an issue with the error.

[andrejusc]

@dmikusa - trying this advice:

I don't think there is a builder, but you should be able to try it if you add --run-image docker.io/paketobuildpacks/run-jammy-static:latest to your pack build command.

and using overall build command in absence of static builder so far like such:

pack build buildpack-sample --path=. --verbose --builder=docker.io/paketobuildpacks/builder-jammy-tiny --trust-builder --docker-host=inherit --run-image docker.io/paketobuildpacks/run-jammy-static:latest

leads to such error:

ERROR: failed to build: invalid run-image 'docker.io/paketobuildpacks/run-jammy-static:latest': run-image stack id 'io.buildpacks.stacks.jammy.static' does not match builder stack 'io.buildpacks.stacks.jammy.tiny'

If such is valid error and you think that it should be allowed to mix stack and runner - I'll then raise separate issue as I'm newbie here and not yet aware fully of what is allowed and what not.

[robdimsdale]

Ah, no, I don't think you can replace the run image from a builder with a stack with a different ID.

I think this is an indication that we need to create a builder for the static image cc @ForestEckhardt

[robdimsdale]

@andrejusc we should be able to create a static builder for you in the next day or so.

[andrejusc]

@ryanmoran - cool, will be waiting quietly for that then and will be your beta tester ;)

[dmikusa]

Thanks @robdimsdale

[robdimsdale]

@andrejusc the static buildpackless builder has been released and you can use it directly as follows (using your command from earlier):

pack build buildpack-sample \
  --path=. \
  --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder \
  --docker-host=inherit

[andrejusc]

@robdimsdale - trying that now:

pack build buildpack-sample:1.0.0 --path=. --verbose --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static --trust-builder --docker-host=inherit

and getting such error:

===> DETECTING
ERROR: No buildpack groups passed detection.
ERROR: Please check that you are running against the correct path.
ERROR: failed to detect: no buildpacks participating
ERROR: failed to build: executing lifecycle: failed with status code: 20

[robdimsdale]

How were you specifying the buildpack in your command above? I just copied that command. I assumed you had specified the buildpack in some config file (e.g. project.toml).

If you aren't doing that, then you'll need to add the rust buildpack explicitly:

pack build buildpack-sample:1.0.0 \
  --buildpack=docker.io/paketocommunity/rust \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  <...>
``

[andrejusc]

@robdimsdale - ok, probably I've messed something. Retrying...

So again with jammy base builder initially:

pack build buildpack-sample:1.0.1 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-base \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust

and launching container out of produced image - works.

Now doing exactly same, but with jammy static (just incrementing image tag to avoid cache reuse):

pack build buildpack-sample:1.0.2 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust

and launching container ouf of produced image - leads to such error:

ERROR: failed to launch: bash exec: no such file or directory

Also worth mentioning - at least at the moment I could see both tiny and static builder images of same sizes - I;d imagine static should be smaller:

$ podman image ls | grep paketo | grep buildpackless
docker.io/paketobuildpacks/builder-jammy-buildpackless-tiny        latest         cd57f6bc96a5  43 years ago  419 MB
docker.io/paketobuildpacks/builder-jammy-buildpackless-static      latest         f986f14083df  43 years ago  419 MB

[robdimsdale]

Ok, I'm confused now. I thought we were trying to get a statically-linked rust binary working via the static builder, so I'm confused as to why we expect bash to exist on the running container.

I'm not familiar enough with the rust buildpack to know what start commands it creates for the app container, but if you got an app working with tiny you should use the same app and same commands for static. The tiny image doesn't have a shell, so you must have done something different to get an app working on that stack/builder. I couldn't see which commands you were using above for that scenario, but whatever you were doing, switch out the tiny builder for the static builder.

The reason why this app works on base and not static is that this app is relying on bash to exist at runtime, which is true for base (and full) and not true for tiny or static.

As far as the image size of the builder, I haven't explored that. I will note that both the tiny and the static images have the effectively the same build image, which is much larger than the run image in both builders. It's surprises me that the images are identical to the MB, but i wouldn't necessarily expect the builder to be significantly smaller.

[robdimsdale]

For reference, the packages on the run-time image for static are here:

    ca-certificates \
    tzdata \

And for tiny are here:

    base-files \
    ca-certificates \
    libc6 \
    libssl3 \
    netbase \
    openssl \
    tzdata \
    zlib1g \

[andrejusc]

@robdimsdale - if such output could be of any help to understand issue with static image.

For base builder case:

pack inspect buildpack-sample:1.0.1
Inspecting image: buildpack-sample:1.0.1

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.jammy

Base Image:
  Reference: a2a7e8cfbf95df0019e581ef6f076274cc0b900f1644c318b63a45dbc341d17b
  Top Layer: sha256:b96de72e9823de1f35c5f4e172702a9258796c7b98f6745723d95eef6517e74c

Run Images:
  index.docker.io/paketobuildpacks/run-jammy-base:latest

Buildpacks:
  ID                                VERSION        HOMEPAGE
  paketo-community/rustup           1.8.1          https://github.com/paketo-community/rustup
  paketo-community/rust-dist        1.16.1         https://github.com/paketo-community/rust-dist
  paketo-buildpacks/syft            1.26.0         https://github.com/paketo-buildpacks/syft
  paketo-community/cargo            0.9.0          https://github.com/paketo-community/cargo
  paketo-buildpacks/procfile        5.6.0          https://github.com/paketo-buildpacks/procfile

Processes:
  TYPE                 SHELL        COMMAND                        ARGS                                 WORK DIR
  web (default)        bash         /workspace/bin/function                                             /workspace
  function                          tini                           -g -- /workspace/bin/function        /workspace

And for static:

pack inspect buildpack-sample:1.0.2
Inspecting image: buildpack-sample:1.0.2

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.jammy.static

Base Image:
  Reference: c53b8f93b93c1de4173803f753bf2dfb18d8daecd9a692696103c53da60c3bd4
  Top Layer: sha256:6153dd9ac2c7b8b0df5f2e7e365d54e2f8abe4350f1c56f69e70f3ba26565e2b

Run Images:
  index.docker.io/paketobuildpacks/run-jammy-static:latest

Buildpacks:
  ID                                VERSION        HOMEPAGE
  paketo-community/rustup           1.8.1          https://github.com/paketo-community/rustup
  paketo-community/rust-dist        1.16.1         https://github.com/paketo-community/rust-dist
  paketo-buildpacks/syft            1.26.0         https://github.com/paketo-buildpacks/syft
  paketo-community/cargo            0.9.0          https://github.com/paketo-community/cargo
  paketo-buildpacks/procfile        5.6.0          https://github.com/paketo-buildpacks/procfile

Processes:
  TYPE                 SHELL        COMMAND                        ARGS                                 WORK DIR
  web (default)        bash         /workspace/bin/function                                             /workspace
  function                          tini                           -g -- /workspace/bin/function        /workspace

My Procfile is such:

web: /workspace/bin/function

and I don't yet fully understand which part dictates to use shell as bash and not have executable as itself.

[dmikusa]

Oh, that's a Profile thing. The stack is different so it's not handling that right. Filed an enhancement request, https://github.com/paketo-buildpacks/procfile/issues/175.

If the application's stack is not io.paketo.stacks.tiny the contents of Procfile will be executed as a shell script.

https://github.com/paketo-buildpacks/procfile#behavior

You might not need a Procfile. Have you tried w/out? The cargo buildpack can inspect, it looks at your project targets, and usually set up process types automatically.

[andrejusc]

@dmikusa -ok, removed Procfile if that is right now not suitable for static use case. Now inspect output is different and without bash:

pack inspect buildpack-sample:1.0.3
Inspecting image: buildpack-sample:1.0.3

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.jammy.static

Base Image:
  Reference: c53b8f93b93c1de4173803f753bf2dfb18d8daecd9a692696103c53da60c3bd4
  Top Layer: sha256:6153dd9ac2c7b8b0df5f2e7e365d54e2f8abe4350f1c56f69e70f3ba26565e2b

Run Images:
  index.docker.io/paketobuildpacks/run-jammy-static:latest

Buildpacks:
  ID                                VERSION        HOMEPAGE
  paketo-community/rustup           1.8.1          https://github.com/paketo-community/rustup
  paketo-community/rust-dist        1.16.1         https://github.com/paketo-community/rust-dist
  paketo-buildpacks/syft            1.26.0         https://github.com/paketo-buildpacks/syft
  paketo-community/cargo            0.9.0          https://github.com/paketo-community/cargo

Processes:
  TYPE                      SHELL        COMMAND        ARGS                                 WORK DIR
  function (default)                     tini           -g -- /workspace/bin/function        /workspace

But trying to launch container - getting now different error:

ERROR: failed to launch: direct exec: no such file or directory

Anything else I'm dong not right/not expected here?

[robdimsdale]

Do you have a sample app we can look at? I could make some guesses but it would be faster if we could iterate on it locally.

Regardless, I'm not sure what is failing to be executed. Unless it's exec itself, which might not be on the static stack. Do you get the same error if you run the following command?

docker run -it --rm --entrypoint=launcher buildpack-sample:1.0.3 tini -g -- /workspace/bin/function

This bypasses the default process list and invokes tini -g -- /workspace/bin/function directly. Whether you get the same error or not could help narrow down the list of possibilities.

[andrejusc]

@robdimsdale - so far no luck, tried proposed by you command:

podman run -it --rm --entrypoint=launcher buildpack-sample:1.0.5 tini -g -- /workspace/bin/function
ERROR: failed to launch: bash exec: no such file or directory

Attached is my use case. Unzip it into buildpack-sample folder (ignore func.yaml file inside for now, hopefully it's not affecting regular pack command), navigate into that folder and invoke (use some non-existent image tag to avoid buildpack cache reuse issue):

pack build buildpack-sample:1.0.0 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust

buildpack-sample.zip

[dmikusa]

When I build with the command you've given:

pack build buildpack-sample:1.0.0 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust

I get this error:

> docker run -it --platform='linux/amd64' c233bd2dac1c
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

Are you sure that you're building a static binary?

When I try to build the sample app with a static binary, the build fails.

Command:

pb rust-sample -b paketo-community/rust --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static -e BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-musl'

Output:

    cargo install --target=x86_64-unknown-linux-musl --color=never --root=/layers/paketo-community_cargo/Cargo --path=.
        Installing function v0.1.0 (/workspace)
          Updating crates.io index
       Downloading crates ...
        Downloaded alloc-stdlib v0.2.2
        Downloaded ppv-lite86 v0.2.17
        Downloaded num_cpus v1.15.0
        Downloaded bytestring v1.3.0
...
         Compiling env_logger v0.10.0
      error: failed to run custom build command for `zstd-sys v2.0.8+zstd.1.5.5`

      Caused by:
        process didn't exit successfully: `/workspace/target/release/build/zstd-sys-85b90089b2485305/build-script-build` (exit status: 1)
        --- stdout
        cargo:rerun-if-env-changed=ZSTD_SYS_USE_PKG_CONFIG
        TARGET = Some("x86_64-unknown-linux-musl")
        OPT_LEVEL = Some("3")
        HOST = Some("x86_64-unknown-linux-gnu")
        cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
        CC_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
        CC_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CC
        TARGET_CC = None
        cargo:rerun-if-env-changed=CC
        CC = None
        RUSTC_LINKER = None
        cargo:rerun-if-env-changed=CROSS_COMPILE
        CROSS_COMPILE = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
        CFLAGS_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
        CFLAGS_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CFLAGS
        TARGET_CFLAGS = None
        cargo:rerun-if-env-changed=CFLAGS
        CFLAGS = None
        cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
        CRATE_CC_NO_DEFAULTS = None
        DEBUG = Some("false")
        CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
        cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
        CC_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
        CC_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CC
        TARGET_CC = None
        cargo:rerun-if-env-changed=CC
        CC = None
        RUSTC_LINKER = None
        cargo:rerun-if-env-changed=CROSS_COMPILE
        CROSS_COMPILE = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
        CFLAGS_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
        CFLAGS_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CFLAGS
        TARGET_CFLAGS = None
        cargo:rerun-if-env-changed=CFLAGS
        CFLAGS = None
        cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
        CRATE_CC_NO_DEFAULTS = None
        CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
        cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
        CC_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
        CC_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CC
        TARGET_CC = None
        cargo:rerun-if-env-changed=CC
        CC = None
        RUSTC_LINKER = None
        cargo:rerun-if-env-changed=CROSS_COMPILE
        CROSS_COMPILE = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
        CFLAGS_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
        CFLAGS_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CFLAGS
        TARGET_CFLAGS = None
        cargo:rerun-if-env-changed=CFLAGS
        CFLAGS = None
        cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
        CRATE_CC_NO_DEFAULTS = None
        CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
        cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-musl
        CC_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_musl
        CC_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CC
        TARGET_CC = None
        cargo:rerun-if-env-changed=CC
        CC = None
        RUSTC_LINKER = None
        cargo:rerun-if-env-changed=CROSS_COMPILE
        CROSS_COMPILE = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-musl
        CFLAGS_x86_64-unknown-linux-musl = None
        cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_musl
        CFLAGS_x86_64_unknown_linux_musl = None
        cargo:rerun-if-env-changed=TARGET_CFLAGS
        TARGET_CFLAGS = None
        cargo:rerun-if-env-changed=CFLAGS
        CFLAGS = None
        cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
        CRATE_CC_NO_DEFAULTS = None
        CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
        running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-o" "/workspace/target/x86_64-unknown-linux-musl/release/build/zstd-sys-1286060b92b95d15/out/zstd/lib/common/debug.o" "-c" "zstd/lib/common/debug.c"
        exit status: 127
        running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-o" "/workspace/target/x86_64-unknown-linux-musl/release/build/zstd-sys-1286060b92b95d15/out/zstd/lib/common/entropy_common.o" "-c" "zstd/lib/common/entropy_common.c"
        exit status: 127

        --- stderr

        error occurred: Command "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-o" "/workspace/target/x86_64-unknown-linux-musl/release/build/zstd-sys-1286060b92b95d15/out/zstd/lib/common/debug.o" "-c" "zstd/lib/common/debug.c" with args "musl-gcc" did not execute successfully (status code exit status: 127).

      warning: build failed, waiting for other jobs to finish...
      <jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
      <jemalloc>: (This is the expected behaviour if you are running under QEMU)
      error: failed to compile `function v0.1.0 (/workspace)`, intermediate artifacts can be found at `/workspace/target`
unable to invoke layer creator
unable to contribute application layer
unable to install single
unable to build
exit status 101

Not sure if that's something with my local setup, Mac M1 creates some weird build issues occasionally. Anyway, you're going to need to make sure you manually set the target for the time being. In the future the buildpack can see you're running on the static stack and do that automatically.

[robdimsdale]

When I try to run the same command:

pack build rust-static-sample -b paketo-community/rust --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static -e BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-musl'

On a linux machine, I get the following output:

[builder] Rust Cargo Build Pack 0.9.0
[builder]   https://github.com/paketo-community/cargo
[builder]   Build Configuration:
[builder]     $BP_CARGO_INSTALL_ARGS        --target=x86_64-unknown-linux-musl    additional arguments to pass to Cargo install
[builder]     $BP_CARGO_INSTALL_TOOLS                                             additional tools to be add with Cargo install
[builder]     $BP_CARGO_INSTALL_TOOLS_ARGS                                        additional arguments to pass to Cargo install for tools
[builder]     $BP_CARGO_TINI_DISABLED       false                                 Skip installing tini
[builder]     $BP_CARGO_WORKSPACE_MEMBERS                                         the subset of workspace members for Cargo to install
[builder]     $BP_DISABLE_SBOM              false                                 Skip running SBOM scan
[builder]     $BP_EXCLUDE_FILES                                                   colon separated list of glob patterns, matched source files are removed
[builder]     $BP_INCLUDE_FILES             static/*:templates/*:public/*:html/*  colon separated list of glob patterns, matched source files are included
[builder]   Tini 0.19.0: Contributing to layer
[builder]     Downloading from https://github.com/krallin/tini/releases/download/v0.19.0/tini-amd64
[builder]     Verifying checksum
[builder]     Copying to /layers/paketo-community_cargo/tini
[builder]     Creating cached target directory /workspace/target
[builder]   Rust Application: Contributing to layer
[builder]     File modification times not restored
[builder]     File modification times not restored
[builder]     File modification times not restored
[builder]     cargo install --target=x86_64-unknown-linux-musl --color=never --root=/layers/paketo-community_cargo/Cargo --path=.
[builder]         Installing function v0.1.0 (/workspace)

<...>
[builder]       error[E0463]: can't find crate for `core`
[builder]         |
[builder]         = note: the `x86_64-unknown-linux-musl` target may not be installed
[builder]         = help: consider downloading the target with `rustup target add x86_64-unknown-linux-musl`
[builder]
[builder]       error[E0463]: can't find crate for `compiler_builtins`
[builder]
[builder]       For more information about this error, try `rustc --explain E0463`.
[builder]       error: could not compile `cfg-if` due to 2 previous errors
[builder]       warning: build failed, waiting for other jobs to finish...
[builder]       error: failed to compile `function v0.1.0 (/workspace)`, intermediate artifacts can be found at `/workspace/target`

When I add a rust-toolchain.toml with the following contents:

[toolchain]
targets = ["x86_64-unknown-linux-musl"]

I get a different error:

[builder]         running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-o" "/workspace/target/x86_64-unknown-linux-musl/release/build/zstd-sys-1286060b92b95d15/out/zstd/lib/common/entropy_common.o" "-c" "zstd/lib/common/entropy_common.c"
[builder]
[builder]         --- stderr
[builder]
[builder]
[builder]         error occurred: Failed to find tool. Is `musl-gcc` installed?
[builder]
[builder]
[builder]       warning: build failed, waiting for other jobs to finish...
[builder]       error: failed to compile `function v0.1.0 (/workspace)`, intermediate artifacts can be found at `/workspace/target`

@dmikusa is musl-gcc something you'd expect the rust buildpacks to provide, or should it be on the stack?

Also, why is the buildpack trying to use musl instead of libc? The static build image has libc on it:

→ docker run -it --rm paketobuildpacks/build-jammy-static /bin/bash -c "dpkg -l | grep libc"
ii  libc-bin                  2.35-0ubuntu3.1                         amd64        GNU C Library: Binaries
ii  libc-dev-bin              2.35-0ubuntu3.1                         amd64        GNU C Library: Development binaries
ii  libc6:amd64               2.35-0ubuntu3.1                         amd64        GNU C Library: Shared libraries
ii  libc6-dev:amd64           2.35-0ubuntu3.1                         amd64        GNU C Library: Development Libraries and Header Files
ii  libcap-ng0:amd64          0.7.9-2.2build3                         amd64        An alternate POSIX capabilities library
ii  libcap2:amd64             1:2.44-1build3                          amd64        POSIX 1003.1e capabilities (library)
ii  libcc1-0:amd64            12.1.0-2ubuntu1~22.04                   amd64        GCC cc1 plugin for GDB
ii  libcom-err2:amd64         1.46.5-2ubuntu1.1                       amd64        common error description library
ii  libcrypt-dev:amd64        1:4.4.27-1                              amd64        libcrypt development files
ii  libcrypt1:amd64           1:4.4.27-1                              amd64        libcrypt shared library
ii  libctf-nobfd0:amd64       2.38-4ubuntu2.1                         amd64        Compact C Type Format library (runtime, no BFD dependency)
ii  libctf0:amd64             2.38-4ubuntu2.1                         amd64        Compact C Type Format library (runtime, BFD dependency)
ii  libcurl3-gnutls:amd64     7.81.0-1ubuntu1.10                      amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  libcurl4:amd64            7.81.0-1ubuntu1.10                      amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  linux-libc-dev:amd64      5.15.0-69.76                            amd64        Linux Kernel Headers for development

And no packages with musl:

→ docker run -it --rm paketobuildpacks/build-jammy-static /bin/bash -c "dpkg -l | grep musl"

Also, I get the exact same error (musl-gcc not found) when using the tiny builder instead of the static builder (and using the same pack build command above).

[andrejusc]

@dmikusa - if it could help by some means. On my Linux x86_64 box doing static Rust building directly without any buildpacks involved:

$ RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu

and then checking output executable to make sure it's static:

$ file target/x86_64-unknown-linux-gnu/release/function
target/x86_64-unknown-linux-gnu/release/function: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=3c2a78e971678e6160c927342bc2d20e35a4083d, for GNU/Linux 3.2.0, with debug_info, not stripped

and such:

$ ldd target/x86_64-unknown-linux-gnu/release/function
    statically linked
$ ls -l target/x86_64-unknown-linux-gnu/release/function
-rwxrwxr-x 2 user user 13854080 Apr  6 15:37 target/x86_64-unknown-linux-gnu/release/function

So my expectation would be that using static builder should produce similar outcome.

I'm not yet very deep into QEMU emulator usage on ARM64 for x86_64 purposes. Did in opposite way for one recent project, but still a lot to learn.

[dmikusa]

Oh, cool. I learned something new today. I've only done static builds previously with musl.

@andrejusc Can you try these arguments?

-b paketo-community/rust --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static -e RUSTFLAGS='-C target-feature=+crt-static' -e BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-gnu' -e BP_CARGO_TINI_DISABLED=true

That will pass through RUSTFLAGS and it built a static binary and it was OK for me. It will also disable tini. I think the tini binary we are installing needs libc. I think that is causing the issue.

I'll log an issue to check if we can get a tini binary that's static. Disabling it is OK too, but you may need to setup some signal handlers in your app to ensure it shutdown properly (it needs to handle SIGTERM).

[dmikusa]

@robdimsdale Ohhhh, yes. That makes sense. Rust doesn't necessarily need musl-gcc but there's a particular package in this project that is trying to build a native c library wrapper, so it must need it for that. The buildpack isn't providing it at the moment so it would fail for that reason. I have some ideas on how to fix that in the future, but that's further down the road kind of bundled with cross-compile support.

[andrejusc]

@dmikusa - cool, appreciate all our collaboration here.

So here is latest. Using such command in my case:

pack build buildpack-sample:1.0.8 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust \
  -e RUSTFLAGS='-C target-feature=+crt-static' \
  -e BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-gnu' \
  -e BP_CARGO_TINI_DISABLED=true

and analyzing then built image:

pack inspect buildpack-sample:1.0.8
Inspecting image: buildpack-sample:1.0.8

REMOTE:
(not present)

LOCAL:

Stack: io.buildpacks.stacks.jammy.static

Base Image:
  Reference: c53b8f93b93c1de4173803f753bf2dfb18d8daecd9a692696103c53da60c3bd4
  Top Layer: sha256:6153dd9ac2c7b8b0df5f2e7e365d54e2f8abe4350f1c56f69e70f3ba26565e2b

Run Images:
  index.docker.io/paketobuildpacks/run-jammy-static:latest

Buildpacks:
  ID                                VERSION        HOMEPAGE
  paketo-community/rustup           1.8.1          https://github.com/paketo-community/rustup
  paketo-community/rust-dist        1.16.1         https://github.com/paketo-community/rust-dist
  paketo-buildpacks/syft            1.26.0         https://github.com/paketo-buildpacks/syft
  paketo-community/cargo            0.9.0          https://github.com/paketo-community/cargo

Processes:
  TYPE                      SHELL        COMMAND                        ARGS        WORK DIR
  function (default)                     /workspace/bin/function                    /workspace

and size of it (which is now good enough):

$ podman image ls | grep sample | grep 1.0.8
docker.io/library/buildpack-sample                                 1.0.8          211189eee732  43 years ago  19.7 MB

Could confirm that I could launch container out of it and it behaves as expected.

I'm happy then to conclude all this long thread. Hope you've enjoyed it as well. Thank you for all your prompt help and comments @dmikusa @robdimsdale :thumbsup:

[dmikusa]

Woohoo! Glad it's working. Thanks for patiently working through this with us and all the feedback!

[robdimsdale]

Great, glad it's working. I was able to run the same command and build the app:

pack build buildpack-sample:1.0.8 \
  --path=. --verbose \
  --builder=docker.io/paketobuildpacks/builder-jammy-buildpackless-static \
  --trust-builder --docker-host=inherit --buildpack=docker.io/paketocommunity/rust \
  -e RUSTFLAGS='-C target-feature=+crt-static' \
  -e BP_CARGO_INSTALL_ARGS='--target=x86_64-unknown-linux-gnu' \
  -e BP_CARGO_TINI_DISABLED=true

I didn't validate that the binary is statically-linked, but given the stack doesn't have libc, I'm fairly confident that it must be statically-linked.

In terms of the size of the image, the breakdown is roughly:

• 14MB for the app
• 2.3MB for the launcher (part of the CNB spec)
• 2.2MB for the base image
  • Of this, most of it (1.6MB) is time-zone data

The app is essentially the same size as when you pre-compiled and dropped it onto a scratch image, so that checks out. In order to be CNB-compliant you will need the launcher, so that's just overhead you can't avoid. So in reality there's only about 2MB more you can gain by dropping the timezone packages.

As an aside, we could make a scratch stack which contains absolutely no packages at runtime, but I do not yet see value in maintaining this. Longer-term the Buildpacks spec will allow us to use "run-time extensions" to dynamically install packages during the build process, and at that point it could be helpful to have a scratch as it would allow you to install exactly (and only) the minimal set of packages your particular app needs.

Regardless, I'm glad it's working. I'm glad we validated the static stack/builder, and I'm glad you are able to get an image size that you're satisfied with. I also learned some interesting stuff about static linking, which is a bonus!

[andrejusc]

Hi folks, in case you're interested - I've published article in Medium where outcome of our discussion here (i.e. smaller image preparation for Rust use case) is used in bigger solution development picture: https://medium.com/@andrejusc/rust-code-as-serverless-function-and-evaluation-via-openshift-dev-sandbox-a9019d17ca47 . Hopefully that will further allow to evolve things. Thank you once again.

[dmikusa]

@andrejusc FYI

Note: image creation date here of 43y is due to made on purpose decision by buildpacks to count from Jan 1st 1980. Discussed here.

You can set --creation-time=now as an argument to pack build and it will use the current time at build for the creation date, if you prefer usable creation dates. I like that when I'm doing work locally.