search

paketo-buildpacks / libpak

An opinionated extension to the libcnb Cloud Native Buildpack Library
Apache License 2.0
15 stars 17 forks source link

Image build without previous-image setting fails with interface conversion error #269

Open mecseid opened 1 year ago

mecseid commented 1 year ago

After PR https://github.com/paketo-buildpacks/libpak/pull/261 it looks like that image creation without any previous image fails with an error.

Expected Behavior

The image can be built without any previous image.

Current Behavior

The following error occurs during image build:

panic: interface conversion: interface {} is time.Time, not string

goroutine 1 [running]:
github.com/paketo-buildpacks/libpak.(*LayerContributor).Equals(0xc00011cc00?, 0x2c5?, 0x300?)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.66.1/layer.go:142 +0x4f7
github.com/paketo-buildpacks/libpak.(*LayerContributor).checkIfMetadataMatches(0xc000233fd8, {{0x0, 0x0, 0x1}, 0xc00027a120, {0xc0000b5c68, 0x3}, {0xc00020e960, 0x2f}, 0xc000213800, ...})
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.66.1/layer.go:122 +0x2ab
github.com/paketo-buildpacks/libpak.(*LayerContributor).Contribute(0xc000233fd8, {{0x0, 0x0, 0x1}, 0xc00027a120, {0xc0000b5c68, 0x3}, {0xc00020e960, 0x2f}, 0xc000213800, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.66.1/layer.go:75 +0x145
github.com/paketo-buildpacks/libpak.(*DependencyLayerContributor).Contribute(0xc000234280, {{0x0, 0x0, 0x1}, 0xc00027a120, {0xc0000b5c68, 0x3}, {0xc00020e960, 0x2f}, 0xc000213800, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.66.1/layer.go:254 +0x2a8
github.com/paketo-buildpacks/libjvm.JRE.Contribute({{0xc0000b5700, 0xa}, {{0x7c0e92, 0x22}, {0x0, 0x0, 0x0}, {0x8467a0, 0xc000154f80}}, 0x1, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libjvm@v1.43.2/jre.go:76 +0x129
github.com/buildpacks/libcnb.Build({0x8472a0, 0xc0001427e0}, {0xc0000e57b8, 0x4, 0x847320?})
    /home/runner/go/pkg/mod/github.com/buildpacks/libcnb@v1.28.0/build.go:280 +0x1d47
github.com/buildpacks/libcnb.Main({0x8472c0, 0xc0001427d0}, {0x8472a0, 0xc0001427e0}, {0xc0000e57b8, 0x4, 0x4})
    /home/runner/go/pkg/mod/github.com/buildpacks/libcnb@v1.28.0/main.go:47 +0x2bd
github.com/paketo-buildpacks/libpak.Main({0x847260?, 0xa7b2a8?}, {0x847240?, 0xc0001ae500?}, {0x0, 0x0, 0x8467a0?})
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.66.1/main.go:28 +0x3ed
main.main()
    /home/runner/work/bellsoft-liberica/bellsoft-liberica/cmd/main/main.go:35 +0x20e
ERROR: failed to build: exit status 2

Possible Solution

Rollback the base image maximum 0.3.279-base

Steps to Reproduce

Create an image with the following command inside a buildpacks/base based container:

/cnb/lifecycle/creator \
  -no-color \
  -skip-restore \
  -app=/workspace \
  -process-type=web \
  -run-image=${RUN_IMAGE} \
  ${APP_IMAGE}:${TAG_NAME}
dmikusa commented 1 year ago

I can see why you're hitting this in the code, but trying to understand your circumstances/use case a little more. Can you reproduce this with pack? Does it work if you remove -skip-restore? Why do you have -skip-restore? I suspect that's going to prevent a lot of the caching that the buildpack can do, which is not ideal.

mecseid commented 1 year ago

Hey @dmikusa!

I tried, but can't reproduce with pack. pack build test_image --builder paketobuildpacks/builder:base --run-image <custom_run_image> --path .

For -skip-restore, some ops guy wants to build the application from scratch if it is triggered from a git tag (release or release candidate build). Regardless, it doesn't work without it either.

Our CI process runs on a Kubernetes cluster as a Jenkins Pipeline, and we run the lifecycle commands inside a paketobuildpacks/builder:base image-based container.

dmikusa commented 1 year ago

OK, thanks for the context. Not sure what would be different about running the creator directly vs pack in this case. Good to know though. Thanks

mecseid commented 1 year ago

Good question. We built our pipeline based on the "official" Tekton pipeline (that uses the creator too), see: https://github.com/tektoncd/catalog/blob/main/task/buildpacks/0.6/buildpacks.yaml

mecseid commented 1 year ago

@dmikusa Do you have any idea how can we bypass this issue with the direct call of the creator?

dmikusa commented 1 year ago

I was thinking it might be the -skip-restore flag, but you tried and that didn't seem to help.

Aside from that, it is likely that a prerequisite of the platform API spec isn't being met by your setup. When you run the creator directly, you're responsible for implementing the platform API spec. I'm basing that on the fact that pack, a known good implementation of the platform spec is working OK.

What I would suggest is turning on debug logging.

    l.Logger.Debugf("Expected metadata: %+v", expected)
    l.Logger.Debugf("Actual metadata: %+v", layer.Metadata)

These two lines run right before the problem and print the locations of the metadata files being compared. The second one is the problematic one. Locate that file, see if it exists and see what's in it. The code is iterating over the contents of the dependencies in that file. It looks like it gets something, but then fails to read part of one of the dependency objects. If we can get the contents of the source, it might be clear what other workarounds are available.

To be clear, we do intend to fix this and make the code more robust. It is just a lower priority because it works with pack/Spring Boot build tools. Also, this is just a slow time of year with lots of people being OOO so there's not as many hands available to help fix things.

jpastoor commented 8 months ago

Please let me know if you want me to create a separate ticket, but I do encounter a very similar problem. The deprecationDate is already a time.Time{} and not a string. I have not exactly been able to reproduce it consistently.

The difference between a successful build and a failed one seems to be with how the deprecationDate is encoded. Sometimes (don't exactly understand in which case) it is encoded as a UTC date and it fails, but if it is encoded in the standard RFC3339 then it succeeds.

Example of a failure (deprecation_date:0001-01-01 00:00:00 +0000)

Result: {BOM: &{Entries:[{Name:jre Metadata:map[layer:jre licenses:[{Type:GPL-2.0 WITH Classpath-exception-2.0 URI:https://openjdk.java.net/legal/gplv2+ce.html}] name:Adoptium JRE sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21] Launch:true Build:false} {Name:helper Metadata:map[layer:helper names:[active-processor-count java-opts jvm-heap link-local-dns memory-calculator security-providers-configurer jmx jfr openssl-certificate-loader security-providers-classpath-9 debug-9 nmt] version:11.2.3] Launch:true Build:false}]}, Labels:[] Layers:[JRE HelperLayerContributor JavaSecurityProperties] PersistentMetadata:map[] Processes:map[] Slices:[], Unmet:[]}
Check If Layer Restored -> tomlExists: %!s(bool=true), layerDirExists: %!s(bool=false), dirContents: [], cache: %!s(bool=false), build: %!s(bool=false)
Expected metadata: map[cert-dir:[map[mode:Lrwxrwxrwx path:/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/d2a6a44e.0 sha256:d464378fbb8b981d2b28a1deafffd0113554e6adfb34535134f411bf3c689e73]] cert-file:82f3a63130e0b47915659d619f177c8157c2bbb29dbf0f58ad2721ff84fb1190 dependency:map[cpes:[cpe:2.3:a:oracle:jre:11.0.21:*:*:*:*:*:*:*] deprecation_date:0001-01-01 00:00:00 +0000 UTC id:jre licenses:[map[type:GPL-2.0 WITH Classpath-exception-2.0 uri:https://openjdk.java.net/legal/gplv2+ce.html]] name:Adoptium JRE purl:pkg:generic/adoptium-jre@11.0.21?arch=amd64 sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21]]
Actual metadata: map[cert-dir:[map[mode:Lrwxrwxrwx path:/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/d2a6a44e.0 sha256:d464378fbb8b981d2b28a1deafffd0113554e6adfb34535134f411bf3c689e73]] cert-file:82f3a63130e0b47915659d619f177c8157c2bbb29dbf0f58ad2721ff84fb1190 dependency:map[cpes:[cpe:2.3:a:oracle:jre:11.0.21:*:*:*:*:*:*:*] deprecation_date:0001-01-01 00:00:00 +0000 UTC id:jre licenses:[map[type:GPL-2.0 WITH Classpath-exception-2.0 uri:https://openjdk.java.net/legal/gplv2+ce.html]] name:Adoptium JRE purl:pkg:generic/adoptium-jre@11.0.21?arch=amd64 sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21]]
panic: interface conversion: interface {} is time.Time, not string

goroutine 1 [running]:
github.com/paketo-buildpacks/libpak.(*LayerContributor).Equals(0xc0000bd938?, 0x1?, 0x1?)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.68.0/layer.go:142 +0x4f7
github.com/paketo-buildpacks/libpak.(*LayerContributor).checkIfMetadataMatches(0xc0000be070, {{0x0, 0x0, 0x1}, 0xc000142c60, {0xc000124668, 0x3}, {0xc000324120, 0x26}, 0xc000142150, ...})
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.68.0/layer.go:122 +0x2ab
github.com/paketo-buildpacks/libpak.(*LayerContributor).Contribute(0xc0000be070, {{0x0, 0x0, 0x1}, 0xc000142c60, {0xc000124668, 0x3}, {0xc000324120, 0x26}, 0xc000142150, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.68.0/layer.go:75 +0x145
github.com/paketo-buildpacks/libpak.(*DependencyLayerContributor).Contribute(0xc0000be318, {{0x0, 0x0, 0x1}, 0xc000142c60, {0xc000124668, 0x3}, {0xc000324120, 0x26}, 0xc000142150, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.68.0/layer.go:254 +0x2a8
github.com/paketo-buildpacks/libjvm.JRE.Contribute({{0xc000125750, 0xa}, {{0xc00002a00e, 0x22}, {0xc0001c4720, 0x1, 0x1}, {0x85dea0, 0xc0001d8f80}}, 0x1, ...}, ...)
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libjvm@v1.44.0/jre.go:74 +0x129
github.com/buildpacks/libcnb.Build({0x85e9a0, 0xc0001c4800}, {0xc000167850, 0x4, 0x85ea20?})
    /home/runner/go/pkg/mod/github.com/buildpacks/libcnb@v1.30.1/build.go:280 +0x1d47
github.com/buildpacks/libcnb.Main({0x85e9c0, 0xc0001c47f0}, {0x85e9a0, 0xc0001c4800}, {0xc000167850, 0x4, 0x4})
    /home/runner/go/pkg/mod/github.com/buildpacks/libcnb@v1.30.1/main.go:47 +0x2bd
github.com/paketo-buildpacks/libpak.Main({0x85e960?, 0xa9e0e8?}, {0x85e940?, 0xc000228a00?}, {0x0, 0x0, 0x85dea0?})
    /home/runner/go/pkg/mod/github.com/paketo-buildpacks/libpak@v1.68.0/main.go:28 +0x3ed
main.main()
    /home/runner/work/adoptium/adoptium/cmd/main/main.go:35 +0xec

Example of a success: (deprecation_date:0001-01-01T00:00:00Z)

Result: {BOM: &{Entries:[{Name:jre Metadata:map[layer:jre licenses:[{Type:GPL-2.0 WITH Classpath-exception-2.0 URI:https://openjdk.java.net/legal/gplv2+ce.html}] name:Adoptium JRE sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21] Launch:true Build:false} {Name:helper Metadata:map[layer:helper names:[active-processor-count java-opts jvm-heap link-local-dns memory-calculator security-providers-configurer jmx jfr openssl-certificate-loader security-providers-classpath-9 debug-9 nmt] version:11.2.3] Launch:true Build:false}]}, Labels:[] Layers:[JRE HelperLayerContributor JavaSecurityProperties] PersistentMetadata:map[] Processes:map[] Slices:[], Unmet:[]}
Check If Layer Restored -> tomlExists: %!s(bool=true), layerDirExists: %!s(bool=false), dirContents: [], cache: %!s(bool=false), build: %!s(bool=false)
Expected metadata: map[cert-dir:[map[mode:Lrwxrwxrwx path:/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/d2a6a44e.0 sha256:d464378fbb8b981d2b28a1deafffd0113554e6adfb34535134f411bf3c689e73]] cert-file:82f3a63130e0b47915659d619f177c8157c2bbb29dbf0f58ad2721ff84fb1190 dependency:map[cpes:[cpe:2.3:a:oracle:jre:11.0.21:*:*:*:*:*:*:*] deprecation_date:0001-01-01 00:00:00 +0000 UTC id:jre licenses:[map[type:GPL-2.0 WITH Classpath-exception-2.0 uri:https://openjdk.java.net/legal/gplv2+ce.html]] name:Adoptium JRE purl:pkg:generic/adoptium-jre@11.0.21?arch=amd64 sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21]]
Actual metadata: map[cert-dir:[map[mode:Lrwxrwxrwx path:/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates/d2a6a44e.0 sha256:d464378fbb8b981d2b28a1deafffd0113554e6adfb34535134f411bf3c689e73]] cert-file:82f3a63130e0b47915659d619f177c8157c2bbb29dbf0f58ad2721ff84fb1190 dependency:map[cpes:[cpe:2.3:a:oracle:jre:11.0.21:*:*:*:*:*:*:*] deprecation_date:0001-01-01T00:00:00Z id:jre licenses:[map[type:GPL-2.0 WITH Classpath-exception-2.0 uri:https://openjdk.java.net/legal/gplv2+ce.html]] name:Adoptium JRE purl:pkg:generic/adoptium-jre@11.0.21?arch=amd64 sha256:156861bb901ef18759e05f6f008595220c7d1318a46758531b957b0c950ef2c3 stacks:[*] uri:https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.21_9.tar.gz version:11.0.21]]
  Adoptium JRE 11.0.21: Reusing cached layer
Writing layer env.build: /layers/paketo-buildpacks_adoptium/jre/env.build <= map[]

Possible fix (workaround?) could be a type switch in LayerContributor::Equals():

    if dep, ok := layerM["dependency"].(map[string]interface{}); ok {
        for k, v := range dep {
            var deprecationDate time.Time
            var err error
            if k == "deprecation_date" {
                switch vDate := v.(type) {
                case time.Time:
                    deprecationDate = vDate
                case string:
                    deprecationDate, err = time.Parse(time.RFC3339, v.(string))
                    if err != nil {
                        return false, fmt.Errorf("unable to parse deprecation_date %s", v.(string))
                    }
                default:
                    return false, fmt.Errorf("unexpected type %T for parse deprecation_date %v", v, v)
                }
[...]

Would a MR like the above be acceptable? Any other things I can check to better isolate/reproduce?

jpastoor commented 8 months ago

In addition to the above, I think I better understand the root cause now.

We were using https://github.com/paketo-buildpacks/ca-certificates/releases/tag/v3.6.6 which relies on libpak v1.67.2. That version is more lenient in his parsing of layer metadata. In libpak v1.68 we essentially became more strict in what we accept as valid in the layer metadata due to this change: https://github.com/paketo-buildpacks/libpak/pull/261/files

After updating ca-certificates to v3.6.7 we haven't seen this issue anymore.

I'll create a MR however to prevent the panic in the Equal method when the metadata is not as expected. We are dealing with quite the ecosystem of various buildpack layers so that will make it a bit more robust. Do you agree @pivotal-david-osullivan ?

jpastoor commented 8 months ago

@dmikusa any thoughts? Greatly appreciated

dmikusa commented 7 months ago

@jpastoor FYI, we cut release 1.68.2 of libpak this week. That has your fix and another fix from @pivotal-david-osullivan which should also fix a caching bug. We've updated all of the buildpacks, as soon as we get them all released you'll be able to pick up that fix. That might happen with today's release, or worst case, next week's release cycle.

jpastoor commented 6 months ago

Thanks, appreciate the work!

Op vr 16 feb 2024 14:36 schreef Daniel Mikusa @.***>:

@jpastoor https://github.com/jpastoor FYI, we cut release 1.68.2 https://github.com/paketo-buildpacks/libpak/releases/tag/v1.68.2 of libpak this week. That has your fix and another fix from @pivotal-david-osullivan https://github.com/pivotal-david-osullivan which should also fix a caching bug. We've updated all of the buildpacks, as soon as we get them all released you'll be able to pick up that fix. That might happen with today's release, or worst case, next week's release cycle.

— Reply to this email directly, view it on GitHub https://github.com/paketo-buildpacks/libpak/issues/269#issuecomment-1948399175, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFTA6U33CI55WJVQH4GQM3YT5OHRAVCNFSM6AAAAAA2SVOMLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBYGM4TSMJXGU . You are receiving this because you were mentioned.Message ID: @.***>