Cannot release package with kctrl

mamachanko commented 2 years ago

What steps did you take:

using registry/project/image as a placeholder for my actual image

I have a directory containing the to-be contents of my package:

config
  |_ upstream # this is an imgpkg bundle, i.e. has upstream/.imgpkg
  |_ downstream # this is plain YTT

I ran

kctrl pkg init

Then I adjusted PackageBuild manually:

apiVersion: kctrl.carvel.dev/v1alpha1
kind: PackageBuild
metadata:
  name: my.package.com
spec:
  release:
  - resource: {}
  template:
    spec:
      export:
      - imgpkgBundle:
          image: registry/project/image
          useKbldImagesLock: false
        includePaths:
        - config/downstream
        - config/upstream
      app:
        spec:
          deploy:
          - {}
          template:
          - ytt:
              paths:
              - .
          - kbld: {}

Then I updated the resources in package-resources.yaml to reflect my needs:

Filled in metadata

Set Package

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
name: my.package.com.0.0.0
spec:
refName: my.package.com
template:
spec:
  deploy:
    - kapp:
        rawOptions:
          - --wait-timeout=5m
          - --kube-api-qps=20
          - --kube-api-burst=30
  fetch:
    - git: { }
  template:
    - ytt:
        paths:
          - config/upstream
          - config/downstream
    - kbld: { }
valuesSchema:
openAPIv3: null
version: 0.0.0

What happened: When I run

kctrl pkg release -v 0.0.0-$USER

it fails with

❯ kctrl pkg release -v 0.0.0-$USER --chdir manifests/build

Prerequisites
1. The host must be authorized to push images to a registry (can be set up by
running `docker login`)
2. Package can be released only after `kctrl package init` has been run
successfully.

The bundle created needs to be pushed to an OCI registry. Registry URL format:
<REGISTRY_URL/REPOSITORY_NAME> e.g. index.docker.io/k8slt/sample-bundle
> Enter the registry URL (regisrty/project/path):
W1019 09:12:46.033515   65993 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

Target cluster 'https://35.195.245.51' (nodes: gke-tinwork-whinny-default-pool-664656e5-3xmv, 4+)

kbld builds images when necessary and ensures that all image references are
resolved to an immutable reference
Building images and resolving references
            | $ ytt -f /var/folders/0r/_173x1jn2d1bp7h4twh90hc80000gp/T/kapp-controller-fetch-template-deploy1953851271/0
            | $ kbld -f - --imgpkg-lock-output=.imgpkg/images.yml

kctrl: Error: Reconciling: kbld:
  Error:
- Resolving image 'regisrty/project/image': GET https://regisrty/v2/project/image/manifests/latest: NOT_FOUND: repository project/image not found

Annoyingly, my package-build.yaml gets overwritten by kctrl every time I run the release command:

It reformats it
It reorders fields
It adds creationTimestamp: null
It removes useKbldImagesLock: false

If there are other (unrelated) YTT files in my repository, it fails as it tries to template them.

What did you expect:

That's hard to say. I think I would expect my package image to be pushed and Package resource to appear somewhere.

Anything else you would like to add:

I am switching back to scripted package releasing.

Environment:

❯ kctrl version
kctrl version 0.41.2

Succeeded

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

100mik commented 2 years ago

This is definitely not expected. We try to maintain user supplied values, gonna take a closer look a it.

100mik commented 2 years ago

Annoyingly, my package-build.yaml gets overwritten by kctrl every time I run the release command:

This seems to be true, and might be a bit buggy. This seems to be happening the command seems to reduce the config to the most minimal version (not intended at all) Even if, useKbldImagesLock is removed it defaults to false, so it should not affect functionality of the release command itself. This might be improved for a better user experience, we could not overwrite the file if there are no changes, or improve how we handle marshalling/unmarshalling.

I did however find some other buggy behaviour, the .imgpkg folder seems to be deleted after a run with useKbldImagesLock set to false. This definitely should not be the case.

100mik commented 2 years ago

The output you shared however seems to be from when useKbldImagesLock output is set to true as it is trying to build images, which confuses me a bit. Was this run after you set the value to false?

More so as of today, kctrl expects the user supplied .imgpkg directory to reside in the root, so something like

.
|-.imgpkg
|-upstream
|-downstream

Thought I would call that out 🤔

joostvdg commented 1 year ago

Using the latest version I could find (0.44.1) this still happens:

❯ kctrl version

kctrl version 0.44.1

Succeeded

Created the initial package via kctrl package init.

If I set useKbldImagesLock to true, it works as expected (image lock file generated, package released). Setting it to false gives unexpected behavior:

the release fails, This directory is not a bundle. It is missing .imgpkg/images.yml
the useKbldImagesLock: false line is removed from the package-build.yml

ps. the reason I want to disable the lock file, is that I have a multi-arch image, and the lock file is only for a single arch, which defeats the purpose of multi-arch 🤷

joostvdg commented 1 year ago

The add more information to my previous comment.

Here's the package-build.yml:

apiVersion: kctrl.carvel.dev/v1alpha1
kind: PackageBuild
metadata:
  creationTimestamp: null
  name: client.gitstafette.kearos.net
spec:
  release:
  - resource: {}
  template:
    spec:
      app:
        spec:
          deploy:
          - kapp: {}
          template:
          - ytt:
              paths:
              - config
          - kbld: {}
      export:
      - imgpkgBundle:
          image: ghcr.io/joostvdg/gitstafette/client-bundle
          useKbldImagesLock: false
        includePaths:
        - config

I can work around this by generating an Image Lock File via kbld before running kctrl package release, though this kinda sucks. But, for that to work, I have to 1) fix my bundle to a specific platform (not desired), 2) add additional config to the package-build.yml, and 3) re-create the .imgpkg folder before running kbld.

      export:
      - imgpkgBundle:
          image: ghcr.io/joostvdg/gitstafette/client-bundle
        includePaths:
        - config
        - .imgpkg/images.yml

My kbld command is as follows:

mkdir -p packages/gitstafette-client/.imgpkg
    ytt -f packages/gitstafette-client/config | kbld -f - --platform linux/arm64 --imgpkg-lock-output packages/gitstafette-client/.imgpkg/images.yml

In both scenarios, useKbldImagesLock: false is removed, and the folder .imgpkg is deleted.

100mik commented 1 year ago

Related #963

100mik commented 1 year ago

@joostvdg you do need a lock file while releasing a Package bundle.

The intention of the useKbldImagesLock is for users who want to supply their own lock file. I am not very familiar with the guidance about lock files for multi-arch images or if it is possible (cc: @joaopapereira ).

Is the intention here to supply an empty images lock file so that no images are locked on to?

100mik commented 1 year ago

Ran into a tricky spot with this one, we are still running kbld and overwriting the user supplied images lock no matter what. Something we definitely do not want. This does mean that the reconciler (which runs ytt and then kbld initially) needs to be aware of whether the export section has useKbldImagesLock.

Makes this a bit tricky, giving it some thought. Even though the current changes achieve the goal of this PR. The useKbldImagesLock option is not exactly functional. Should we include that within the goals of this PR? (cc: @praveenrewar )

praveenrewar commented 1 year ago

Even though the current changes achieve the goal of this PR. The useKbldImagesLock option is not exactly functional. Should we include that within the goals of this PR?

I think we can go ahead with the current PR have another one to fix this.

joaopapereira commented 1 year ago

The intention of the useKbldImagesLock is for users who want to supply their own lock file. I am not very familiar with the guidance about lock files for multi-arch images or if it is possible (cc: @joaopapereira ).

@joostvdg multi-arch is somewhat of a strange situation, because both kbld and imgpkg work with multi-arch indexes well, but it feels like they do not. An example of this is if you create a file with the following configuration

image: nginx:1.14.2

and run kbld you will get the following output

---
image: index.docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: 1.14.2
            url: nginx:1.14.2
        url: index.docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d

and an ImagesLock that look like this

---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- annotations:
    kbld.carvel.dev/id: nginx:1.14.2
    kbld.carvel.dev/origins: |
      - resolved:
          tag: 1.14.2
          url: nginx:1.14.2
  image: index.docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
kind: ImagesLock

if you follow the SHA's you see the following

$ crane manifest index.docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 948,
         "digest": "sha256:706446e9c6667c0880d5da3f39c09a6c7d2114f5a5d6b74a2fafd24ae30d2078",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
.......

It is pointing to the index. imgpkg itself when copying the index will copy all the images associated with the index to the destination of your choosing. So if what you want is the multi-arch index to be pasted on your final configuration you should be good. The problem will be if you want an OS-specific image to be pasted in the image field. For that kbld does have a feature on the override section called platformSelection that allow you to specify what is the OS you want images for.

This was maybe a lot of information above. In your use case (@joostvdg) what is the problem that you are experiencing with the multi-arch images?

microwavables commented 1 year ago

Checking in on this -- @joostvdg have you had a chance to read over @joaopapereira comment?

joostvdg commented 1 year ago

My apologies @joaopapereira and @microwavables , I totally missed the response.

The problem I saw, or at least, I believe I saw, was the following:

build multi-arch image, published to ghcr (amd64, arm64)
publish package with kctrl package release, including creating an image lock file
deploy package in cluster with one amd64 node and several arm64 (Raspberry Pis) nodes
sometimes the application fails to start, because it was running on the wrong architecture

The above I'm quite clear on, from there on, I made assumptions, which in my frustration did not verify thoroughly, my apologies.

see kbld resolves image to sha
verify sha with images in ghcr -> notice the sha coincides with the amd64 image, and thus cannot run on the pi's (there's more of them, so decent chance)
I assumed, this is because kbld, or at least in combination with ghcr, resolves to a single platform
for now, changed the image to arm64 only, as I don't need it as multi-arch currently

This is for a hobby project, I don't work on it every week, and once things "sort of work", I tend to go on to the next challenge. I'll spend some time this weekend to re-produce the original setup, and do a more thorough investigation.

Due to the above described issue, I tried setting useKbldImagesLock: false, which promptly got removed. Which triggered me finding this issue writing my original comment. If there really is an issue in the multi-arch + kbld + ghcr setup, I'll create a new issue.

joostvdg commented 1 year ago

@microwavables @joaopapereira I went through my release process again, and found no issues. So feel free to ignore my comments here.

The only thing that still stands, is the automatic removal/file update when using useKbldImagesLock: false. At least for the version reported.

neil-hickey commented 1 year ago

@joaopapereira / @100mik . Seems like this got resolved? Shall we close this one out?

imusmanmalik commented 10 months ago

helloWorld()

This is something i stumbled upon as well, basically generating using impkg ImagesLock using kbld before doing a kctrl package release

ytt -f ... | kbld -f - --imgpkg-lock-output=./images.yaml

apiVersion: kctrl.carvel.dev/v1alpha1
kind: PackageBuild
metadata:
  creationTimestamp: null
  name: my-app
spec:
  template:
    spec:
      app:
        spec:
          deploy:
          - kapp: {}
          template:
            - ytt:
                paths:
                  - ./manifests
                  - ./config
            - kbld:
                paths:
                  - '-'
                  - images.yml
      export:
        - imgpkgBundle:
            image: ttl.sh/my-app
            useKbldImagesLock: true
          includePaths:
            - ./manifests
            - ./config

I would expect that i am also able to use a provided kbld ImagesLock file but cannot as it gets overwritten as mentioned: https://github.com/carvel-dev/kapp-controller/issues/943#issuecomment-1366037798 and https://github.com/carvel-dev/kapp-controller/issues/943#issue-1414457052

I believe useKbldImagesLock is not respected which user can choose to provide a lock file.

renuy commented 9 months ago

cc: @100mik

carvel-dev / kapp-controller

Cannot release package with kctrl #943