List bundle dependencies

ashvinmoro commented 3 years ago

Describe the problem/challenge you have List dependencies: An imgpkg bundle can either be a standalone bundle or a package repository, with them referencing dependent artifacts which can either be OCI artifacts or other imgpkg bundles. The imgpkg CLI today does not provide an option to query a bundle to list all dependent artifacts (regardless of the level of nesting) without doing a pull and parsing through the contents of the bundle.

This requires knowing the internals of the imgpkg bundle layout which is an implementation detail and causes too tight a coupling for clients consuming the CLI.

This feature is very helpful to inspect contents of the bundle metadata without requiring to download it, similar to how artifact manifest can be inspected using the crane command.

Describe the solution you'd like Provide a CLI option and an API to allow clients consuming the bundles to query for all dependent artifacts and their types (bundle, repo, image) with their references locked to the relocated repository, similar to how “imgpkg pull” would fix the images lock.

Example: imgpkg -b <bundle-ref> --list-deps returns images referenced after the images are locked Images referenced in the ImagesLock of the bundle: index.docker.io/test/package-repo@sha256:b5e1cfb8731bf5ba6a0fb7060c37bbca03e9fb12be56dbec4a1fe2bbdfff2bd3 index.docker.io/test/package-repo@sha256:42030f63b66e25a03c7d9ef28fa5369d230df5697b4ab2d94860e3f69026d18a

Anything else you would like to add: None

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

ashvinmoro commented 3 years ago

cc: @ncdc

DennisDenuto commented 3 years ago

Thanks for creating this issue.

Provide a CLI option and an API... to query for... types

The CLI option can be added to the pull command. If this flag is present, the last section of output can be a flat list of all the dependent images that were pulled (also whether the dependent image is a bundle or an OCI image) Assuming we display whether the dependent image is a bundle or OCI image, would you still expect an API to 'query' the type?

Are you able to expand on the use case driving this request? or, why do you want a full list of dependent images per bundle?

I am trying to understand whether this is purely for information (which in that case the pull command outputting additional information seems like a fine idea), or whether this is being used to maybe verify / validate something?

In which case, I think discussing a new command to 'inspect' a bundle, and output pertinent information such as dependencies, ImagesLocation, signatures and tags might be useful. As well as any additional options such as filtering too.

ashvinmoro commented 3 years ago

The CLI option can be added to the pull command. If this flag is present, the last section of output can be a flat list of all the dependent images that were pulled (also whether the dependent image is a bundle or an OCI image) Assuming we display whether the dependent image is a bundle or OCI image, would you still expect an API to 'query' the type?

To keep the semantic separation a separate query command is better. Under the hood, this command may likely do a pull command but that will then be an implementation detail.

Are you able to expand on the use case driving this request? or, why do you want a full list of dependent images per bundle?

I am trying to understand whether this is purely for information (which in that case the pull command outputting additional information seems like a fine idea), or whether this is being used to maybe verify / validate something?

The use case is to track the dependent images and establish policy based access controls for users downloading the bundles and its images. Right now we are having to unpack the bundle and identify the dependents manually which is too dependent on the bundle implementation.

In which case, I think discussing a new command to 'inspect' a bundle, and output pertinent information such as dependencies, ImagesLocation, signatures and tags might be useful. As well as any additional options such as filtering too.

+1

DennisDenuto commented 3 years ago

Thanks @ashvinmoro

I think given the specific use case for this feature, a new command to 'inspect' the bundle should be done.

pivotaljohn commented 3 years ago

We've gotten information that this is a higher priority:

there was an assumption that this could achieved through inspecting images.yml by pull recursive. However, we get an incomplete set of images; that approach is not viable.

DennisDenuto commented 3 years ago

TODO: Prior to picking this up / working on this

detail what the command options/flags will be, and the format of the output and get 'early' feedback from @ashvinmoro

pivotaljohn commented 2 years ago

Review #124 — which may be requesting the same functionality.

note the sample output: https://github.com/vmware-tanzu/carvel-imgpkg/issues/124#issuecomment-823325183
that other story was describing this feature, in general. What context from this story can help us prioritize what aspects of this feature to implement now (and what we might want to defer).

joaopapereira commented 2 years ago

As @pivotaljohn mentioned we have discussed the creation of a describe command in issue #124, @ashvinmoro do you think that the proposed output can solve your needs?

joaopapereira commented 2 years ago

@ashvinmoro any comments about the output of the describe command proposed in #124 ?

ashvinmoro commented 2 years ago

@pivotaljohn @joaopapereira I went through the issue #124. There are a couple of aspects I had suggestions about

List all of the image references that are part of the bundle. This appears to be covered in issue #124.
If the bundle is a nested bundle, list the image references of nested bundles as well as the image references of each nested bundle. This could either be provided as a recursive listing or as a separate option by passing in the reference of the nested bundle.

$ imgpkg describe -b new.registry.io/simple-app-install-package

Bundle SHA: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
Tags: latest, v1.0.0

Authors:
  Carvel Team <carvel@vmware.com>
Websites:
  carvel.dev/imgpkg
Metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

Images:
  - new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
     **Type: Bundle**
    Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Images:
      - new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        **Type: Image**
        Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application

      - new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        **Type: Image**
        Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

joaopapereira commented 2 years ago

@ashvinmoro Take a look at the following example to see if they look good for your use case:

Examples

Example 1: Describe command without any argument

$ imgpkg describe -b new.registry.io/simple-app-install-package 

Bundle SHA: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
Tags: latest, v1.0.0

Authors:
  Carvel Team <carvel@vmware.com>
Websites:
  carvel.dev/imgpkg
Metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Type: Bundle
    Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    Images:
      - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Type: Image
        Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        Annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application

      - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        Type: Image
        Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Type: Image
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 2: Describe command listing all images

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 3: Describe command listing all images in the current bundle without searching inner bundles

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --no-recursion

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 4: Describe command listing all images displaying the origin

$ imgpkg describe -b new.registry.io/simple-app-install-package --images

Bundles:
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0

Images:
  - Image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 5: Describe command listing all images when Bundle was not copied

$ imgpkg describe -b new.registry.io/simple-app-install-package --images

Bundles:
  - Origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4

Images:
  - Origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - Origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Previous examples but using a yaml output

Example 1: Describe command with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --output-type yaml

sha: aaaaad700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
tags: 
 - latest
 - v1.0.0
authors:
  - name: Carvel Team
    email: carvel@vmware.com
websites:
  - carvel.dev/imgpkg
metadata:
  - Some Version: 1.0.0
  - Other Information: Some text here

images:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    type: Bundle
    origin: my.registry.io/bundle1@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
    images:
      - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        type: Image
        origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        annotations:
          kbld.carvel.dev/id: my.registry.io/simple-application
      - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
        type: Image
        origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    type: Image
    origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 2: Describe command listing all images with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
images:
  - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 3: Describe command listing all images in the current bundle without searching inner bundles with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --current-location --no-recursion --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4
images:
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

Example 4: Describe command listing all images displaying the origin with yaml output

$ imgpkg describe -b new.registry.io/simple-app-install-package --images --output-type yaml

bundles:
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0

images:
  - image: new.registry.io/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    origin: registry.io/img1@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  - image: new.registry.io/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f
    origin: registry.io/img2@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f

jorgemoralespou commented 2 years ago

Nice, but as mentioned before in other discussions, it'll be nice to somehow have the original image tags annotated so that describe can provide some information that a human can understand :-D

joaopapereira commented 2 years ago

In some sense, the original tag will be present if you use kbld to generate your ImagesLock and that tag is used in the files read by kbld. Look at this example:

$  kbld -f basic-step-2/config.yml --imgpkg-lock-output basic-step-2/.imgpkg/images.yml
resolve | final: index.docker.io/dkalinin/k8s-simple-app:latest -> index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
---
apiVersion: v1
kind: Service
metadata:
  name: simple-app
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    simple-app: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: latest
            url: index.docker.io/dkalinin/k8s-simple-app:latest
        url: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
  name: simple-app
  namespace: default
spec:
  selector:
    matchLabels:
      simple-app: ""
  template:
    metadata:
      labels:
        simple-app: ""
    spec:
      containers:
      - env:
        - name: HELLO_MSG
          value: stranger
        image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
        name: simple-app

Succeeded

$ imgpkg push -b localhost:5000/test -f basic-step-2
dir: .
dir: .imgpkg
file: .imgpkg/bundle.yml
file: .imgpkg/images.yml
file: config.yml
Pushed 'localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291'
Succeeded

$ imgpkg describe -b localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291 -o yaml
sha: sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291
content:
  images:
    sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0:
      annotations:
        kbld.carvel.dev/id: index.docker.io/dkalinin/k8s-simple-app:latest
        kbld.carvel.dev/origins: |
          - resolved:
              tag: latest
              url: index.docker.io/dkalinin/k8s-simple-app:latest
      image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
      imageType: Image
      origin: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
image: localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291
metadata: {}
origin: localhost:5000/test@sha256:3cd8f0d917a92fafeef2d44eb63195d5351c35cf46b8b7448cb126c7d68c6291

Succeeded

Another thing we need to keep in mind is that imgpkg only deal with Full Described Images(uses digest only), so unless the tool that creates the ImagesLock provides this information, imgpkg as no way to know tag information

jorgemoralespou commented 2 years ago

Nice thanks for clarification. Is this also true for imgpkg copy, which is how most of the times images get to local clusters?

joaopapereira commented 2 years ago

Not following 100% what you mean, but the information we are displaying is the information that is part of the ImagesLock, so if the information is in the ImagesLock it will be displayed

carvel-dev / imgpkg

List bundle dependencies #204

Examples

Previous examples but using a yaml output