search

kubernetes-sigs / promo-tools

Container and file artifact promotion tooling for the Kubernetes project
Apache License 2.0
143 stars 72 forks source link

Recognize Child Images Defined In Promotion Manifests #402

Open tylerferrara opened 2 years ago

tylerferrara commented 2 years ago

What would you like to be added:

If a child image is defined within a sub-project's promotion manifest, it will not be seen by the Auditor since the parent image (manifest list) is under a different name. Our incoming child image will never match an existing child definition, because it's under a different name.

Example

Most manifest lists look something like this:

# k8s.io/k8s.gcr.io/images/k8s-staging-sub-project/images.yaml
- name: logger
  dmap:
    "sha256:c4151a15c8439265d98f66d25ef17964e9e975d894822a54ed7e72db78dba6c6": ["parent"]
- name: logger-amd
  dmap:
    "sha256:2c9c8df42ac7525e556bbff81aa9a62960888c69d5faad4aad408893bc95cbc9": ["child_amd"]
- name: logger-arm
  dmap:
    "sha256:a41a91e366e973da0bfd6fce44ba131d561ab435119ff7e1050d1e226a06dbda": ["child_arm"]

If the logger image is of mediaType: manifest.list (a parent image) which contains both logger-amd and logger-arm images, our Auditor does not recognize these children are actually defined within the promotion manifest. The incoming Pub/Sub message for a child image of logger looks like this:

gcr.io/k8s-sub-project/logger@sha256:2c9c8df42ac7525e556bbff81aa9a62960888c69d5faad4aad408893bc95cbc9

If you look carefully, this image does not exist! But in actuality, this is the child image logger-amd we defined within the promotion manifest. Since this is how an incoming Pub/Sub child image looks like, we must widen our criteria for linking images with a promotion manifest. ​

Why is this needed:

Looking at the sha256 digest, instead of the fully qualified image name (FQIN), the Auditor will be able to recognize incoming child images if they are define in a promotion manifest. Not all sub-projects explicitly define child images, however for the ones that do follow this convention the verification will not require a full read of the source registry. This change has the potential to dramatically decrease the number of HTTP request send to GCR if all child images can be found in the kubernetes/k8s.io repository. The result of this feature would reduce the number of instances the Auditor exceeds GCR Quotas and causes false alarms (Issue: Noisy Auditor)

cc: @listx @amwat @justaugustus @kubernetes-sigs/release-engineering

justaugustus commented 2 years ago

@tylerferrara -- in "What would you like to be added", would you mind adding a few lines about the requested feature?

You jump into a problem statement, but this should also include a crisp statement about the feature you're interested in seeing implemented.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

cpanato commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 1 year ago

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

varshith257 commented 1 month ago

After going through the context of this issue, I have an approach in mind:

To address the issue of recognizing child images defined in promotion manifests, we need to update the auditor logic to match incoming child images based on their SHA256 digest rather than the fully qualified image name (FQIN). We also need to ensure the auditor parses the promotion manifests to extract all child images along with their associated SHA256 digests and store these mappings in a data structure for quick lookup.

When a Pub/Sub message for an incoming image is received, extract the SHA256 digest from the image URL and compare this digest against the stored digests from the promotion manifests. If a match is found, recognize the image as a valid child image defined in the promotion manifest and proceed with the usual auditing process for matched images.

For this, we need to define the necessary data structures to hold the manifest information something would look like the following: 

type ManifestEntry struct {
    Name string            `json:"name"`
    Dmap map[string][]string `json:"dmap"`
}

type Auditor struct {
    ManifestDigests map[string]string
}

The loading and parsing of the promotion manifest to extract and store SHA256 digests for child images logic would look like the following:

func (a *Auditor) LoadManifests(manifestFiles []string) {
    for _, file := range manifestFiles {
        data, err := ioutil.ReadFile(file)
        if err != nil {
            log.Fatalf("Failed to read manifest file: %v", err)
        }
        var manifests []ManifestEntry
        if err := json.Unmarshal(data, &manifests); err != nil {
            log.Fatalf("Failed to unmarshal manifest data: %v", err)
        }
        a.ParseManifest(manifests)
    }
}

func (a *Auditor) ParseManifest(manifests []ManifestEntry) {
    for _, entry := range manifests {
        for digest := range entry.Dmap {
            a.ManifestDigests[digest] = entry.Name
        }
    }
}

Similarly, we must implement the logic to recognize incoming child images based on these stored digests. So that the Auditor can effectively recognize child images defined in promotion manifests, improving both efficiency and accuracy

cc: @justaugustus @xmudrii @kubernetes-sigs/release-engineering WDYT? Correct me, if any wrong in my approach.

varshith257 commented 1 month ago

/honk

k8s-ci-robot commented 1 month ago

@varshith257: goose image

In response to [this](https://github.com/kubernetes-sigs/promo-tools/issues/402#issuecomment-2143397227): >/honk Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
varshith257 commented 3 weeks ago

PTAL @xmudrii @cpanato for my approach