openshift / oc-mirror

Lifecycle manager for internet-disconnected OpenShift environments
Apache License 2.0
82 stars 80 forks source link

OCPBUGS-33196: skip images with both tag and digest #853

Closed lmzuccarelli closed 1 month ago

lmzuccarelli commented 1 month ago

Description

This addresses the issue when images being mirrored, having both tag and digest would cause the mirror execution to fail, the fix ensures these type images will be skipped with an appropriate warning message.

Fixes # OCPBUGS-33196

Type of change

How Has This Been Tested?

I used the following imagesetconfig for testing

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
   additionalImages:
   - name: quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc
   - name: quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9

Executed the following mirror to disk workflow

bin/oc-mirror --config ocpbugs-33196.yaml file://taganddigest --v2

Expected Outcome

2024/05/14 14:18:35  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2024/05/14 14:18:35  [INFO]   : 👋 Hello, welcome to oc-mirror
2024/05/14 14:18:35  [INFO]   : ⚙️  setting up the environment for you...
2024/05/14 14:18:35  [INFO]   : 🔀 workflow mode: mirrorToDisk 
2024/05/14 14:18:35  [INFO]   : 🕵️  going to discover the necessary images...
2024/05/14 14:18:35  [INFO]   : 🔍 collecting release images...
2024/05/14 14:18:35  [INFO]   : 🔍 collecting operator images...
2024/05/14 14:18:35  [INFO]   : 🔍 collecting additional images...
2024/05/14 14:18:35  [WARN]   : [AdditionalImagesCollector] mirroring skipped : source image quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc has both tag and digest
2024/05/14 14:18:35  [WARN]   : [AdditionalImagesCollector] mirroring skipped : source image quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 has both tag and digest
2024/05/14 14:18:35  [INFO]   : 🚀 Start copying the images...
2024/05/14 14:18:35  [INFO]   : === Results ===
2024/05/14 14:18:35  [INFO]   : All release images mirrored successfully 0 / 0 ✅
2024/05/14 14:18:35  [INFO]   : All operator images mirrored successfully 0 / 0 ✅
2024/05/14 14:18:35  [INFO]   : All additional images mirrored successfully 0 / 0 ✅
2024/05/14 14:18:35  [INFO]   : 📦 Preparing the tarball archive...
2024/05/14 14:18:35  [INFO]   : mirror time     : 1.770588ms
2024/05/14 14:18:35  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
openshift-ci-robot commented 1 month ago

@lmzuccarelli: This pull request references Jira Issue OCPBUGS-33196, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/oc-mirror/pull/853): ># Description > >This addresses the issue when images being mirrored, having both tag and digest would cause the mirror execution to fail, the fix ensures these type images will be skipped with an appropriate warning message. > >Fixes # OCPBUGS-33196 > >## Type of change > >- [x] Bug fix (non-breaking change which fixes an issue) >- [ ] New feature (non-breaking change which adds functionality) >- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) >- [ ] This change requires a documentation update > ># How Has This Been Tested? > >I used the following imagesetconfig for testing > >``` >kind: ImageSetConfiguration >apiVersion: mirror.openshift.io/v2alpha1 >mirror: > additionalImages: > - name: quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc > - name: quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 > >``` > >Executed the following mirror to disk workflow > >``` >bin/oc-mirror --config ocpbugs-33196.yaml file://taganddigest --v2 >``` > >## Expected Outcome > >``` >2024/05/14 14:18:35 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. >2024/05/14 14:18:35 [INFO] : 👋 Hello, welcome to oc-mirror >2024/05/14 14:18:35 [INFO] : ⚙️ setting up the environment for you... >2024/05/14 14:18:35 [INFO] : 🔀 workflow mode: mirrorToDisk >2024/05/14 14:18:35 [INFO] : 🕵️ going to discover the necessary images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting release images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting operator images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting additional images... >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc has both tag and digest >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 has both tag and digest >2024/05/14 14:18:35 [INFO] : 🚀 Start copying the images... >2024/05/14 14:18:35 [INFO] : === Results === >2024/05/14 14:18:35 [INFO] : All release images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All operator images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All additional images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : 📦 Preparing the tarball archive... >2024/05/14 14:18:35 [INFO] : mirror time : 1.770588ms >2024/05/14 14:18:35 [INFO] : 👋 Goodbye, thank you for using oc-mirror > >``` > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Foc-mirror). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
lmzuccarelli commented 1 month ago

/jira refresh

openshift-ci-robot commented 1 month ago

@lmzuccarelli: This pull request references Jira Issue OCPBUGS-33196, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.16.0) matches configured target version for branch (4.16.0) * bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact: /cc @zhouying7780

In response to [this](https://github.com/openshift/oc-mirror/pull/853#issuecomment-2110210321): >/jira refresh Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Foc-mirror). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci[bot] commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lmzuccarelli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/oc-mirror/blob/main/OWNERS)~~ [lmzuccarelli] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
kasturinarra commented 1 month ago

Tested, have few questions, waiting for reply from Luigi.

lmzuccarelli commented 1 month ago

@kasturinarra - I updated the console message as requested

kasturinarra commented 1 month ago

@kasturinarra - I updated the console message as requested

Thank you !! Let me test again

kasturinarra commented 1 month ago

@kasturinarra - I updated the console message as requested

Thank you !! Let me test again

could see log has been adjusted. see below

2024/05/16 10:28:44 [WARN] : [AdditionalImagesCollector] quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc has both tag and digest : SKIPPING 2024/05/16 10:28:44 [WARN] : [AdditionalImagesCollector] quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 has both tag and digest : SKIPPING

kasturinarra commented 1 month ago

/label qe-approved

openshift-ci-robot commented 1 month ago

@lmzuccarelli: This pull request references Jira Issue OCPBUGS-33196, which is valid.

3 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.16.0) matches configured target version for branch (4.16.0) * bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact: /cc @kasturinarra

In response to [this](https://github.com/openshift/oc-mirror/pull/853): ># Description > >This addresses the issue when images being mirrored, having both tag and digest would cause the mirror execution to fail, the fix ensures these type images will be skipped with an appropriate warning message. > >Fixes # OCPBUGS-33196 > >## Type of change > >- [x] Bug fix (non-breaking change which fixes an issue) >- [ ] New feature (non-breaking change which adds functionality) >- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) >- [ ] This change requires a documentation update > ># How Has This Been Tested? > >I used the following imagesetconfig for testing > >``` >kind: ImageSetConfiguration >apiVersion: mirror.openshift.io/v2alpha1 >mirror: > additionalImages: > - name: quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc > - name: quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 > >``` > >Executed the following mirror to disk workflow > >``` >bin/oc-mirror --config ocpbugs-33196.yaml file://taganddigest --v2 >``` > >## Expected Outcome > >``` >2024/05/14 14:18:35 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. >2024/05/14 14:18:35 [INFO] : 👋 Hello, welcome to oc-mirror >2024/05/14 14:18:35 [INFO] : ⚙️ setting up the environment for you... >2024/05/14 14:18:35 [INFO] : 🔀 workflow mode: mirrorToDisk >2024/05/14 14:18:35 [INFO] : 🕵️ going to discover the necessary images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting release images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting operator images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting additional images... >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc has both tag and digest >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 has both tag and digest >2024/05/14 14:18:35 [INFO] : 🚀 Start copying the images... >2024/05/14 14:18:35 [INFO] : === Results === >2024/05/14 14:18:35 [INFO] : All release images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All operator images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All additional images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : 📦 Preparing the tarball archive... >2024/05/14 14:18:35 [INFO] : mirror time : 1.770588ms >2024/05/14 14:18:35 [INFO] : 👋 Goodbye, thank you for using oc-mirror > >``` > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Foc-mirror). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
sherine-k commented 1 month ago

lgtm I was wondering if there would be a benefit to add these images to the list of images that fail to mirror by batch ?

sherine-k commented 1 month ago

/lgtm /label acknowledge-critical-fixes-only

openshift-ci[bot] commented 1 month ago

@lmzuccarelli: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
openshift-ci-robot commented 1 month ago

@lmzuccarelli: Jira Issue OCPBUGS-33196: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-33196 has been moved to the MODIFIED state.

In response to [this](https://github.com/openshift/oc-mirror/pull/853): ># Description > >This addresses the issue when images being mirrored, having both tag and digest would cause the mirror execution to fail, the fix ensures these type images will be skipped with an appropriate warning message. > >Fixes # OCPBUGS-33196 > >## Type of change > >- [x] Bug fix (non-breaking change which fixes an issue) >- [ ] New feature (non-breaking change which adds functionality) >- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) >- [ ] This change requires a documentation update > ># How Has This Been Tested? > >I used the following imagesetconfig for testing > >``` >kind: ImageSetConfiguration >apiVersion: mirror.openshift.io/v2alpha1 >mirror: > additionalImages: > - name: quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc > - name: quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 > >``` > >Executed the following mirror to disk workflow > >``` >bin/oc-mirror --config ocpbugs-33196.yaml file://taganddigest --v2 >``` > >## Expected Outcome > >``` >2024/05/14 14:18:35 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. >2024/05/14 14:18:35 [INFO] : 👋 Hello, welcome to oc-mirror >2024/05/14 14:18:35 [INFO] : ⚙️ setting up the environment for you... >2024/05/14 14:18:35 [INFO] : 🔀 workflow mode: mirrorToDisk >2024/05/14 14:18:35 [INFO] : 🕵️ going to discover the necessary images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting release images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting operator images... >2024/05/14 14:18:35 [INFO] : 🔍 collecting additional images... >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/cilium/cilium-etcd-operator:v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc has both tag and digest >2024/05/14 14:18:35 [WARN] : [AdditionalImagesCollector] mirroring skipped : source image quay.io/coreos/etcd:v3.5.4@sha256:a67fb152d4c53223e96e818420c37f11d05c2d92cf62c05ca5604066c37295e9 has both tag and digest >2024/05/14 14:18:35 [INFO] : 🚀 Start copying the images... >2024/05/14 14:18:35 [INFO] : === Results === >2024/05/14 14:18:35 [INFO] : All release images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All operator images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : All additional images mirrored successfully 0 / 0 ✅ >2024/05/14 14:18:35 [INFO] : 📦 Preparing the tarball archive... >2024/05/14 14:18:35 [INFO] : mirror time : 1.770588ms >2024/05/14 14:18:35 [INFO] : 👋 Goodbye, thank you for using oc-mirror > >``` > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Foc-mirror). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-bot commented 1 month ago

[ART PR BUILD NOTIFIER]

This PR has been included in build oc-mirror-plugin-container-v4.16.0-202405161711.p0.g2adde69.assembly.stream.el9 for distgit oc-mirror-plugin. All builds following this will include this PR.

openshift-merge-robot commented 1 month ago

Fix included in accepted release 4.16.0-0.nightly-2024-05-17-180525