openshift / oc-mirror

Lifecycle manager for internet-disconnected OpenShift environments
Apache License 2.0
82 stars 80 forks source link

CLID-99: consolidates the release collector for delete feature #883

Closed aguidirh closed 3 days ago

aguidirh commented 1 week ago

Description

This PR consolidates the release collector to only one to be used in both mirroring and deleting process. In this way it is possible to collect the same images in both process since the collector is the same.

NOTE: The deletion of operators when the previous workflow was mirrorToMirror might be impacted by this change and it is going to be handled in a different PR.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

mirror to disk and disk to mirror scenario

apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    channels:
      - name: stable-4.13 
        minVersion: 4.13.3
        maxVersion: 4.13.3

1 step: mirror to disk

./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 --v2

2 step: disk to mirror

./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml --from file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 docker://localhost:6000 --v2

3 step: check the blobs/repositories on the localhost:6000

output:

blobs:  849 directories, 614 files
repositories:  640 directories, 621 files

4 step: delete generate

./bin/oc-mirror delete -c ./alex-tests/alex-isc/clid-99-delete.yaml --generate --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 docker://localhost:6000 --v2

5 step: delete

./bin/oc-mirror delete --delete-yaml-file /home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99/working-dir/delete/delete-images.yaml docker://localhost:6000 --v2

6 step: run the garbage collector on localhost:6000

output:

registry garbage-collect /etc/docker/registry/config.yml

7 step: check the blobs/repositories on the localhost:6000

output:

blobs: 236 directories, 0 files
repositories: 635 directories, 435 files

=================================================================================================== mirror to mirror scenario

apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    channels:
      - name: stable-4.13 
        minVersion: 4.13.3
        maxVersion: 4.13.3

1 step: mirror to mirror

./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror docker://localhost:7000 --dest-tls-verify=false --v2 

2 step: check the blobs/repositories on the localhost:7000

output:

blobs: 850 directories, 614 files
repositories: 640 directories, 621 files

3 step: delete generate

./bin/oc-mirror delete -c ./alex-tests/alex-isc/clid-99-delete.yaml --generate --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror docker://localhost:7000 --v2

4 step: delete

./bin/oc-mirror delete --delete-yaml-file /home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror/working-dir/delete/delete-images.yaml docker://localhost:7000 --dest-tls-verify=false --v2 

5 step: run the garbage collector on localhost:7000

output:

registry garbage-collect /etc/docker/registry/config.yml

6 step: check the blobs/repositories on the localhost:6000

output:

blobs: 236 directories, 0 files
repositories: 635 directories, 435 files
openshift-ci-robot commented 1 week ago

@aguidirh: This pull request references CLID-99 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/oc-mirror/pull/883): ># Description > >This PR consolidates the release collector to only one to be used in both mirroring and deleting process. In this way it is possible to collect the same images in both process since the collector is the same. > >**NOTE**: The deletion of operators when the previous workflow was mirrorToMirror might be impacted by this change and it is going to be handled in a different PR. > >Fixes # (issue) > >## Type of change > >Please delete options that are not relevant. > >- [ ] Bug fix (non-breaking change which fixes an issue) >- [ ] New feature (non-breaking change which adds functionality) >- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) >- [ ] This change requires a documentation update > ># How Has This Been Tested? >mirror to disk and disk to mirror scenario > >``` >apiVersion: mirror.openshift.io/v2alpha1 >kind: ImageSetConfiguration >mirror: > platform: > channels: > - name: stable-4.13 > minVersion: 4.13.3 > maxVersion: 4.13.3 >``` > >1 step: mirror to disk > >``` >./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 --v2 >``` > >2 step: disk to mirror >``` >./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml --from file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 docker://localhost:6000 --v2 >``` > >3 step: check the blobs/repositories on the localhost:6000 > >output: > >``` >blobs: 849 directories, 614 files >repositories: 640 directories, 621 files >``` > >4 step: delete generate > >``` >./bin/oc-mirror delete -c ./alex-tests/alex-isc/clid-99-delete.yaml --generate --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99 docker://localhost:6000 --v2 >``` > >5 step: delete > >``` >./bin/oc-mirror delete --delete-yaml-file /home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99/working-dir/delete/delete-images.yaml docker://localhost:6000 --v2 >``` > >6 step: run the garbage collector on localhost:6000 > >output: > >``` >registry garbage-collect /etc/docker/registry/config.yml >``` > >7 step: check the blobs/repositories on the localhost:6000 > >output: > >``` >blobs: 236 directories, 0 files >repositories: 635 directories, 435 files >``` > > > > >=================================================================================================== >mirror to mirror scenario > >``` >apiVersion: mirror.openshift.io/v2alpha1 >kind: ImageSetConfiguration >mirror: > platform: > channels: > - name: stable-4.13 > minVersion: 4.13.3 > maxVersion: 4.13.3 >``` > >1 step: mirror to mirror > >``` >./bin/oc-mirror -c ./alex-tests/alex-isc/clid-99.yaml --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror docker://localhost:7000 --dest-tls-verify=false --v2 >``` > >2 step: check the blobs/repositories on the localhost:7000 > >output: > >``` >blobs: 850 directories, 614 files >repositories: 640 directories, 621 files >``` > >3 step: delete generate >``` >./bin/oc-mirror delete -c ./alex-tests/alex-isc/clid-99-delete.yaml --generate --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror docker://localhost:7000 --v2 >``` > >4 step: delete >``` >./bin/oc-mirror delete --delete-yaml-file /home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-99-mirrortomirror/working-dir/delete/delete-images.yaml docker://localhost:7000 --dest-tls-verify=false --v2 >``` > >5 step: run the garbage collector on localhost:7000 > >output: > >``` >registry garbage-collect /etc/docker/registry/config.yml >``` > >6 step: check the blobs/repositories on the localhost:6000 > >output: > >``` >blobs: 236 directories, 0 files >repositories: 635 directories, 435 files >``` Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Foc-mirror). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci[bot] commented 1 week ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aguidirh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/oc-mirror/blob/main/OWNERS)~~ [aguidirh] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
lmzuccarelli commented 1 week ago

Verified locally using the steps as described in the initial comments section.

Before delete (garbage-collect)

du -h
17G var/lib/registry/docker/registry/

After delete

du -h 
1.7M    var/lib/registry/docker/registry/
sherine-k commented 3 days ago

Great improvement @aguidirh ! LGTM from my side

openshift-ci[bot] commented 3 days ago

@aguidirh: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
lmzuccarelli commented 3 days ago

/lgtm