search

oras-project / oras

OCI registry client - managing content like artifacts, images, packages
https://oras.land
Apache License 2.0
1.41k stars 171 forks source link

Incomplete output when copying #916

Open toddysm opened 1 year ago

toddysm commented 1 year ago

What happened in your environment?

I have the following hierarchy in one registry:

~ oras discover $IMAGE -o tree
ghcr.io/toddysm/cssc-pipeline/copy-testimage@sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7
├── application/vnd.cyclonedx+json
│   └── sha256:0c3b33f215cb656e66954bfdbb0a68929c7a23843100f2cafa0b3fb2d6be6375
│       └── application/vnd.dev.cosign.artifact.sig.v1+json
│           └── sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7
├── application/sarif
│   └── sha256:bacff58d032857055dcacb3e4fe418e1dd00c54c636b0196b0efe565a6ba7817
└── application/vnd.dev.cosign.artifact.sig.v1+json
    └── sha256:fae8a85a05bfb8c9c6d63cda2b2c216a277458c7519e6382d6919d3c9926333f
        └── application/vnd.dev.cosign.artifact.sig.v1+json
            └── sha256:26971ffc0fd8b5464ab7144e72f02c0ca7cbaf819d810dcab2cfb3488a0b0b53

I am using oras copy -r to copy the above hierarchy to another registry. The command ouputs the following:

~ oras copy -r $IMAGE registry.twnt.co/copy-testimage:latest
Copying 0d3493b9c999 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/vulnerability-reports/copy-testimage-latest-20230404.sarif
Copied  0d3493b9c999 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/vulnerability-reports/copy-testimage-latest-20230404.sarif
Copying 2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Copied  2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Copied [registry] ghcr.io/toddysm/cssc-pipeline/copy-testimage:latest => [registry] registry.twnt.co/copy-testimage:latest
Digest: sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7

The output shows only three artifacts copied to the new registry, which leaves the user wondering whether all artifacts have been copied. In reality all are, but the command output is misleading. Here is the tree in the new registry:

~  oras discover registry.twnt.co/copy-testimage:latest -o tree
registry.twnt.co/copy-testimage@sha256:b4c0ed46d75ff7a1e8bcc166156db356dbb3768181d010a639ad4b4dd2f43db7
├── application/sarif
│   └── sha256:bacff58d032857055dcacb3e4fe418e1dd00c54c636b0196b0efe565a6ba7817
├── application/vnd.dev.cosign.artifact.sig.v1+json
│   └── sha256:fae8a85a05bfb8c9c6d63cda2b2c216a277458c7519e6382d6919d3c9926333f
│       └── application/vnd.dev.cosign.artifact.sig.v1+json
│           └── sha256:26971ffc0fd8b5464ab7144e72f02c0ca7cbaf819d810dcab2cfb3488a0b0b53
└── application/vnd.cyclonedx+json
    └── sha256:0c3b33f215cb656e66954bfdbb0a68929c7a23843100f2cafa0b3fb2d6be6375
        └── application/vnd.dev.cosign.artifact.sig.v1+json
            └── sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7

What did you expect to happen?

I expect the command to output correct information about all artifacts that are copied and provide a summary

How can we reproduce it?

Use oras copy -r on a deep hierarchy

What is the version of your ORAS CLI?

Version: 1.0.0-rc.2 Go version: go1.20.1 Git commit: 3c5e899f6f32299080074e665ba7c44e25aa639f Git tree state: clean

What is your OS environment?

MacOS

Are you willing to submit PRs to fix it?

qweeah commented 1 year ago

I expect the command to output correct information about all artifacts that are copied and provide a summary

Sorry I must miss something here, are you suggesting all the nodes should be mentioned in the output even if --verbose is not set?

I think there are some gaps that need to be filled here: originally oras was designed as a client tool to upload/download files to/from OCI registries. IMHO, to expose less low-level storage detail like how a file is stored via OCI image/artifacts, unnamed manifests and blobs should not be in the default output of oras. For example, if you run oras pull on the signature artifact, only the copy-testimage-latest.json will be copied to your local file system:

$ oras pull ghcr.io/toddysm/cssc-pipeline/copy-testimage@sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7 -T --include-subject
Downloading 2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Downloaded  2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Pulled [registry] ghcr.io/toddysm/cssc-pipeline/copy-testimage@sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7
Digest: sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7

Same for copy-testimage-latest.json

$ oras pull ghcr.io/toddysm/cssc-pipeline/copy-testimage@sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7 -T --include-subject
Downloading 2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Downloaded  2eb165eb8127 /Users/toddysm/Documents/Development/kubecon-eu-2023-talks/sboms/copy-testimage-latest.json
Pulled [registry] ghcr.io/toddysm/cssc-pipeline/copy-testimage@sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7
Digest: sha256:914ec14e0ff6f79f2f9df3dbb03421fdb9d4cdbfaf0313198b72b62f127e04a7

So for oras cp, only copy-testimage-latest.json and copy-testimage-latest.json is shown in the default output.

toddysm commented 1 year ago

Sorry I must miss something here, are you suggesting all the nodes should be mentioned in the output even if --verbose is not set?

Correct, this is what I expect. I have 6 artifacts that I am copying but I see only 3 outputted by the command. This is misleading.

qweeah commented 1 year ago

Correct, this is what I expect. I have 6 artifacts that I am copying but I see only 3 outputted by the command. This is misleading.

I am okay with this change, since oras cp doesn't operate file store at all, the user usually care more about artifacts/images than files.