search

ratify-project / ratify

Artifact Ratification Framework
https://ratify.dev
Apache License 2.0
221 stars 60 forks source link

Should we have a repository to keep ratify images #521

Open susanshi opened 1 year ago

susanshi commented 1 year ago

Describe the solution you'd like Currently we have two uses cases for test images:

The images are currently kept in wabbitnetworks registry, lets use this issue to discuss:

  1. should we keep test images for e2e test, or should our e2e test produce these artifact as a test step. It is convenient to use prebuilt test image, but we will have to keep them up to date.
  2. what is the process to update test images? Today, not all of us has access to wabbitnetworks registry.
binbin-li commented 1 year ago

I feel once we have the tests on local private registry, we can just build those artifacts in each test instead of a prebuilt image since not everyone has access to the remote registry.

dtzar commented 1 year ago

Re: 2 - the process would be to run the pipeline in the wabbitnetworks repository. The pipeline could be enhanced with this purpose if desired #389

Also - the access is easy to provide to anyone on the dev team. I gave everyone I knew of on the team access.

toddysm commented 1 year ago

May I ask what is the value to build the new images every time we run the tests?

Also, doing this will test with only one registry, and this registry may or may not be used by the actual users of Ratify. To do a proper testing we need to test with few different registries because they support different capabilities. If I am to choose, the current test matrix should be:

Registry Support Notes
MAR OCI 1.1 MAR is the only public cloud registry that currently supports OCI 1.1 (RC.1)
Docker Hub Hybrid OCI 1.1 and OCI 1.0 Docker Hub has a hybrid support for artifacts and referrers and it is good test case
GHCR OCI 1.0 This will test the legacy support

I believe the goal here is to test the variety of options and not just one specific case.

In the future, we should add other public registries that have signatures.

toddysm commented 1 year ago

To add to the above, I agree with what @dtzar wrote in #389 - we should populate withe different content if possible in those registries.