Proposal: refactor conformance tests

[rchincha]

The feedback about conformance tests is that they have been useful but UX in general is terrible.

Here are some UX requirements from the last OCI meeting (08/01/2024)

[ ] Root-cause-analysis of test failures should be easier/clearer

[ ] Conformance tests should be configuration (config-file) driven so that

• the MAY, MUST, SHOULD, etc can be set, and individual tests and suites can be turned on or off
• the config-file can be shared so that it can be used to verify and derive
• OCI specs can publish a recommended config-file on GA

[ ] Clients expectations (of registries) from conformance tests

[ ] Registries' expectations (of clients) from conformance tests

[ ] A website revamp for the results/config matrix

[rchincha]

https://github.com/opencontainers/distribution-spec/issues/441

[sudo-bmitch]

I've volunteered to work on the registry server tests when time permits (additional help is always appreciated). Someone else would need to write the tests for clients. LF/OCI may help with a resource to work on the website, but a volunteer to help with that would also be useful.

[rchincha]

Some notes about gingko which is the testing framework we use in conformance tests. It is quite powerful and we may just be using a subset of its features.

https://onsi.github.io/ginkgo/#filtering-specs

https://onsi.github.io/ginkgo/#spec-labels

https://onsi.github.io/ginkgo/#table-specs

https://onsi.github.io/ginkgo/#controlling-ginkgos-output

[sudo-bmitch]

My preference is to use the Go stdlib where possible. My goal is to minimize the number of external libraries and frameworks someone needs to learn to submit a PR to the project.

[briheet]

would like to volunteer

[sudo-bmitch]

Oh, I also wasn't volunteering to redesign the configuration around a configuration file. My plan was to stick with environment variables, maybe CLI parameters, to simplify the integration with GitHub actions and similar CI tooling. Someone else would need to work on that task.

I'm worried this issue is very broad and unlikely to be fully resolved (who's writing the conformance tests for client tools?), particularly not by a single PR. Complex tasks that require a lot of work are being listed as a single bullet point with little detail of the requirements. This may be better of being closed and individual issues or PRs opened under the conformance label (assuming they don't already exist).

As an aside, I may try rewriting it as a standard Go binary instead of with the Go test interface it is using now. That would simplify building it and also allow the conformance test to be tested itself.

[rchincha]

My preference is to use the Go stdlib where possible. My goal is to minimize the number of external libraries and frameworks someone needs to learn to submit a PR to the project.

I have the opposite concern that we may NIH this and end up with the same functionality.

[sudo-bmitch]

Okay, I'll hold off on doing anything pending agreement on these concerns.

[rchincha]

Some more notes ...

If test configuration is passed as env vars, "pull.subtest1.arg2.subarg3=X"

(vs)

as a config file (which can be downloaded,shared,derived,and shared again)

{ "pull": { "subtest1": { "arg2": { "subarg3": X } } }