opencontainers / distribution-spec

OCI Distribution Specification
https://opencontainers.org
Apache License 2.0
811 stars 202 forks source link

conformance: initial commit for references test #375

Closed rchincha closed 1 year ago

rchincha commented 1 year ago

Based off of 01_pull_test.go

Fixes #370

Signed-off-by: Ramkumar Chinchani rchincha@cisco.com

rchincha commented 1 year ago

Testing against zot ...

      you have skipped this test; if this is an error, check your environment variable settings:
        OCI_TEST_CONTENT_DISCOVERY=0
        OCI_TEST_CONTENT_MANAGEMENT=1
        OCI_REFERRERS=1
        OCI_TEST_PULL=0
        OCI_TEST_PUSH=0

      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:408
------------------------------
•••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 23 of 78 Specs in 0.085 seconds
SUCCESS! -- 23 Passed | 0 Failed | 0 Pending | 55 Skipped
jdolitsky commented 1 year ago

can you rebase?

rchincha commented 1 year ago

@jdolitsky @sudo-bmitch @sajayantony Some eyes on this pls.

FYI, top of zot/main passes.

rchincha commented 1 year ago

Couple minor updates. Does Zot still pass after the latest changes?

Yes, top of zot/main does.

      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:491
------------------------------
••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.152 seconds
SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped
rchincha commented 1 year ago

bin/zot-linux-amd64 serve examples/config-conformance.json ^ to test

Made the warning code/output a little "nicer"

rchincha commented 1 year ago

As a data point, top of zot/main passes

••••••••••••••••••••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml

Ran 81 of 86 Specs in 0.255 seconds SUCCESS! -- 81 Passed | 0 Failed | 0 Pending | 5 Skipped

rchincha commented 1 year ago

Instrumented zot warns like so,


      /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/setup.go:490
------------------------------
••••••••••••••••••••••••••••
    WARNING: filtering by artifact-type is not implemented                                                                                                                                                                                                                                                                                              /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/05_referrers.go:240
rchincha commented 1 year ago

https://github.com/opencontainers/image-spec/pull/1077 ^ tests for this in a separate PR after it lands