search

opencontainers / distribution-spec

OCI Distribution Specification
https://opencontainers.org
Apache License 2.0
811 stars 202 forks source link

Conformance GitHub action sometimes panics #369

Closed SuperSandro2000 closed 1 year ago

SuperSandro2000 commented 1 year ago

We are running the conformance github action of part of our registry CI. Sometimes it fails randomly and succeeds with the next unrelated commit again. We are running the conformance tests in our own CI, too, where this is not happening. I am not familiar with ginkgo and currently I don't have a good idea how to debug this.

•! Panic [0.000 seconds]
OCI Distribution Conformance Tests
/go/src/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Pull
  /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:13
    Setup
    /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:17
      Populate registry with test blob [It]
      /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:18

      Test Panicked
      runtime error: invalid memory address or nil pointer dereference
      /usr/local/go/src/runtime/panic.go:199

      Full Stack Trace
      github.com/bloodorangeio/reggie.(*Response).GetRelativeLocation(0x0, 0xc00001e060, 0x0)
        /go/pkg/mod/github.com/bloodorangeio/reggie@v0.5.0/response.go:22 +0x3a
      github.com/opencontainers/distribution-spec/conformance.glob..func1.1.1.1()
        /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:[23](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:24) +0xbc
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00008e9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/leafnodes/runner.go:113 +0xb8
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc00008e9c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/leafnodes/runner.go:64 +0xcf
      github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00009dce0, 0x9eb060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/leafnodes/it_node.go:[26](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:27) +0x64
      github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc000346000, 0x0, 0x9eb060, 0xc000070b80)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/spec/spec.go:215 +0x5b5
      github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc000346000, 0x9eb060, 0xc000070b80)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/spec/spec.go:138 +0x101
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0000b6840, 0xc000346000, 0x0)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/specrunner/spec_runner.go:200 +0x10f
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0000b6840, 0x1)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/specrunner/spec_runner.go:170 +0x120
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0000b6840, 0xc0002fca78)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/specrunner/spec_runner.go:66 +0x117
      github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0000f59d0, 0x7f9a[27](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:28)6e72a8, 0xc0000c6600, 0x943418, 0x22, 0xc000170690, 0x3, 0x3, 0x9f58a0, 0xc000070b80, ...)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/internal/suite/suite.go:79 +0x586
      github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x9eb720, 0xc0000c6600, 0x943418, 0x22, 0xc000071900, 0x3, 0x4, 0x4)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/ginkgo_dsl.go:2[29](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:30) +0x217
      github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters(0x9eb720, 0xc0000c6600, 0x94[34](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:35)18, 0x22, 0xc0000fff40, 0x2, 0x2, 0x3651cab1)
        /go/pkg/mod/github.com/onsi/ginkgo@v1.15.2/ginkgo_dsl.go:217 +0xad
      github.com/opencontainers/distribution-spec/conformance.TestConformance(0xc0000c6600)
        /go/src/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:21 +0x18b
      testing.tRunner(0xc0000c6600, 0x9627c8)
        /usr/local/go/src/testing/testing.go:909 +0xc9
      created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:960 +0x[35](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:36)0

https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220

sudo-bmitch commented 1 year ago

Looks like the POST to blob/uploads failed, and we don't have error handling on the call in the currently released branch. There is error handling on the repo main branch. (See #356.)

The intermittent failure points to a race condition, likely starting the conformance test before the registry is fully started. You may want to use some kind of wait-for-it like tool to pause until the server is running.

SuperSandro2000 commented 1 year ago

I added a wait condition on our healthcheck endpoint and that improved things.