search

deckhouse / k8s-image-availability-exporter

Alert if an image used in Kubernetes cannot be pulled from container registry
Apache License 2.0
215 stars 26 forks source link

Keychain panic fix #64

Closed zuzzas closed 1 year ago

zuzzas commented 1 year ago
Log ``` time="2022-12-23T13:53:46Z" level=info msg="Waiting for cache sync" [101/1856] W1223 13:53:46.545466 1 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob W1223 13:53:46.558175 1 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob I1223 13:54:03.551100 1 trace.go:205] Trace[911902081]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.21.0/tools/cache/reflector.go:167 (23-Dec-2022 13:53:46.523) (total time: 17027ms): Trace[911902081]: ---"Objects listed" 17012ms (13:54:03.536) Trace[911902081]: [17.027337457s] [17.027337457s] END time="2022-12-23T13:54:03Z" level=info msg="Caches populated successfully" time="2022-12-23T13:54:19Z" level=error msg="Head \"https://docker-remote-deckhouse-io.art.lmru.tech/v2/deckhouse/fe/manifests/63071429aa21699ea47d80c24215c4b8bf9b62b763962f8a52bcf4ef-1662400526099\": context de adline exceeded" availability_mode=unknown_error image_name="docker-remote-deckhouse-io.art.lmru.tech/deckhouse/fe:63071429aa21699ea47d80c24215c4b8bf9b62b763962f8a52bcf4ef-1662400526099" time="2022-12-23T13:54:34Z" level=error msg="Head \"https://docker-remote-deckhouse-io.art.lmru.tech/v2/deckhouse/fe/manifests/1b4fdea2e05bddf21d91e579268b381d461e5c1c4dbb768d7f5ea54a-1655994633660\": context de adline exceeded" availability_mode=unknown_error image_name="docker-remote-deckhouse-io.art.lmru.tech/deckhouse/fe:1b4fdea2e05bddf21d91e579268b381d461e5c1c4dbb768d7f5ea54a-1655994633660" time="2022-12-23T13:54:49Z" level=error msg="Head \"https://docker-remote-deckhouse-io.art.lmru.tech/v2/deckhouse/fe/manifests/e7189bc43b9bf6cc5cb8fe37c4c889117a3281abc7c2e99f903f6132-1655994060354\": context de adline exceeded" availability_mode=unknown_error image_name="docker-remote-deckhouse-io.art.lmru.tech/deckhouse/fe:e7189bc43b9bf6cc5cb8fe37c4c889117a3281abc7c2e99f903f6132-1655994060354" E1223 13:55:01.598408 1 runtime.go:78] Observed a panic: runtime.boundsError{x:3, y:3, signed:true, code:0x0} (runtime error: index out of range [3] with length 3) goroutine 1 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic({0x15843c0, 0xc002fbf590}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x85 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00007c250}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x75 panic({0x15843c0, 0xc002fbf590}) /usr/local/go/src/runtime/panic.go:1038 +0x215 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.lazyProvider.Authorization({0xc01a99ae40, {0xc003fdb7c0, 0x0}}) /src/pkg/registry_checker/registry_authn.go:22 +0x1ec github.com/google/go-containerregistry/pkg/v1/remote/transport.(*bearerTransport).refresh(0xc0001729c0, {0x1847e80, 0xc02148f8c0}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/bearer.go:126 +0x4f github.com/google/go-containerregistry/pkg/v1/remote/transport.NewWithContext({0x1847e80, 0xc02148f8c0}, {0x70, {0xc000865d80, 0x181fca0}}, {0x181f840, 0xc01ab3a5a0}, {0x181d620, 0xc01a9e3440}, {0xc01a9ccb70, .. .}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/transport.go:96 +0x4d3 github.com/google/go-containerregistry/pkg/v1/remote.makeFetcher({0x1862258, 0xc00c367ea0}, 0xc0028dda40) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:220 +0x165 github.com/google/go-containerregistry/pkg/v1/remote.Head({0x1862258, 0xc00c367ea0}, {0xc0206316f8, 0x3, 0x3}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:106 +0x2a9 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.check({0x1862258, 0xc00c367ea0}, 0xc01a99ae40, 0xc0003fc280) /src/pkg/registry_checker/checker.go:318 +0x47e github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability.func1() /src/pkg/registry_checker/checker.go:267 +0x39 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0206317d8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:211 +0x72 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x3b9aca00, 0x4000000000000000, 0x0, 0x2, 0x0}, 0xc0206319a8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:399 +0x5f github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability(0xc0002b81c0, 0x165c1c3, {0xc000865d80, 0x13d8aa0}, 0xc01a99ae40) /src/pkg/registry_checker/checker.go:261 +0x108 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).Check(0xc0002b81c0, {0xc000865d80, 0x40}) /src/pkg/registry_checker/checker.go:252 +0xef github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).popCheckPush(0xc000514fa0, 0x0, 0x32) /src/pkg/store/image_store.go:184 +0xed github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).Check(0xc000514fa0) /src/pkg/store/image_store.go:162 +0x87 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.RegistryChecker.Tick(...) /src/pkg/registry_checker/checker.go:199 main.main.func2() /src/main.go:93 +0x5d k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fd27a2930e0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:155 +0x67 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x43f10e, {0x181e7a0, 0xc020597260}, 0x1, 0xc0000a6ea0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:156 +0xb6 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x23cc601, 0xdf8475800, 0x0, 0x60, 0x1707d38) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:133 +0x89 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:90 main.main() /src/main.go:92 +0x708 E1223 13:55:01.598529 1 runtime.go:78] Observed a panic: runtime.boundsError{x:3, y:3, signed:true, code:0x0} (runtime error: index out of range [3] with length 3) goroutine 1 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic({0x15843c0, 0xc002fbf590}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x85 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4d9014}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x75 panic({0x15843c0, 0xc002fbf590}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00007c250}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:55 +0xd8 panic({0x15843c0, 0xc002fbf590}) /usr/local/go/src/runtime/panic.go:1038 +0x215 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.lazyProvider.Authorization({0xc01a99ae40, {0xc003fdb7c0, 0x0}}) /src/pkg/registry_checker/registry_authn.go:22 +0x1ec github.com/google/go-containerregistry/pkg/v1/remote/transport.(*bearerTransport).refresh(0xc0001729c0, {0x1847e80, 0xc02148f8c0}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/bearer.go:126 +0x4f github.com/google/go-containerregistry/pkg/v1/remote/transport.NewWithContext({0x1847e80, 0xc02148f8c0}, {0x70, {0xc000865d80, 0x181fca0}}, {0x181f840, 0xc01ab3a5a0}, {0x181d620, 0xc01a9e3440}, {0xc01a9ccb70, .. .}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/transport.go:96 +0x4d3 github.com/google/go-containerregistry/pkg/v1/remote.makeFetcher({0x1862258, 0xc00c367ea0}, 0xc0028dda40) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:220 +0x165 github.com/google/go-containerregistry/pkg/v1/remote.Head({0x1862258, 0xc00c367ea0}, {0xc0206316f8, 0x3, 0x3}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:106 +0x2a9 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.check({0x1862258, 0xc00c367ea0}, 0xc01a99ae40, 0xc0003fc280) /src/pkg/registry_checker/checker.go:318 +0x47e github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability.func1() /src/pkg/registry_checker/checker.go:267 +0x39 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0206317d8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:211 +0x72 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x3b9aca00, 0x4000000000000000, 0x0, 0x2, 0x0}, 0xc0206319a8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:399 +0x5f github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability(0xc0002b81c0, 0x165c1c3, {0xc000865d80, 0x13d8aa0}, 0xc01a99ae40) /src/pkg/registry_checker/checker.go:261 +0x108 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).Check(0xc0002b81c0, {0xc000865d80, 0x40}) /src/pkg/registry_checker/checker.go:252 +0xef github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).popCheckPush(0xc000514fa0, 0x0, 0x32) /src/pkg/store/image_store.go:184 +0xed github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).Check(0xc000514fa0) /src/pkg/store/image_store.go:162 +0x87 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.RegistryChecker.Tick(...) /src/pkg/registry_checker/checker.go:199 main.main.func2() /src/main.go:93 +0x5d k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fd27a2930e0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:155 +0x67 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x43f10e, {0x181e7a0, 0xc020597260}, 0x1, 0xc0000a6ea0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:156 +0xb6 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x23cc601, 0xdf8475800, 0x0, 0x60, 0x1707d38) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:133 +0x89 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:90 main.main() /src/main.go:92 +0x708 panic: runtime error: index out of range [3] with length 3 [recovered] panic: runtime error: index out of range [3] with length 3 [recovered] panic: runtime error: index out of range [3] with length 3 goroutine 1 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4d9014}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:55 +0xd8 panic({0x15843c0, 0xc002fbf590}) /usr/local/go/src/runtime/panic.go:1038 +0x215 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00007c250}) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:55 +0xd8 panic({0x15843c0, 0xc002fbf590}) /usr/local/go/src/runtime/panic.go:1038 +0x215 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.lazyProvider.Authorization({0xc01a99ae40, {0xc003fdb7c0, 0x0}}) /src/pkg/registry_checker/registry_authn.go:22 +0x1ec github.com/google/go-containerregistry/pkg/v1/remote/transport.(*bearerTransport).refresh(0xc0001729c0, {0x1847e80, 0xc02148f8c0}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/bearer.go:126 +0x4f github.com/google/go-containerregistry/pkg/v1/remote/transport.NewWithContext({0x1847e80, 0xc02148f8c0}, {0x70, {0xc000865d80, 0x181fca0}}, {0x181f840, 0xc01ab3a5a0}, {0x181d620, 0xc01a9e3440}, {0xc01a9ccb70, .. .}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/transport/transport.go:96 +0x4d3 github.com/google/go-containerregistry/pkg/v1/remote.makeFetcher({0x1862258, 0xc00c367ea0}, 0xc0028dda40) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:220 +0x165 github.com/google/go-containerregistry/pkg/v1/remote.Head({0x1862258, 0xc00c367ea0}, {0xc0206316f8, 0x3, 0x3}) /go/pkg/mod/github.com/google/go-containerregistry@v0.6.0/pkg/v1/remote/descriptor.go:106 +0x2a9 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.check({0x1862258, 0xc00c367ea0}, 0xc01a99ae40, 0xc0003fc280) /src/pkg/registry_checker/checker.go:318 +0x47e github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability.func1() /src/pkg/registry_checker/checker.go:267 +0x39 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0xc0206317d8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:211 +0x72 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x3b9aca00, 0x4000000000000000, 0x0, 0x2, 0x0}, 0xc0206319a8) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:399 +0x5f github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).checkImageAvailability(0xc0002b81c0, 0x165c1c3, {0xc000865d80, 0x13d8aa0}, 0xc01a99ae40) /src/pkg/registry_checker/checker.go:261 +0x108 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.(*RegistryChecker).Check(0xc0002b81c0, {0xc000865d80, 0x40}) /src/pkg/registry_checker/checker.go:252 +0xef github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).popCheckPush(0xc000514fa0, 0x0, 0x32) /src/pkg/store/image_store.go:184 +0xed github.com/flant/k8s-image-availability-exporter/pkg/store.(*ImageStore).Check(0xc000514fa0) /src/pkg/store/image_store.go:162 +0x87 github.com/flant/k8s-image-availability-exporter/pkg/registry_checker.RegistryChecker.Tick(...) /src/pkg/registry_checker/checker.go:199 main.main.func2() /src/main.go:93 +0x5d k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fd27a2930e0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:155 +0x67 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x43f10e, {0x181e7a0, 0xc020597260}, 0x1, 0xc0000a6ea0) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:156 +0xb6 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x23cc601, 0xdf8475800, 0x0, 0x60, 0x1707d38) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:133 +0x89 k8s.io/apimachinery/pkg/util/wait.Until(...) /go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/wait/wait.go:90 main.main() /src/main.go:92 +0x708 ```
codecov[bot] commented 1 year ago

Codecov Report

Base: 19.12% // Head: 19.00% // Decreases project coverage by -0.12% :warning:

Coverage data is based on head (ffd2d84) compared to base (d6c1709). Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #64 +/- ## ========================================== - Coverage 19.12% 19.00% -0.13% ========================================== Files 5 5 Lines 596 600 +4 ========================================== Hits 114 114 - Misses 474 478 +4 Partials 8 8 ``` | [Impacted Files](https://codecov.io/gh/deckhouse/k8s-image-availability-exporter/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deckhouse) | Coverage Δ | | |---|---|---| | [pkg/registry\_checker/registry\_authn.go](https://codecov.io/gh/deckhouse/k8s-image-availability-exporter/pull/64/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deckhouse#diff-cGtnL3JlZ2lzdHJ5X2NoZWNrZXIvcmVnaXN0cnlfYXV0aG4uZ28=) | `0.00% <0.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deckhouse). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deckhouse)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.