search

openshift / hypershift

Hyperscale OpenShift - clusters with hosted control planes
https://hypershift-docs.netlify.app
Apache License 2.0
434 stars 321 forks source link

Ignition server failure investigation (properly handling not ready cache/timeout analysis) #343

Closed relyt0925 closed 3 years ago

relyt0925 commented 3 years ago 
2021-07-09T01:36:37.594Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-6scwx\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:36:57 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:36:57 Invalid token value
2021/07/09 01:37:36 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:38:26 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:38:35 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:19 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:21 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:22 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:42:05.271Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:42:06.668Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-f68mc\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
relyt0925 commented 3 years ago 
2021-07-09T01:42:06.668Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-f68mc\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:42:20 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:42:22 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:46:59.273Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:47:10.618Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:47:29 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:47:55 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:03 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:26 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:27 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:28 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:53:02.030Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:53:13.386Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
relyt0925 commented 3 years ago

Logs of a run not never see successful IgnitionProvider generated payload which I would expect to see to populate cache

Tylers-MacBook-Pro:~ tylerlisowski$ kubectl logs -n master-cruiser7 ignition-server-b6dc5d677-pljhc -f
2021/07/09 01:29:00 Listening on 0.0.0.0:9090
I0709 01:29:01.423635       1 request.go:655] Throttling request took 1.030730096s, request: GET:https://172.21.0.1:443/apis/etcd.database.coreos.com/v1beta2?timeout=32s
2021-07-09T01:29:03.369Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-07-09T01:29:03.380Z    INFO    Setting up token controller SetupWithManager    {"ns": "master-cruiser7"}
2021-07-09T01:29:03.380Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-07-09T01:29:03.380Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-07-09T01:29:03.382Z    INFO    controller-runtime.manager.controller.secret    Starting EventSource    {"reconciler group": "", "reconciler kind": "Secret", "source": "kind source: /, Kind="}
2021-07-09T01:29:03.389Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
2021-07-09T01:29:03.588Z    INFO    controller-runtime.manager.controller.secret    Starting Controller {"reconciler group": "", "reconciler kind": "Secret"}
2021-07-09T01:29:03.588Z    INFO    controller-runtime.manager.controller.secret    Starting workers    {"reconciler group": "", "reconciler kind": "Secret", "worker count": 1}
2021-07-09T01:29:03.589Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:18.524Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:18.529Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:19.886Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-94jrs\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:19.896Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:31.219Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:31.240Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:32.561Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-kcrrb\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:32.601Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:43.930Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:44.011Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:29:55.340Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:29:55.500Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:06.824Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:07.145Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:08.498Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-hl8x8\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:09.138Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:20.451Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:21.732Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:23.056Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-jljrk\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:25.616Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:26.936Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-rnbth\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:32.057Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:30:43.402Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:30:53.645Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:31:04.958Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:31:25.440Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:31:36.797Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:32:17.758Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:32:29.078Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021-07-09T01:33:51.000Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:33:52.387Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-r89gq\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:35:09 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:36:36.228Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:36:37.594Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-6scwx\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:36:57 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:36:57 Invalid token value
2021/07/09 01:37:36 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:38:26 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:38:35 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:19 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:21 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:40:22 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:42:05.271Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:42:06.668Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: Pod \"machine-config-server-f68mc\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:42:20 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:42:22 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:46:59.273Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:47:10.618Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 01:47:29 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:47:55 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:03 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:26 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:27 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 01:49:28 User Agent: curl/7.64.1. Requested: /ignition
2021-07-09T01:53:02.030Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T01:53:13.386Z    ERROR   controller-runtime.manager.controller.secret    Reconciler error    {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7", "error": "error getting ignition payload: failed to get payload from machine config server Pod: timed out waiting for the condition"}
github.com/go-logr/zapr.(*zapLogger).Error
    /hypershift/vendor/github.com/go-logr/zapr/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:301
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
    /hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /hypershift/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99
2021/07/09 02:00:22 User Agent: curl/7.64.1. Requested: /ignition
2021/07/09 02:00:35 User Agent: curl/7.64.1. Requested: /ignition
relyt0925 commented 3 years ago

To fix currently: I bumped timeout and now it appears to work

Tylers-MacBook-Pro:armada-hypershift-operator tylerlisowski$ kubectl logs -n master-cruiser7 ignition-server-5b677857d4-c4jlz
2021/07/09 02:33:24 Listening on 0.0.0.0:9090
I0709 02:33:25.560018       1 request.go:655] Throttling request took 1.042007962s, request: GET:https://172.21.0.1:443/apis/operator.openshift.io/v1?timeout=32s
2021-07-09T02:33:27.500Z    INFO    controller-runtime.metrics  metrics server is starting to listen    {"addr": ":8080"}
2021-07-09T02:33:27.504Z    INFO    Setting up token controller SetupWithManager    {"ns": "master-cruiser7"}
2021-07-09T02:33:27.505Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-07-09T02:33:27.505Z    INFO    controller-runtime.injectors-warning    Injectors are deprecated, and will be removed in v0.10.x
2021-07-09T02:33:27.505Z    INFO    controller-runtime.manager  starting metrics server {"path": "/metrics"}
2021-07-09T02:33:27.505Z    INFO    controller-runtime.manager.controller.secret    Starting EventSource    {"reconciler group": "", "reconciler kind": "Secret", "source": "kind source: /, Kind="}
2021-07-09T02:33:27.807Z    INFO    controller-runtime.manager.controller.secret    Starting Controller {"reconciler group": "", "reconciler kind": "Secret"}
2021-07-09T02:33:27.807Z    INFO    controller-runtime.manager.controller.secret    Starting workers    {"reconciler group": "", "reconciler kind": "Secret", "worker count": 1}
2021-07-09T02:33:27.812Z    INFO    controller-runtime.manager.controller.secret    Reconciling {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
2021-07-09T02:33:53.968Z    INFO    controller-runtime.manager.controller.secret    IgnitionProvider generated payload  {"reconciler group": "", "reconciler kind": "Secret", "name": "token-cruiser7-1-4.8.0-fc.8", "namespace": "master-cruiser7"}
enxebre commented 3 years ago

/assign

relyt0925 commented 3 years ago

Just to note: The two main things here are getting a proper response returned when the cache is not ready that gets the machine calling to ignition to retry and not enter emergency mode

The other is adjusting the timeout or maybe strategy (likely timeout) to ensure there is enough time under varying load/network conditions to properly run and execute the generation logic.

relyt0925 commented 3 years ago

this is fixed