Open joerawr opened 6 years ago
Could you provide failed node-agent pod logs?
Sure. I'm not sure which one(s) are failed, so here is all five.
tectonic-system node-agent-dj2l2 1/1 Running 4 57d tectonic-system node-agent-fplfv 1/1 Running 2 57d tectonic-system node-agent-mlf6t 1/1 Running 2 57d tectonic-system node-agent-q4q9s 1/1 Running 4 57d tectonic-system node-agent-szlmp 1/1 Running 4 57d
# kubectl logs node-agent-dj2l2 -n tectonic-system
I0509 02:49:41.182278 1 main.go:78] starting node agent, watching node: worker3.preprod.com
I0509 03:00:19.770215 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:31.045639 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:32.132094 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:41.189267 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:42.254500 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:52.132958 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
W0509 04:00:25.107193 1 reflector.go:326] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: watch of *v1.Node ended with: very short watch: github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Unexpected watch close - watch lasted less than a second and no items received
E0509 04:00:26.108665 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dworker3.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
E0509 04:00:27.110001 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dworker3.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
# kubectl logs -n tectonic-system node-agent-fplfv
I0509 02:41:07.031962 1 main.go:78] starting node agent, watching node: vmaster0.preprod.com
I0509 03:00:20.772975 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:30.446394 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:33.649097 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:37.043777 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:43.669826 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:53.686511 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:01:07.044255 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
E0509 04:00:25.145760 1 reflector.go:307] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to watch *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dvmaster0.preprod.com&watch=true: dial tcp 10.3.0.1:443: getsockopt: connection refused
E0509 04:00:26.147005 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dvmaster0.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
E0509 04:00:27.148166 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dvmaster0.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
# kubectl logs -n tectonic-system node-agent-mlf6t
I0509 04:02:50.066717 1 main.go:78] starting node agent, watching node: vmaster1.preprod.com
# kubectl logs -n tectonic-system node-agent-q4q9s
I0509 02:43:48.652562 1 main.go:78] starting node agent, watching node: worker0.preprod.com
I0509 03:00:17.571822 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:28.227165 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:29.115090 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:31.413663 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
# kubectl logs -n tectonic-system node-agent-szlmp
I0509 02:50:01.962067 1 main.go:78] starting node agent, watching node: worker1.preprod.com
I0509 03:00:18.595144 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:27.569860 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
I0509 03:00:31.966333 1 agent.go:175] pulling image: quay.io/coreos/hyperkube:v1.8.7_coreos.0
W0509 04:00:25.047181 1 reflector.go:326] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: watch of *v1.Node ended with: very short watch: github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Unexpected watch close - watch lasted less than a second and no items received
E0509 04:00:26.047631 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dworker1.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
E0509 04:00:27.048119 1 reflector.go:199] github.com/coreos-inc/kube-version-operator/cmd/node-agent/main.go:103: Failed to list *v1.Node: Get https://10.3.0.1:443/api/v1/nodes?fieldSelector=metadata.name%3Dworker1.preprod.com: dial tcp 10.3.0.1:443: getsockopt: connection refused
I didn't find any errors in node-agent logs. Can you please provide output of kubectl describe appversion kubernetes -n tectonic-system
?
I have exactly the same problem, but from Tectonic 1.9.6-tectonic.1 ➝ 1.9.6-tectonic.2.
Firstly, it got stuck on prewarm images and I followed this: https://support.coreos.com/hc/en-us/articles/360006498434-Tectonic-update-stuck-on-pre-warm-cache
Now it is stuck on upgrade node-agent.
The output of the "kubectl describe appversion kubernetes -n tectonic-system" is the following:
$ kubectl describe appversion kubernetes -n tectonic-system
Name: kubernetes
Namespace: tectonic-system
Labels: managed-by-channel-operator=true
Annotations: <none>
API Version: tco.coreos.com/v1
Kind: AppVersion
Metadata:
Cluster Name:
Creation Timestamp: 2018-07-26T13:48:44Z
Generation: 0
Resource Version: 53571333
Self Link: /apis/tco.coreos.com/v1/namespaces/tectonic-system/appversions/kubernetes
UID: 9d0184c8-90da-11e8-ba7e-1a493cb07586
Spec:
Desired Version: 1.9.6+tectonic.2
Paused: false
Status:
Current Version: 1.9.6+tectonic.1
Paused: false
Target Version: 1.9.6+tectonic.2
Task Statuses:
Name: Update pod-checkpointer
Reason:
State: Completed
Type:
Name: Update node-agent
Reason:
State: Running
Type:
Name: Update kube-apiserver
Reason:
State: NotStarted
Type:
Name: Update kube-scheduler
Reason:
State: NotStarted
Type:
Name: Update kube-controller-manager
Reason:
State: NotStarted
Type:
Name: Update kube-proxy
Reason:
State: NotStarted
Type:
Name: Update Node Updater
Reason:
State: NotStarted
Type:
Name: Update tectonic-identity
Reason:
State: NotStarted
Type:
Name: Update kube-dns
Reason:
State: NotStarted
Type:
Name: Update tectonic-console
Reason:
State: NotStarted
Type:
Name: Update tectonic-identity-api
Reason:
State: NotStarted
Type:
Name: Update tectonic-stats-emitter
Reason:
State: NotStarted
Type:
Name: Update tectonic-ingress-controller
Reason:
State: NotStarted
Type:
Name: Update kube-flannel
Reason:
State: NotStarted
Type:
Name: prewarm-container-image-cache
Reason:
State: Completed
Type:
Events: <none>
Issue Report Template
Tectonic Version
Tectonic 1.8.4-tectonic.4 ➝ 1.8.9-tectonic.1
Environment
What hardware/cloud provider/hypervisor is being used with Tectonic?
Bare Metal
Expected Behavior
Upgrade should complete
Actual Behavior
Upgrade loops at Update node-agent
Reproduction Steps
Other Information
Let me know what other logs I can collect to help the upgrade complete.
Feature Request
Environment
What hardware/cloud provider/hypervisor is being used with Tectonic?
Bare Metal Dell servers for the workers and vmware vms for the masters.
Desired Feature
Other Information