search

coreos / tectonic-forum

Apache License 2.0
30 stars 9 forks source link

Upgrade failing for: Tectonic 1.8.4-tectonic.4 => 1.8.9-tectonic.3 #311

Closed TerraTech closed 6 years ago

TerraTech commented 6 years ago

Issue Report Template

Tectonic Version

Tectonic 1.8.4-tectonic.4

Environment

metal

What hardware/cloud provider/hypervisor is being used with Tectonic? Container Linux 1576.5.0 (it was 1800.6.0 but I had to revert due to a torcx/docker issue)

Expected Behavior

Upgrade completes without failure.

Actual Behavior

Upgrade fails in an intermediate upgrade step.

In short, I'm trying to upgrade a Tectonic cluster from: Tectonic 1.8.4-tectonic.4 => 1.8.9-tectonic.3

The intermediate upgrade step where it is failing: Kubernetes 1.8.4+tectonic.2 => 1.8.7+tectonic.2

The error that is occurring is within the kube-version-operator:

E0812 05:22:28.024847       1 kube_version_operator.go:121] Error updating cluster: error updating to 1.8.7+tectonic.1: Failed update of component: daemonset/kube-system/kube-apiserver due to: flag already set to unknown values

Does anyone know exactly what this error means and how to fix/correct the problem? I'm not finding anything truly meaningful in the logs that points to exactly what the problem is. Nor have I found the source code for this operator so I could see exactly what is triggering it.

At this point, I'm probably going to destroy the whole cluster and re-terraform with 1.9.6-tectonic.1 mostly because I've spent too much time chasing this glitch down its winding rabbit hole. Overall, I hope that whomever can shed light on this will help out others that hit it in the future.

Thanks!

robszumski commented 6 years ago

What changes have you manually made to the API server object? It's conflicting with something that is managed via the upgrade.

TerraTech commented 6 years ago

The only tweak I've made to kube-apiserver.yaml is adding the following command line switch:

containers:
- name: kube-apiserver
  command:
  [...]
  - --kubelet-preferred-address-types=InternalIP

Would that addition be enough to trip up the upgrade processing? If not, please let me know what you need to see and an example command to provide that info to you.

Thank you for taking a look!

robszumski commented 6 years ago

Yes, that would. Set it back to --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname and see if that solves it.

TerraTech commented 6 years ago

Thanks @robszumski setting it to what you recommended allowed it to go past that point. Much appreciated!