Closed msheldyakov closed 3 years ago
Attempt to reverse update with (no luck): 1) oc adm upgrade --force to 4.7.0-0.okd-2021-03-21-094146 and complete fresh reinstall of some nodes. 2) Manual disable of cluster-version-operator and setting RELEASE_VERSION on machine-config-operator and restoring machine-config-operator-images/machine-config-osimageurl configmaps from backup.
Do you have a must-gather after a failed upgrade?
Currently no. Possibly later on another cluster.
Hello, we've been hit by the same issue yesterday.
Upgrading a UPI cluster on BareMetal (6 nodes) from 4.7.0-0.okd-2021-03-21-094146 to 4.7.0-0.okd-2021-03-28-152009 as well.
I've done a must-gather archive but I'm wondering if there's an easy way to kind of "anonymize" hostnames and such ? There's like 775MB of data (65MB gzipped).
Thanks !
W0329 17:26:34.389625 2308 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a failed: error: unable to load --registry-config: error occurred while trying to unmarshal json
This is odd - your kubelet pull secret is invalid. Do you use "fake" pull secret?
W0329 17:26:34.389625 2308 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a failed: error: unable to load --registry-config: error occurred while trying to unmarshal json
This is odd - your kubelet pull secret is invalid. Do you use "fake" pull secret?
Yes, fake.
pullSecret: '{"auths":{"fake":{"auth": "bar"}}}'
This cluster was installed as 4.6, went through several updates around 4.6 and then 4.7. Failed only at 4.7.0-0.okd-2021-03-28-152009
If your cluster is still up, can you change
/var/lib/kubelet/config.json
on one of the SchedulingDisabled nodes to
See Below...
and reboot it?
If your cluster is still up
No, it has already been restored from a backup.
If you try again, try changing the config.json on your nodes as indicated above
I got the same error as the initial post above. I have an UPI bare metal installation with 3 master and 6 worker nodes.
Both machine config pools are degraded with the following error:
Node master2 is reporting: "error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-093307771: chcon: cannot access '/run/mco-machine-os-content/os-content-093307771': No such file or directory\n: exit status 1"
I also found the following error in the machine-config pod of the worker node which the cluster tries to upgrade:
I0330 12:53:11.601667 1488282 run.go:18] Running: nice -- ionice -c 3 podman cp bc16533866ca77f9cd06bdf343f507c0581341b42ef768d450df1ab272ac22d3:/ /run/mco-machine-os-content/os-content-395424672
Error: 2 errors occurred:
* error copying to host: error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/run/mco-machine-os-content", rootPrefix:"/run/mco-machine-os-content", Directory:"/", preservedDirectory:"/run/mco-machine-os-content", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil)}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(0xc000621400), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000621410), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: put: error setting extended attributes on "/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": error setting value of extended attribute "user.Zif.MdChecksum[1616584500]" on "/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": operation not supported
* error copying from container: error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", rootPrefix:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", Directory:"/", preservedDirectory:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", Globs:[]string{"/"}, preservedGlobs:[]string{"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged/."}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(0xc0005053f0), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000505400), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil)}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: get: "/"("/"): error copying /extensions/okd/checkpolicy-3.1-3.fc33.x86_64.rpm: write bulk-writer: broken pipe
W0330 12:53:13.125688 1488282 run.go:44] nice failed: running nice -- ionice -c 3 podman cp bc16533866ca77f9cd06bdf343f507c0581341b42ef768d450df1ab272ac22d3:/ /run/mco-machine-os-content/os-content-395424672 failed: Error: 2 errors occurred:
* error copying to host: error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/run/mco-machine-os-content", rootPrefix:"/run/mco-machine-os-content", Directory:"/", preservedDirectory:"/run/mco-machine-os-content", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil)}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(0xc000621400), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000621410), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: put: error setting extended attributes on "/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": error setting value of extended attribute "user.Zif.MdChecksum[1616584500]" on "/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": operation not supported
* error copying from container: error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", rootPrefix:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", Directory:"/", preservedDirectory:"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged", Globs:[]string{"/"}, preservedGlobs:[]string{"/var/lib/containers/storage/overlay/d3c885a10d5ad2ce003363349c76849d48eefc1a9b1c513e24c9b46e0afe5043/merged/."}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(0xc0005053f0), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000505400), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil)}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: get: "/"("/"): error copying /extensions/okd/checkpolicy-3.1-3.fc33.x86_64.rpm: write bulk-writer: broken pipe
: exit status 125; retrying...
Raw log file: machine-config-daemon.txt
I have a must gather of the cluster which is 34MB compressed. I don't want to share it publicly but I am willing to share it with the OKD team members who want to analyze it.
Actually, use
{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}
instead
The issue is that the example fake pull secret in the documentation is bad and during the upgrade oc cannot parse it properly...
The "auth" section needs to be the base64 encoding of "\<id>:\<passwd>"
for example:
echo "id:pass" | base64
aWQ6cGFzcwo=
cat /var/lib/kubelet/config.json
{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}
workaround by @fortinj66 works.. we have this issue from
4.7.0-0.okd-2021-03-21-094146
to 4.7.0-0.okd-2021-04-11-124433
I think that this breaks in minor updates is not very good experience, and it worked before with {"auths":{"fake":{"auth": "bar"}}}
perfectly
It is a minor release step for okd, but a major release step for the underlying Podman version (2.x -> 3.0).
FWIW, I am seeing the same symptoms as listed at the top of this issue with a clean UPI install.
This is using FCOS 33.20210328.3.0
The prominent error is:
Apr 25 12:21:55 okd4-bootstrap.dc2.clg.lab release-image-download.sh[1869]: Error: 2 errors occurred:
Apr 25 12:21:55 okd4-bootstrap.dc2.clg.lab release-image-download.sh[1869]: * error copying to host: error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/run/mco-machine-os-content", rootPrefix:"/run/mco-machine-os-content", Directory:"/", preservedDirectory:"/run/mco-machine-os-content", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(0xc000527be0), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc000527bf0), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, Rename:map[string]string{"/":"os-content-657866381"}}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: put: error setting extended attributes on "/os-content-657866381/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": error setting value of extended attribute "user.Zif.MdChecksum[1616584500]" on "/os-content-657866381/extensions/okd/NetworkManager-ovs-1.26.6-1.fc33.x86_64.rpm": operation not supported
Apr 25 12:21:55 okd4-bootstrap.dc2.clg.lab release-image-download.sh[1869]: * error copying from container: error during bulk transfer for copier.request{Request:"GET", Root:"/", preservedRoot:"/var/lib/containers/storage/overlay/e12d97851b8676322e815a03b18e91aced64065369c2349c7194651aa306bb69/merged", rootPrefix:"/var/lib/containers/storage/overlay/e12d97851b8676322e815a03b18e91aced64065369c2349c7194651aa306bb69/merged", Directory:"/", preservedDirectory:"/var/lib/containers/storage/overlay/e12d97851b8676322e815a03b18e91aced64065369c2349c7194651aa306bb69/merged", Globs:[]string{"/"}, preservedGlobs:[]string{"/var/lib/containers/storage/overlay/e12d97851b8676322e815a03b18e91aced64065369c2349c7194651aa306bb69/merged/."}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string{"dev", "proc", "sys"}, ExpandArchives:false, ChownDirs:(*idtools.IDPair)(0xc00010f570), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(0xc00010f580), ChmodFiles:(*os.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:true, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*os.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*os.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*os.FileMode)(nil), StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:false, NoOverwriteDirNonDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*os.FileMode)(nil)}}: copier: get: "/"("/"): error copying /extensions/okd/checkpolicy-3.1-3.fc33.x86_64.rpm: write bulk-writer: broken pipe
Apr 25 12:21:55 okd4-bootstrap.dc2.clg.lab release-image-download.sh[1869]: W0425 12:21:55.438527 1869 run.go:44] nice failed: running nice -- ionice -c 3 podman cp 9dc403fa1ced5073a4195fdb8d3884ab47c381e7e9fc2e2b7caa856986bac988:/ /run/mco-machine-os-content/os-content-657866381 failed: Error: 2 errors occurred:
I dropped my initial FCOS version back to 33.20210104.3.0
and now it seems to be working. I'll update when the install completes, of fails.
Successful UPI install booting from version 33.20210104.3.0
Should be resolved in 4.7.0-0.okd-2021-06-13-090745
Update from 4.7.0-0.okd-2021-03-21-094146 to 4.7.0-0.okd-2021-03-28-152009 UPI
All (worker/master) machine pools in degraded state: Node hw-control-okd-master-01.domain is reporting: "error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-681716591: chcon: cannot access '/run/mco-machine-os-content/os-content-681716591': No such file or directory\n: exit status 1"
On the node side, i saw os-content directory for a few seconds:
Log from machine-config-daemon-jv8gs `I0329 17:22:37.519626 2308 start.go:108] Version: machine-config-daemon-4.6.0-202006240615.p0-665-g2e8f00c4 (2e8f00c41266630862b66dd47730e52e94d794b5) I0329 17:22:37.522735 2308 start.go:121] Calling chroot("/rootfs") I0329 17:22:37.522894 2308 rpm-ostree.go:261] Running captured: rpm-ostree status --json I0329 17:22:38.779075 2308 daemon.go:224] Booted osImageURL: quay.io/openshift/okd-content@sha256:1fc150fd7b47122e3ff344839cd9b2f1085024beb279f6c5eb5c3c35fbbc9215 () I0329 17:22:38.875045 2308 daemon.go:231] Installed Ignition binary version: 2.9.0 I0329 17:22:38.886274 2308 start.go:97] Copied self to /run/bin/machine-config-daemon on host I0329 17:22:38.888803 2308 metrics.go:105] Registering Prometheus metrics I0329 17:22:38.888859 2308 metrics.go:110] Starting metrics listener on 127.0.0.1:8797 I0329 17:22:38.889775 2308 update.go:1942] Starting to manage node: hw-control-okd-master-01.tutu.ru I0329 17:22:38.892063 2308 rpm-ostree.go:261] Running captured: rpm-ostree status I0329 17:22:38.895240 2308 daemon.go:679] Detected a new login session: New session 1 of user core. I0329 17:22:38.895252 2308 daemon.go:680] Login access is discouraged! Applying annotation: machineconfiguration.openshift.io/ssh I0329 17:22:39.249394 2308 daemon.go:863] State: idle Deployments:
pivot://quay.io/openshift/okd-content@sha256:1fc150fd7b47122e3ff344839cd9b2f1085024beb279f6c5eb5c3c35fbbc9215 CustomOrigin: Managed by machine-config-operator Timestamp: 2021-03-20T17:21:34Z LayeredPackages: NetworkManager-ovs glusterfs glusterfs-fuse qemu-guest-agent
ostree://fedora:fedora/x86_64/coreos/stable Version: 33.20210104.3.0 (2021-01-18T11:30:26Z) Commit: 085622ea1fbe723226fe09ddd7388d3d39e4b43ec6bb828b9a9aaa842a057287 GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31 I0329 17:22:39.249412 2308 rpm-ostree.go:261] Running captured: journalctl --list-boots I0329 17:22:39.255709 2308 daemon.go:870] journalctl --list-boots: -1 fc4b9a97810741f4a839c2eff697315c Mon 2021-03-29 17:17:21 UTC—Mon 2021-03-29 17:20:20 UTC 0 9e2ce8d613c84ece930668746644cafb Mon 2021-03-29 17:21:16 UTC—Mon 2021-03-29 17:22:39 UTC I0329 17:22:39.255723 2308 rpm-ostree.go:261] Running captured: systemctl list-units --state=failed --no-legend I0329 17:22:39.263598 2308 daemon.go:883] systemctl --failed: ● coreos-reset-stub-resolv-selinux-context.path loaded failed failed Monitor /run/systemd/resolve/stub-resolv.conf to fixup SELinux context. ● coreos-reset-stub-resolv-selinux-context.service loaded failed failed Fixup SELinux context on /run/systemd/resolve/stub-resolv.conf
I0329 17:22:39.263615 2308 daemon.go:617] Starting MachineConfigDaemon I0329 17:22:39.263744 2308 daemon.go:624] Enabling Kubelet Healthz Monitor I0329 17:23:08.890763 2308 trace.go:205] Trace[646203300]: "Reflector ListAndWatch" name:github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101 (29-Mar-2021 17:22:38.889) (total time: 30000ms): Trace[646203300]: [30.000727987s] [30.000727987s] END I0329 17:23:08.890888 2308 trace.go:205] Trace[1106410694]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:134 (29-Mar-2021 17:22:38.889) (total time: 30000ms): Trace[1106410694]: [30.000768881s] [30.000768881s] END E0329 17:23:08.890888 2308 reflector.go:138] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to watch v1.MachineConfig: failed to list v1.MachineConfig: Get "https://172.30.0.1:443/apis/machineconfiguration.openshift.io/v1/machineconfigs?limit=500&resourceVersion=0": dial tcp 172.30.0.1:443: i/o timeout E0329 17:23:08.890933 2308 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch v1.Node: failed to list v1.Node: Get "https://172.30.0.1:443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 172.30.0.1:443: i/o timeout W0329 17:23:10.563922 2308 daemon.go:645] Got an error from auxiliary tools: error: cannot apply annotation for SSH access due to: unable to update node "nil": node "hw-control-okd-master-01.tutu.ru" not found I0329 17:23:10.895324 2308 daemon.go:395] Node hw-control-okd-master-01.tutu.ru is part of the control plane I0329 17:23:12.820518 2308 node.go:24] No machineconfiguration.openshift.io/currentConfig annotation on node hw-control-okd-master-01.tutu.ru: map[io.cilium.network.ipv4-cilium-host:10.128.5.140 io.cilium.network.ipv4-health-ip:10.128.5.182 io.cilium.network.ipv4-pod-cidr:10.128.4.0/23 volumes.kubernetes.io/controller-managed-attach-detach:true], in cluster bootstrap, loading initial node annotation from /etc/machine-config-daemon/node-annotations.json I0329 17:23:12.821312 2308 node.go:45] Setting initial node config: rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:23:12.932860 2308 daemon.go:781] In bootstrap mode I0329 17:23:12.932881 2308 daemon.go:809] Current+desired config: rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:23:12.940798 2308 daemon.go:508] Detected a login session before the daemon took over on first boot I0329 17:23:12.940821 2308 daemon.go:509] Applying annotation: machineconfiguration.openshift.io/ssh I0329 17:23:12.951453 2308 daemon.go:1061] No bootstrap pivot required; unlinking bootstrap node annotations I0329 17:23:12.951530 2308 daemon.go:1099] Validating against pending config rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:23:12.976980 2308 daemon.go:1110] Validated on-disk state I0329 17:23:12.994460 2308 daemon.go:1165] Completing pending config rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:23:12.994496 2308 update.go:1942] completed update for config rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:23:12.996584 2308 daemon.go:1181] In desired config rendered-master-6f07c6beae2c83dc74572f71ca4c598a I0329 17:24:21.706227 2308 update.go:596] Checking Reconcilable for config rendered-master-6f07c6beae2c83dc74572f71ca4c598a to rendered-master-a05308b8c49c6c31cfcdb230f61c74f4 I0329 17:24:21.732883 2308 update.go:1942] Starting update from rendered-master-6f07c6beae2c83dc74572f71ca4c598a to rendered-master-a05308b8c49c6c31cfcdb230f61c74f4: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false} I0329 17:24:21.743333 2308 update.go:1942] Update prepared; beginning drain E0329 17:24:23.007991 2308 daemon.go:342] WARNING: ignoring DaemonSet-managed Pods: cilium/cilium-zd48f, fluent-bit/fluent-bit-xrwz7, goldpinger/goldpinger-bwbnq, openshift-cluster-node-tuning-operator/tuned-k7rqg, openshift-controller-manager/controller-manager-ffc6m, openshift-dns/dns-default-f8kwt, openshift-image-registry/node-ca-898tp, openshift-machine-config-operator/machine-config-daemon-jv8gs, openshift-machine-config-operator/machine-config-server-t8vdz, openshift-monitoring/node-exporter-pv2zq, openshift-multus/multus-admission-controller-959wc, openshift-multus/multus-l5jtk, openshift-multus/network-metrics-daemon-w9rg8, openshift-network-diagnostics/network-check-target-xrtz8, platform-hacks/blackbox-exporter-fcrzp, projectcontour/envoy-kxpwt; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: openshift-kube-controller-manager/revision-pruner-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011220 2308 daemon.go:342] evicting pod openshift-etcd/etcd-quorum-guard-68f484d8cc-vtdj2 I0329 17:24:23.011227 2308 daemon.go:342] evicting pod openshift-kube-apiserver/installer-29-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011233 2308 daemon.go:342] evicting pod openshift-kube-controller-manager/installer-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011252 2308 daemon.go:342] evicting pod openshift-etcd/installer-16-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011220 2308 daemon.go:342] evicting pod prometheus-proxy/prometheus-proxy-2-fn5kl I0329 17:24:23.011259 2308 daemon.go:342] evicting pod openshift-ingress/router-default-664dcc74bf-4dxbc I0329 17:24:23.011273 2308 daemon.go:342] evicting pod openshift-kube-scheduler/installer-13-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011274 2308 daemon.go:342] evicting pod openshift-kube-apiserver/installer-28-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011276 2308 daemon.go:342] evicting pod openshift-kube-controller-manager/revision-pruner-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.011282 2308 daemon.go:342] evicting pod openshift-oauth-apiserver/apiserver-67d5d4f5d5-btr4d I0329 17:24:23.011299 2308 daemon.go:342] evicting pod openshift-apiserver/apiserver-c47ffd7cd-84cwv I0329 17:24:23.011329 2308 daemon.go:342] evicting pod openshift-etcd/installer-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.050548 2308 daemon.go:328] Evicted pod openshift-kube-apiserver/installer-28-hw-control-okd-master-01.tutu.ru I0329 17:24:23.055056 2308 daemon.go:328] Evicted pod openshift-kube-controller-manager/installer-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.065098 2308 daemon.go:328] Evicted pod openshift-etcd/installer-16-hw-control-okd-master-01.tutu.ru I0329 17:24:23.065106 2308 daemon.go:328] Evicted pod openshift-etcd/installer-15-hw-control-okd-master-01.tutu.ru I0329 17:24:23.071119 2308 daemon.go:328] Evicted pod openshift-kube-scheduler/installer-13-hw-control-okd-master-01.tutu.ru I0329 17:24:23.443937 2308 daemon.go:328] Evicted pod openshift-kube-apiserver/installer-29-hw-control-okd-master-01.tutu.ru I0329 17:24:29.238859 2308 request.go:655] Throttling request took 1.166125055s, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-apiserver/pods/apiserver-c47ffd7cd-84cwv I0329 17:24:34.243356 2308 daemon.go:328] Evicted pod openshift-oauth-apiserver/apiserver-67d5d4f5d5-btr4d I0329 17:24:35.443681 2308 daemon.go:328] Evicted pod openshift-kube-controller-manager/revision-pruner-15-hw-control-okd-master-01.tutu.ru I0329 17:24:43.365185 2308 daemon.go:679] Detected a new login session: New session 3 of user core. I0329 17:24:43.365218 2308 daemon.go:680] Login access is discouraged! Applying annotation: machineconfiguration.openshift.io/ssh I0329 17:24:45.071318 2308 daemon.go:328] Evicted pod openshift-etcd/etcd-quorum-guard-68f484d8cc-vtdj2 I0329 17:25:05.058184 2308 daemon.go:328] Evicted pod prometheus-proxy/prometheus-proxy-2-fn5kl I0329 17:25:53.075637 2308 update.go:218] Draining failed with: [error when waiting for pod "router-default-664dcc74bf-4dxbc" terminating: global timeout reached: 1m30s, error when waiting for pod "apiserver-c47ffd7cd-84cwv" terminating: global timeout reached: 1m30s], retrying E0329 17:26:04.316887 2308 daemon.go:342] WARNING: ignoring DaemonSet-managed Pods: cilium/cilium-zd48f, fluent-bit/fluent-bit-xrwz7, goldpinger/goldpinger-bwbnq, openshift-cluster-node-tuning-operator/tuned-k7rqg, openshift-controller-manager/controller-manager-ffc6m, openshift-dns/dns-default-f8kwt, openshift-image-registry/node-ca-898tp, openshift-machine-config-operator/machine-config-daemon-jv8gs, openshift-machine-config-operator/machine-config-server-t8vdz, openshift-monitoring/node-exporter-pv2zq, openshift-multus/multus-admission-controller-959wc, openshift-multus/multus-l5jtk, openshift-multus/network-metrics-daemon-w9rg8, openshift-network-diagnostics/network-check-target-xrtz8, platform-hacks/blackbox-exporter-fcrzp, projectcontour/envoy-kxpwt I0329 17:26:04.320232 2308 daemon.go:342] evicting pod openshift-etcd/installer-16-hw-control-okd-master-01.tutu.ru I0329 17:26:04.320255 2308 daemon.go:342] evicting pod openshift-etcd/revision-pruner-16-hw-control-okd-master-01.tutu.ru I0329 17:26:04.320283 2308 daemon.go:342] evicting pod openshift-kube-apiserver/installer-29-hw-control-okd-master-01.tutu.ru I0329 17:26:04.320287 2308 daemon.go:342] evicting pod openshift-kube-scheduler/installer-13-hw-control-okd-master-01.tutu.ru I0329 17:26:04.320324 2308 daemon.go:342] evicting pod openshift-apiserver/apiserver-c47ffd7cd-84cwv I0329 17:26:04.320232 2308 daemon.go:342] evicting pod openshift-kube-scheduler/revision-pruner-13-hw-control-okd-master-01.tutu.ru I0329 17:26:04.320333 2308 daemon.go:342] evicting pod openshift-ingress/router-default-664dcc74bf-4dxbc I0329 17:26:04.339729 2308 daemon.go:328] Evicted pod openshift-kube-scheduler/installer-13-hw-control-okd-master-01.tutu.ru I0329 17:26:04.339729 2308 daemon.go:328] Evicted pod openshift-kube-scheduler/revision-pruner-13-hw-control-okd-master-01.tutu.ru I0329 17:26:04.339976 2308 daemon.go:328] Evicted pod openshift-etcd/installer-16-hw-control-okd-master-01.tutu.ru I0329 17:26:04.340024 2308 daemon.go:328] Evicted pod openshift-etcd/revision-pruner-16-hw-control-okd-master-01.tutu.ru I0329 17:26:04.340070 2308 daemon.go:328] Evicted pod openshift-kube-apiserver/installer-29-hw-control-okd-master-01.tutu.ru I0329 17:26:15.335021 2308 daemon.go:328] Evicted pod openshift-apiserver/apiserver-c47ffd7cd-84cwv I0329 17:26:15.337413 2308 daemon.go:328] Evicted pod openshift-ingress/router-default-664dcc74bf-4dxbc I0329 17:26:15.337460 2308 update.go:1942] drain complete I0329 17:26:15.339196 2308 update.go:234] Successful drain took 113.593771076 seconds I0329 17:26:15.339215 2308 update.go:1219] Updating files I0329 17:26:15.348682 2308 update.go:1654] Writing file "/etc/NetworkManager/conf.d/99-keyfiles.conf" I0329 17:26:15.351644 2308 update.go:1654] Writing file "/etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt" I0329 17:26:15.373455 2308 update.go:1654] Writing file "/etc/kubernetes/apiserver-url.env" I0329 17:26:15.376285 2308 update.go:1654] Writing file "/etc/tmpfiles.d/cleanup-cni.conf" I0329 17:26:15.378885 2308 update.go:1654] Writing file "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem" I0329 17:26:15.381013 2308 update.go:1654] Writing file "/usr/local/bin/configure-ovs.sh" I0329 17:26:15.384037 2308 update.go:1654] Writing file "/etc/containers/storage.conf" I0329 17:26:15.386881 2308 update.go:1654] Writing file "/etc/mco/proxy.env" I0329 17:26:15.389683 2308 update.go:1654] Writing file "/etc/systemd/system.conf.d/10-default-env-godebug.conf" I0329 17:26:15.392539 2308 update.go:1654] Writing file "/etc/modules-load.d/iptables.conf" I0329 17:26:15.395382 2308 update.go:1654] Writing file "/etc/kubernetes/kubelet-ca.crt" I0329 17:26:15.398448 2308 update.go:1654] Writing file "/etc/systemd/system.conf.d/kubelet-cgroups.conf" I0329 17:26:15.401306 2308 update.go:1654] Writing file "/etc/systemd/system/kubelet.service.d/20-logging.conf" I0329 17:26:15.404076 2308 update.go:1654] Writing file "/etc/NetworkManager/conf.d/sdn.conf" I0329 17:26:15.407151 2308 update.go:1654] Writing file "/etc/tmpfiles.d/nm.conf" I0329 17:26:15.409851 2308 update.go:1654] Writing file "/var/lib/kubelet/config.json" I0329 17:26:15.412648 2308 update.go:1654] Writing file "/etc/kubernetes/ca.crt" I0329 17:26:15.415407 2308 update.go:1654] Writing file "/etc/ssh/sshd_config.d/10-disable-ssh-key-dir.conf" I0329 17:26:15.418518 2308 update.go:1654] Writing file "/etc/sysctl.d/forward.conf" I0329 17:26:15.421510 2308 update.go:1654] Writing file "/etc/sysctl.d/inotify.conf" I0329 17:26:15.424130 2308 update.go:1654] Writing file "/usr/local/bin/recover-kubeconfig.sh" I0329 17:26:15.426698 2308 update.go:1654] Writing file "/usr/local/sbin/set-valid-hostname.sh" I0329 17:26:15.429484 2308 update.go:1654] Writing file "/etc/kubernetes/kubelet-plugins/volume/exec/.dummy" I0329 17:26:15.431430 2308 update.go:1654] Writing file "/etc/containers/registries.conf" I0329 17:26:15.434176 2308 update.go:1654] Writing file "/etc/crio/crio.conf.d/00-default" I0329 17:26:15.436883 2308 update.go:1654] Writing file "/etc/containers/policy.json" I0329 17:26:15.439475 2308 update.go:1654] Writing file "/etc/kubernetes/cloud.conf" I0329 17:26:15.441474 2308 update.go:1654] Writing file "/etc/kubernetes/kubelet.conf" I0329 17:26:15.444406 2308 update.go:1654] Writing file "/etc/chrony.conf" I0329 17:26:15.447009 2308 update.go:1654] Writing file "/etc/crio/crio.conf.d/01-ctrcfg-pidsLimit" I0329 17:26:15.449455 2308 update.go:1654] Writing file "/etc/pivot/kernel-args" I0329 17:26:15.452120 2308 update.go:1654] Writing file "/etc/NetworkManager/dispatcher.d/15-resolv" I0329 17:26:15.454916 2308 update.go:1654] Writing file "/etc/NetworkManager/NetworkManager.conf" I0329 17:26:15.457632 2308 update.go:1547] Dropin for 10-mco-default-env.conf has no content, skipping write I0329 17:26:15.457645 2308 update.go:1554] Removing "/etc/systemd/system/crio.service.d/10-mco-default-env.conf", updated file has zero length I0329 17:26:15.457689 2308 update.go:1561] Writing systemd unit dropin "10-mco-profile-unix-socket.conf" I0329 17:26:15.460138 2308 update.go:1561] Writing systemd unit dropin "10-mco-default-madv.conf" I0329 17:26:16.004035 2308 update.go:1501] Preset systemd unit crio.service I0329 17:26:16.004056 2308 update.go:1561] Writing systemd unit dropin "mco-disabled.conf" I0329 17:26:16.488656 2308 update.go:1501] Preset systemd unit docker.socket I0329 17:26:16.488688 2308 update.go:1547] Dropin for 10-mco-default-env.conf has no content, skipping write I0329 17:26:16.488700 2308 update.go:1554] Removing "/etc/systemd/system/kubelet.service.d/10-mco-default-env.conf", updated file has zero length I0329 17:26:16.488736 2308 update.go:1561] Writing systemd unit dropin "10-mco-default-madv.conf" I0329 17:26:16.490700 2308 update.go:1595] Writing systemd unit "kubelet.service" I0329 17:26:16.492292 2308 update.go:1595] Writing systemd unit "machine-config-daemon-firstboot.service" I0329 17:26:16.493912 2308 update.go:1595] Writing systemd unit "machine-config-daemon-pull.service" I0329 17:26:16.495545 2308 update.go:1595] Writing systemd unit "etc-NetworkManager-system\x2dconnections\x2dmerged.mount" I0329 17:26:16.497377 2308 update.go:1595] Writing systemd unit "node-valid-hostname.service" I0329 17:26:16.499326 2308 update.go:1595] Writing systemd unit "nodeip-configuration.service" I0329 17:26:16.501183 2308 update.go:1595] Writing systemd unit "ovs-configuration.service" I0329 17:26:16.503071 2308 update.go:1561] Writing systemd unit dropin "10-ovs-vswitchd-restart.conf" I0329 17:26:17.005376 2308 update.go:1501] Preset systemd unit ovs-vswitchd.service I0329 17:26:17.005408 2308 update.go:1561] Writing systemd unit dropin "10-ovsdb-restart.conf" I0329 17:26:17.006995 2308 update.go:1547] Dropin for 10-mco-default-env.conf has no content, skipping write I0329 17:26:17.007009 2308 update.go:1554] Removing "/etc/systemd/system/pivot.service.d/10-mco-default-env.conf", updated file has zero length I0329 17:26:17.014457 2308 update.go:1632] Could not reset unit preset for pivot.service, skipping. (Error msg: error running preset on unit: Failed to preset unit: Unit file pivot.service does not exist. ) I0329 17:26:17.014487 2308 update.go:1561] Writing systemd unit dropin "mco-controlplane-nice.conf" I0329 17:26:17.470423 2308 update.go:1501] Preset systemd unit rpm-ostreed.service I0329 17:26:17.470442 2308 update.go:1561] Writing systemd unit dropin "mco-disabled.conf" I0329 17:26:17.938006 2308 update.go:1501] Preset systemd unit zincati.service I0329 17:26:18.385525 2308 update.go:1479] Enabled systemd units: [kubelet.service machine-config-daemon-firstboot.service machine-config-daemon-pull.service etc-NetworkManager-system\x2dconnections\x2dmerged.mount node-valid-hostname.service nodeip-configuration.service] I0329 17:26:18.841774 2308 update.go:1490] Disabled systemd units [openvswitch.service ovs-configuration.service ovsdb-server.service] I0329 17:26:18.842071 2308 update.go:1297] Deleting stale data I0329 17:26:18.842170 2308 update.go:1360] Removed stale file "/etc/kubernetes/recycler-pod.yaml" I0329 17:26:18.842258 2308 update.go:1360] Removed stale file "/etc/kubernetes/manifests/recycler-pod.yaml" I0329 17:26:18.857108 2308 update.go:1773] Writing SSHKeys at "/home/core/.ssh/authorized_keys" I0329 17:26:18.871193 2308 controlplane.go:42] Device /sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda already uses scheduler bfq I0329 17:26:18.871266 2308 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a error: unable to load --registry-config: error occurred while trying to unmarshal json W0329 17:26:19.223199 2308 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a failed: error: unable to load --registry-config: error occurred while trying to unmarshal json : exit status 1; retrying... I0329 17:26:24.223256 2308 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a error: unable to load --registry-config: error occurred while trying to unmarshal json W0329 17:26:24.315939 2308 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a failed: error: unable to load --registry-config: error occurred while trying to unmarshal json : exit status 1; retrying... I0329 17:26:34.316095 2308 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a error: unable to load --registry-config: error occurred while trying to unmarshal json W0329 17:26:34.389625 2308 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-759348359 --registry-config /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a failed: error: unable to load --registry-config: error occurred while trying to unmarshal json : exit status 1; retrying... I0329 17:26:34.389663 2308 update.go:368] Falling back to using podman cp to fetch OS image content I0329 17:26:34.389688 2308 run.go:18] Running: nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a 5034281716ecf33a24631bde35264d4ab4ce5da77f48965ab64bb9efbafcc536 I0329 17:27:17.795165 2308 rpm-ostree.go:261] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-dfbaca4a-36a1-492a-b768-db4e748e750f quay.io/openshift/okd-content@sha256:13a16cecc46dea3c626fa231825aa1a4e158d95616632856c32c73b992a19a9a I0329 17:27:17.937940 2308 run.go:18] Running: nice -- ionice -c 3 podman cp 185d8e2643a5714a8401345da894c26ee64c6c48e40674e50d85c0712a19f252:/ /run/mco-machine-os-content/os-content-759348359 Error: 2 errors occurred: