search

openshift-metal3 / dev-scripts

Scripts to automate development/test setup for openshift integration with https://github.com/metal3-io/
Apache License 2.0
93 stars 187 forks source link

Delpoyment fails: unhealthy cluster and/or container with ID starting with discovery not found: ID does not exist #631

Closed achuzhoy closed 5 years ago

achuzhoy commented 5 years ago

From the deployment log 06_create_cluster-2019-06-19-115647.log: level=debug msg="Using \"Install Config\" loaded from state file" level=debug msg="Reusing previously-fetched \"Install Config\"" level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform" level=error msg="boostrap host address and at least one control plane host address must be provided" level=fatal msg="waiting for Kubernetes API: context deadline exceeded"

From the bootstrap VM:

[core@dhcp-19-134-205 ~]$ journalctl -b -f -u bootkube.service -- Logs begin at Wed 2019-06-19 16:06:24 UTC. -- Jun 19 17:02:48 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-2.qe1.kni.lab.eng.bos.redhat.com:2379 is unhealthy: failed to connect: dial tcp 10.19.134.4:2379: connect: connection refused Jun 19 17:02:48 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: Error: unhealthy cluster Jun 19 17:02:48 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: etcdctl failed. Retrying in 5 seconds... Jun 19 17:02:54 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-1.qe1.kni.lab.eng.bos.redhat.com:2379 is healthy: successfully committed proposal: took = 830.456µs Jun 19 17:02:54 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-0.qe1.kni.lab.eng.bos.redhat.com:2379 is healthy: successfully committed proposal: took = 1.567924ms Jun 19 17:12:54 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-2.qe1.kni.lab.eng.bos.redhat.com:2379 is unhealthy: failed to connect: dial tcp 10.19.134.4:2379: connect: connection refused Jun 19 17:12:54 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: Error: unhealthy cluster Jun 19 17:12:54 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: etcdctl failed. Retrying in 5 seconds... Jun 19 17:12:59 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-1.qe1.kni.lab.eng.bos.redhat.com:2379 is healthy: successfully committed proposal: took = 961.599µs Jun 19 17:12:59 dhcp-19-134-205.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1933]: https://etcd-0.qe1.kni.lab.eng.bos.redhat.com:2379 is healthy: successfully committed proposal: took = 948.629µs

On etcd-1.qe1.kni.lab.eng.bos.redhat.com:

[root@openshift-master-2 ~]# crictl ps -a CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT POD ID e4f1ceb886872 8c0a60e9d4dc242b0307098c4b623d55cbd0b42b0fb1c09b14cd893bf5a8d80c 2 minutes ago Running discovery 7 2be7307487c71 1d47dddc54ab4 8c0a60e9d4dc242b0307098c4b623d55cbd0b42b0fb1c09b14cd893bf5a8d80c 12 minutes ago Exited discovery 6 2be7307487c71 6911b9136596a e92a20d050d2ada81ea032d4c92f47d53fc6fe9b15908cecf7e2c434ed1e1357 45 minutes ago Running haproxy-monitor 0 2157b5155a732 81acf732b59a0 docker.io/library/haproxy@sha256:f88bca67a2782e7bd88af168d5c11216c32104d12fe9240fac54a1d3196e3f9c 45 minutes ago Running haproxy 0 2157b5155a732 d375b5f45844a quay.io/celebdor/keepalived@sha256:a364cb7a4eb2bdf2ab5d63093702b3fda6db29c9b1e860543f47b47ab13e5026 About an hour ago Running keepalived 0 cf51ec34441ea b9527e0a5f815 quay.io/openshift-metalkube/mdns-publisher@sha256:3f12f9b7dd48ce3b26bad80f57b58cfbaaf81cad65400608ae6ccc63a94b0a56 About an hour ago Running mdns-publisher 0 be8e3bf827e41 14201b400bf11 quay.io/openshift-metalkube/coredns-mdns@sha256:64ffd5d2175c5befb7a06099088c871ae67f73b1c93bce942f0545ccf6e45129 About an hour ago Running coredns 0 664417f594aa9 c52ff7b49607c quay.io/openshift/origin-node@sha256:459eb3885340ddd69aa582560cec3081c919b9ab75a072784246e151d6565973 About an hour ago Exited render-corefile 0 664417f594aa9 f21bdd50cf955 quay.io/openshift/origin-node@sha256:459eb3885340ddd69aa582560cec3081c919b9ab75a072784246e151d6565973 About an hour ago Exited render-keepalived-conf 0 cf51ec34441ea 0302559401bf7 quay.io/openshift/origin-node@sha256:459eb3885340ddd69aa582560cec3081c919b9ab75a072784246e151d6565973 About an hour ago Exited render-haproxy-cfg 0 2157b5155a732 504d7828fe2af quay.io/openshift/origin-node@sha256:459eb3885340ddd69aa582560cec3081c919b9ab75a072784246e151d6565973 About an hour ago Exited render-config 0 be8e3bf827e41 df8908978009f quay.io/openshift-metalkube/kubeconfig-extractor@sha256:a19d93db72cdfc4d02c1747ff2400962a8f8d3b738c297a060f1fdb116cb21fe About an hour ago Exited clusterrc-generation 0 664417f594aa9 8815ea2692d47 quay.io/openshift-metalkube/kubeconfig-extractor@sha256:a19d93db72cdfc4d02c1747ff2400962a8f8d3b738c297a060f1fdb116cb21fe About an hour ago Exited clusterrc-generation 0 be8e3bf827e41 20fef5db39df4 quay.io/openshift-metalkube/kubeconfig-extractor@sha256:a19d93db72cdfc4d02c1747ff2400962a8f8d3b738c297a060f1fdb116cb21fe About an hour ago Exited clusterrc-generation 0 cf51ec34441ea cc2210bca7567 quay.io/openshift-metalkube/kubeconfig-extractor@sha256:a19d93db72cdfc4d02c1747ff2400962a8f8d3b738c297a060f1fdb116cb21fe About an hour ago Exited clusterrc-generation 0 2157b5155a732

[root@openshift-master-2 ~]# sudo crictl logs discovery E0619 17:20:13.447474 34660 remote_runtime.go:321] ContainerStatus "discovery" from runtime service failed: rpc error: code = Unknown desc = container with ID starting with discovery not found: ID does not exist FATA[0000] rpc error: code = Unknown desc = container with ID starting with discovery not found: ID does not exist

mcornea commented 5 years ago

openshift-master-2 doesn't include the ns1 address in /etc/resolv.conf so the DNS lookup fail. openshift-master-0 and openshift-master-1 do include the ns1 address so the question is what edits resolv.conf?

[root@openshift-master-2 core]# dig +short ns1.qe1.kni.lab.eng.bos.redhat.com
10.19.134.14
[root@openshift-master-2 core]# cat /etc/resolv.conf 
# Generated by NetworkManager
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.42.41
nameserver 10.11.5.19
nameserver 10.5.30.160
[root@openshift-master-2 core]# crictl ps -a | grep discover
b872f14d38cc4       8c0a60e9d4dc242b0307098c4b623d55cbd0b42b0fb1c09b14cd893bf5a8d80c                                                           About a minute ago   Running             discovery                9                   2be7307487c71
cac5cc2b01664       8c0a60e9d4dc242b0307098c4b623d55cbd0b42b0fb1c09b14cd893bf5a8d80c                                                           12 minutes ago       Exited              discovery                8                   2be7307487c71
[root@openshift-master-2 core]# crictl logs -f cac5cc2b01664
I0619 17:26:04.825683       1 run.go:46] Version: 4.0.0-alpha.0-459-gfb707ca3-dirty (fb707ca3c8740c787435d63f8ae843449c69ce60)
E0619 17:26:04.827645       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:26:09.948626       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.11.5.19:53: no such host
E0619 17:26:39.949876       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:26:39.950400       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:27:09.949687       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:27:09.950261       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:27:39.949570       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:27:39.950102       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:28:09.949624       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:28:09.950024       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:28:39.949570       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:28:39.950031       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:29:09.949623       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:29:09.950044       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:29:39.949617       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:29:39.950163       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:30:09.949664       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:30:09.950141       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:30:39.949568       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:30:39.950048       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:31:09.949671       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:31:09.950169       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:31:09.950570       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
E0619 17:31:09.950961       1 run.go:63] error looking up self: lookup _etcd-server-ssl._tcp.qe1.kni.lab.eng.bos.redhat.com on 10.19.42.41:53: no such host
F0619 17:31:09.950974       1 main.go:30] Error executing etcd-setup-environment: could not find self: timed out waiting for the condition

[core@openshift-master-0 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.134.14
nameserver 10.19.42.41
nameserver 10.11.5.19
nameserver 10.5.30.160

[core@openshift-master-1 ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.134.14
nameserver 10.19.42.41
nameserver 10.11.5.19
nameserver 10.5.30.160
mcornea commented 5 years ago

render-corefile logs on master-2:

[root@openshift-master-2 core]# crictl logs c52ff7b49607c
+ source /etc/kubernetes/static-pod-resources/clusterrc
++ NAME=qe1
++ APIURL=https://api-int.qe1.kni.lab.eng.bos.redhat.com:6443
++ APIHOST=api-int.qe1.kni.lab.eng.bos.redhat.com
++ DOMAIN=qe1.kni.lab.eng.bos.redhat.com
++ BASE_DOMAIN=kni.lab.eng.bos.redhat.com
+ export DOMAIN
+ export NAME
++ dig +noall +answer api.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ API_VIP=10.19.134.13
+ export API_VIP
+ /usr/libexec/platform-python -c 'from __future__ import print_function
import os
with open('\''/etc/kubernetes/static-pod-resources/Corefile.template'\'', '\''r'\'') as f:
    content = f.read()
with open('\''/etc/coredns/Corefile'\'', '\''w'\'') as dest:
    print(os.path.expandvars(content), file=dest)'
++ dig +noall +answer ns1.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ DNS_VIP=10.19.134.14
+ grep -v 10.19.134.14 /etc/resolv.conf
+ tee /etc/coredns/resolv.conf
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.42.41
nameserver 10.11.5.19
nameserver 10.5.30.160
mcornea commented 5 years ago

render-corefile logs on master-0 and master-1 where resolv.conf was properly generated:

[root@openshift-master-0 core]# crictl logs $(crictl ps -a | grep render-corefile | awk {'print $1'})
+ source /etc/kubernetes/static-pod-resources/clusterrc
++ NAME=qe1
++ APIURL=https://api-int.qe1.kni.lab.eng.bos.redhat.com:6443
++ APIHOST=api-int.qe1.kni.lab.eng.bos.redhat.com
++ DOMAIN=qe1.kni.lab.eng.bos.redhat.com
++ BASE_DOMAIN=kni.lab.eng.bos.redhat.com
+ export DOMAIN
+ export NAME
++ dig +noall +answer api.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ API_VIP=10.19.134.13
+ export API_VIP
+ /usr/libexec/platform-python -c 'from __future__ import print_function
import os
with open('\''/etc/kubernetes/static-pod-resources/Corefile.template'\'', '\''r'\'') as f:
    content = f.read()
with open('\''/etc/coredns/Corefile'\'', '\''w'\'') as dest:
    print(os.path.expandvars(content), file=dest)'
++ dig +noall +answer ns1.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ DNS_VIP=10.19.134.14
+ grep -v 10.19.134.14 /etc/resolv.conf
+ tee /etc/coredns/resolv.conf
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.42.41
nameserver 10.11.5.19

[root@openshift-master-1 core]# crictl logs $(crictl ps -a | grep render-corefile | awk {'print $1'})
+ source /etc/kubernetes/static-pod-resources/clusterrc
++ NAME=qe1
++ APIURL=https://api-int.qe1.kni.lab.eng.bos.redhat.com:6443
++ APIHOST=api-int.qe1.kni.lab.eng.bos.redhat.com
++ DOMAIN=qe1.kni.lab.eng.bos.redhat.com
++ BASE_DOMAIN=kni.lab.eng.bos.redhat.com
+ export DOMAIN
+ export NAME
++ dig +noall +answer api.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ API_VIP=10.19.134.13
+ export API_VIP
+ /usr/libexec/platform-python -c 'from __future__ import print_function
import os
with open('\''/etc/kubernetes/static-pod-resources/Corefile.template'\'', '\''r'\'') as f:
    content = f.read()
with open('\''/etc/coredns/Corefile'\'', '\''w'\'') as dest:
    print(os.path.expandvars(content), file=dest)'
++ dig +noall +answer ns1.qe1.kni.lab.eng.bos.redhat.com
++ awk '{print $NF}'
+ DNS_VIP=10.19.134.14
+ grep -v 10.19.134.14 /etc/resolv.conf
+ tee /etc/coredns/resolv.conf
search qe1.kni.lab.eng.bos.redhat.com
nameserver 10.19.42.41
nameserver 10.11.5.19
cybertron commented 5 years ago

If you search through the journal on the broken node for "prepender", do you see the dns-vip-prepender running and are there any errors?

mcornea commented 5 years ago

Yes, it's running but I can't spot any obvious errors:

Jun 19 16:28:58 localhost.localdomain root[2389]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:28:58 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:59 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 16:29:01 localhost.localdomain NetworkManager[2103]: <info>  [1560961741.9704] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:01 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 21 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPOFFER from 10.19.134.254
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPACK from 10.19.134.254 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8994] dhcp4 (eno2):   address 10.19.134.4
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   lease time 86400
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.5.30.160'
lines 2269-2298
Jun 19 16:28:58 localhost.localdomain dhclient[2374]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 7 (xid=0x62f2ba41)
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: Created duid "\000\004b\373\246)\376\004J\036\247c\260^V>\224\203".
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPDISCOVER on eno1 to 255.255.255.255 port 67 interval 8 (xid=0x6325402)
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPREQUEST on eno1 to 255.255.255.255 port 67 (xid=0x6325402)
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPOFFER from 172.22.0.1
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPACK from 172.22.0.1 (xid=0x6325402)
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   address 172.22.0.25
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   plen 24 (255.255.255.0)
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   lease time 3600
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7066] dhcp4 (eno1): state changed unknown -> bound
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7072] device (eno1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: bound to 172.22.0.25 -- renewal in 1728 seconds.
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: req:3 'pre-up' [eno1]: new request (1 scripts)
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: req:3 'pre-up' [eno1]: start running ordered scripts...
Jun 19 16:28:58 localhost.localdomain root[2389]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:28:58 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:59 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 16:29:01 localhost.localdomain NetworkManager[2103]: <info>  [1560961741.9704] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:01 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 21 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPOFFER from 10.19.134.254
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPACK from 10.19.134.254 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8994] dhcp4 (eno2):   address 10.19.134.4
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   lease time 86400
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.5.30.160'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   domain name 'qe1.kni.lab.eng.bos.redhat.com'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2): state changed unknown -> bound
Jun 19 16:29:05 localhost.localdomain nm-dispatcher[2226]: req:5 'dhcp4-change' [eno2]: new request (3 scripts)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: bound to 10.19.134.4 -- renewal in 34916 seconds.
Jun 19 16:29:11 localhost.localdomain root[2500]: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2713] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2715] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2724] policy: set 'Wired connection 2' (eno2) as default for IPv4 routing and DNS
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2727] policy: set 'Wired connection 2' (eno2) as default for IPv6 routing and DNS
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2731] device (eno1): Activation: successful, device activated.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:6 'up' [eno1]: new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:7 'connectivity-change': new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain root[2502]: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:11 localhost.localdomain root[2604]: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:11 localhost.localdomain root[2608]: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: prepend domain-name-servers 10.19.134.14;
Jun 19 16:29:11 localhost.localdomain root[2611]: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:5 'dhcp4-change' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5528] device (eno2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5531] device (eno2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5535] manager: NetworkManager state is now CONNECTED_SITE
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5548] device (eno2): Activation: successful, device activated.
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5554] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:8 'up' [eno2]: new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5560] manager: startup complete
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:9 'connectivity-change': new request (3 scripts)
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPREQUEST on eno1 to 255.255.255.255 port 67 (xid=0x6325402)
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPOFFER from 172.22.0.1
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: DHCPACK from 172.22.0.1 (xid=0x6325402)
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   address 172.22.0.25
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   plen 24 (255.255.255.0)
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7065] dhcp4 (eno1):   lease time 3600
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7066] dhcp4 (eno1): state changed unknown -> bound
Jun 19 16:28:58 localhost.localdomain NetworkManager[2103]: <info>  [1560961738.7072] device (eno1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:28:58 localhost.localdomain dhclient[2373]: bound to 172.22.0.25 -- renewal in 1728 seconds.
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: req:3 'pre-up' [eno1]: new request (1 scripts)
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: req:3 'pre-up' [eno1]: start running ordered scripts...
Jun 19 16:28:58 localhost.localdomain root[2389]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:58 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:28:58 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:28:59 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 16:29:01 localhost.localdomain NetworkManager[2103]: <info>  [1560961741.9704] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:01 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 21 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPOFFER from 10.19.134.254
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: DHCPACK from 10.19.134.254 (xid=0x62f2ba41)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8994] dhcp4 (eno2):   address 10.19.134.4
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   lease time 86400
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   nameserver '10.5.30.160'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2):   domain name 'qe1.kni.lab.eng.bos.redhat.com'
Jun 19 16:29:05 localhost.localdomain NetworkManager[2103]: <info>  [1560961745.8995] dhcp4 (eno2): state changed unknown -> bound
Jun 19 16:29:05 localhost.localdomain nm-dispatcher[2226]: req:5 'dhcp4-change' [eno2]: new request (3 scripts)
Jun 19 16:29:05 localhost.localdomain dhclient[2374]: bound to 10.19.134.4 -- renewal in 34916 seconds.
Jun 19 16:29:11 localhost.localdomain root[2500]: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2713] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2715] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2724] policy: set 'Wired connection 2' (eno2) as default for IPv4 routing and DNS
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2727] policy: set 'Wired connection 2' (eno2) as default for IPv6 routing and DNS
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.2731] device (eno1): Activation: successful, device activated.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:6 'up' [eno1]: new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:7 'connectivity-change': new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain root[2502]: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:11 localhost.localdomain root[2604]: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:11 localhost.localdomain root[2608]: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: prepend domain-name-servers 10.19.134.14;
Jun 19 16:29:11 localhost.localdomain root[2611]: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: <13>Jun 19 16:29:11 root: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:5 'dhcp4-change' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5528] device (eno2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5531] device (eno2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5535] manager: NetworkManager state is now CONNECTED_SITE
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5548] device (eno2): Activation: successful, device activated.
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5554] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:8 'up' [eno2]: new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.5560] manager: startup complete
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:9 'connectivity-change': new request (3 scripts)
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started Network Manager Wait Online.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:6 'up' [eno1]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Reached target Network is Online.
Jun 19 16:29:11 localhost.localdomain systemd[1]: Starting Generate /run/issue.d/console-login-helper-messages.issue...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Starting Open Container Initiative Daemon...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Starting NFS status monitor for NFSv2/3 locking....
Jun 19 16:29:11 localhost.localdomain systemd[1]: Starting RPC Bind...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started Generate /run/issue.d/console-login-helper-messages.issue.
Jun 19 16:29:11 localhost.localdomain systemd[1]: iscsi.service: Unit cannot be reloaded because it is inactive.
Jun 19 16:29:11 localhost.localdomain systemd[1]: Starting Permit User Sessions...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started RPC Bind.
Jun 19 16:29:11 localhost.localdomain rpc.statd[2644]: Version 2.3.3 starting
Jun 19 16:29:11 localhost.localdomain rpc.statd[2644]: Flags: TI-RPC
Jun 19 16:29:11 localhost.localdomain rpc.statd[2644]: Initializing NSM state
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started Permit User Sessions.
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started NFS status monitor for NFSv2/3 locking..
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:7 'connectivity-change': start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started Serial Getty on ttyS0.
Jun 19 16:29:11 localhost.localdomain nm-dispatcher[2226]: req:8 'up' [eno2]: start running ordered scripts...
Jun 19 16:29:11 localhost.localdomain systemd[1]: Started Getty on tty1.
Jun 19 16:29:11 localhost.localdomain systemd[1]: Reached target Login Prompts.
Jun 19 16:29:11 localhost.localdomain NetworkManager[2103]: <info>  [1560961751.6690] policy: set-hostname: set hostname to 'openshift-master-2.qe1.kni.lab.eng.bos.redhat.com' (from address lookup)
Jun 19 16:29:11 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com systemd-hostnamed[2209]: Changed host name to 'openshift-master-2.qe1.kni.lab.eng.bos.redhat.com'
Jun 19 16:29:11 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2226]: req:10 'hostname': new request (3 scripts)
mcornea commented 5 years ago

This is how it looks on openshift-0:

19 16:29:04 localhost.localdomain root[2407]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:29:04 localhost.localdomain nm-dispatcher[2316]: <13>Jun 19 16:29:04 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 16:29:04 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 16:29:10 localhost.localdomain dhclient[2392]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 7 (xid=0xaf75a10a)
Jun 19 16:29:10 localhost.localdomain dhclient[2392]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0xaf75a10a)
Jun 19 16:29:10 localhost.localdomain dhclient[2392]: DHCPOFFER from 10.19.134.254
Jun 19 16:29:10 localhost.localdomain dhclient[2392]: DHCPACK from 10.19.134.254 (xid=0xaf75a10a)
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5672] dhcp4 (eno2):   address 10.19.134.2
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5672] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5672] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5672] dhcp4 (eno2):   lease time 86400
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5673] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5673] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5673] dhcp4 (eno2):   nameserver '10.5.30.160'
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5673] dhcp4 (eno2):   domain name 'qe1.kni.lab.eng.bos.redhat.com'
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5673] dhcp4 (eno2): state changed unknown -> bound
Jun 19 16:29:10 localhost.localdomain NetworkManager[2209]: <info>  [1560961750.5680] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:10 localhost.localdomain dhclient[2392]: bound to 10.19.134.2 -- renewal in 36972 seconds.
Jun 19 16:29:10 localhost.localdomain nm-dispatcher[2316]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 16:29:16 localhost.localdomain root[2506]: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:16 localhost.localdomain nm-dispatcher[2316]: <13>Jun 19 16:29:16 root: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 16:29:16 localhost.localdomain nm-dispatcher[2316]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 16:29:16 localhost.localdomain NetworkManager[2209]: <info>  [1560961756.9984] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:16 localhost.localdomain NetworkManager[2209]: <info>  [1560961756.9986] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:16 localhost.localdomain NetworkManager[2209]: <info>  [1560961756.9997] policy: set 'Wired connection 2' (eno2) as default for IPv4 routing and DNS
Jun 19 16:29:17 localhost.localdomain NetworkManager[2209]: <info>  [1560961757.0000] policy: set 'Wired connection 2' (eno2) as default for IPv6 routing and DNS
Jun 19 16:29:17 localhost.localdomain NetworkManager[2209]: <info>  [1560961757.0004] device (eno1): Activation: successful, device activated.
Jun 19 16:29:17 localhost.localdomain nm-dispatcher[2316]: req:5 'up' [eno1]: new request (3 scripts)
Jun 19 16:29:17 localhost.localdomain nm-dispatcher[2316]: req:6 'connectivity-change': new request (3 scripts)
Jun 19 16:29:17 localhost.localdomain root[2508]: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:17 localhost.localdomain nm-dispatcher[2316]: <13>Jun 19 16:29:17 root: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 16:29:17 localhost.localdomain NetworkManager[2209]: <info>  [1560961757.0021] policy: set-hostname: set hostname to 'openshift-master-0.qe1.kni.lab.eng.bos.redhat.com' (from address lookup)
Jun 19 16:29:17 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com systemd-hostnamed[2229]: Changed host name to 'openshift-master-0.qe1.kni.lab.eng.bos.redhat.com'
Jun 19 16:29:17 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: req:7 'hostname': new request (3 scripts)
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2605]: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: <13>Jun 19 16:29:18 root: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2609]: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: <13>Jun 19 16:29:18 root: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: prepend domain-name-servers 10.19.134.14;
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2612]: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: <13>Jun 19 16:29:18 root: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134>
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: req:5 'up' [eno1]: start running ordered scripts...
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.3998] device (eno2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.4000] device (eno2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.4004] manager: NetworkManager state is now CONNECTED_SITE
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.4016] device (eno2): Activation: successful, device activated.
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.4021] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: req:8 'up' [eno2]: new request (3 scripts)
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2316]: req:9 'connectivity-change': new request (3 scripts)
Jun 19 16:29:18 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2209]: <info>  [1560961758.4028] manager: startup complete
hardys commented 5 years ago

I saw something similar ref https://github.com/openshift-metal3/dev-scripts/issues/522 and selinux was the cause, it prevents the prepender script working when a nic gets restarted, I don't yet know why it works the first time.

mcornea commented 5 years ago

On a 2nd run we didn't hit this issue anymore so it's probably some sort of race condition. We should re-run the deployment on this environment and see if/how often this issue shows up.

In the logs on the failed deployment we can see that the policy: set-hostname: on the failing node run late, after the NM dns-vip-prepender tasks so perhaps this could be related.

Jun 19 19:21:32 localhost.localdomain root[2412]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 19:21:32 localhost.localdomain nm-dispatcher[2256]: <13>Jun 19 19:21:32 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 19:21:32 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 19:21:36 localhost.localdomain dhclient[2397]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 8 (xid=0xe550c67a)
Jun 19 19:21:36 localhost.localdomain dhclient[2397]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0xe550c67a)
Jun 19 19:21:36 localhost.localdomain dhclient[2397]: DHCPOFFER from 10.19.134.254
Jun 19 19:21:36 localhost.localdomain NetworkManager[2185]: <info>  [1560972096.9732] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:36 localhost.localdomain nm-dispatcher[2256]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 19:21:37 localhost.localdomain dhclient[2397]: DHCPACK from 10.19.134.254 (xid=0xe550c67a)
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1165] dhcp4 (eno2):   address 10.19.134.2
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   lease time 86400
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   nameserver '10.5.30.160'
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2):   domain name 'qe1.kni.lab.eng.bos.redhat.com'
Jun 19 19:21:37 localhost.localdomain NetworkManager[2185]: <info>  [1560972097.1166] dhcp4 (eno2): state changed unknown -> bound
Jun 19 19:21:37 localhost.localdomain nm-dispatcher[2256]: req:5 'dhcp4-change' [eno2]: new request (3 scripts)
Jun 19 19:21:37 localhost.localdomain dhclient[2397]: bound to 10.19.134.2 -- renewal in 40021 seconds.
Jun 19 19:21:45 localhost.localdomain root[2511]: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 19:21:45 localhost.localdomain nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 19:21:45 localhost.localdomain nm-dispatcher[2256]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0812] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0814] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0822] policy: set 'Wired connection 2' (eno2) as default for IPv4 routing and DNS
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0823] policy: set 'Wired connection 2' (eno2) as default for IPv6 routing and DNS
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0828] device (eno1): Activation: successful, device activated.
Jun 19 19:21:45 localhost.localdomain nm-dispatcher[2256]: req:6 'up' [eno1]: new request (3 scripts)
Jun 19 19:21:45 localhost.localdomain nm-dispatcher[2256]: req:7 'connectivity-change': new request (3 scripts)
Jun 19 19:21:45 localhost.localdomain root[2513]: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 19:21:45 localhost.localdomain nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 19:21:45 localhost.localdomain NetworkManager[2185]: <info>  [1560972105.0844] policy: set-hostname: set hostname to 'openshift-master-0.qe1.kni.lab.eng.bos.redhat.com' (from address lookup)
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com systemd-hostnamed[2238]: Changed host name to 'openshift-master-0.qe1.kni.lab.eng.bos.redhat.com'
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: req:8 'hostname': new request (3 scripts)
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2617]: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2621]: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: prepend domain-name-servers 10.19.134.14;
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2624]: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134>
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: # Generated by NetworkManager
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: search qe1.kni.lab.eng.bos.redhat.com
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: # Generated by NetworkManager
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: search qe1.kni.lab.eng.bos.redhat.com
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: nameserver 10.19.134.14
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: nameserver 10.19.42.41
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: nameserver 10.11.5.19
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: <13>Jun 19 19:21:45 root: nameserver 10.5.30.160
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: nameserver 10.19.134.14
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: nameserver 10.19.42.41
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: nameserver 10.11.5.19
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com root[2626]: nameserver 10.5.30.160
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: req:5 'dhcp4-change' [eno2]: start running ordered scripts...
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3474] device (eno2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3475] device (eno2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3478] manager: NetworkManager state is now CONNECTED_SITE
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3486] device (eno2): Activation: successful, device activated.
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3490] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2185]: <info>  [1560972105.3495] manager: startup complete
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: req:9 'up' [eno2]: new request (3 scripts)
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: req:10 'connectivity-change': new request (3 scripts)
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com systemd[1]: Started Network Manager Wait Online.
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2256]: req:6 'up' [eno1]: start running ordered scripts...
Jun 19 19:21:45 openshift-master-0.qe1.kni.lab.eng.bos.redhat.com systemd[1]: Reached target Network is Online.
Jun 19 19:21:46 localhost.localdomain root[2392]: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 19:21:46 localhost.localdomain nm-dispatcher[2302]: <13>Jun 19 19:21:46 root: NM dns-vip-prepender triggered by pre-upping eno1.
Jun 19 19:21:46 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
Jun 19 19:21:50 localhost.localdomain NetworkManager[2096]: <info>  [1560972110.0107] device (eno2): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:50 localhost.localdomain nm-dispatcher[2302]: req:4 'pre-up' [eno2]: new request (1 scripts)
Jun 19 19:21:50 localhost.localdomain dhclient[2377]: DHCPDISCOVER on eno2 to 255.255.255.255 port 67 interval 9 (xid=0xd3efe267)
Jun 19 19:21:51 localhost.localdomain dhclient[2377]: DHCPREQUEST on eno2 to 255.255.255.255 port 67 (xid=0xd3efe267)
Jun 19 19:21:51 localhost.localdomain dhclient[2377]: DHCPOFFER from 10.19.134.254
Jun 19 19:21:51 localhost.localdomain dhclient[2377]: DHCPACK from 10.19.134.254 (xid=0xd3efe267)
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3680] dhcp4 (eno2):   address 10.19.134.4
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   plen 24 (255.255.255.0)
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   gateway 10.19.134.254
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   lease time 86400
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   nameserver '10.19.42.41'
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   nameserver '10.11.5.19'
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   nameserver '10.5.30.160'
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2):   domain name 'qe1.kni.lab.eng.bos.redhat.com'
Jun 19 19:21:51 localhost.localdomain NetworkManager[2096]: <info>  [1560972111.3681] dhcp4 (eno2): state changed unknown -> bound
Jun 19 19:21:51 localhost.localdomain nm-dispatcher[2302]: req:5 'dhcp4-change' [eno2]: new request (3 scripts)
Jun 19 19:21:51 localhost.localdomain dhclient[2377]: bound to 10.19.134.4 -- renewal in 42558 seconds.
Jun 19 19:21:58 localhost.localdomain root[2500]: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 19:21:58 localhost.localdomain nm-dispatcher[2302]: <13>Jun 19 19:21:58 root: NM dns-vip-prepender: nameserver could not be resolved, exiting
Jun 19 19:21:58 localhost.localdomain nm-dispatcher[2302]: req:4 'pre-up' [eno2]: start running ordered scripts...
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8747] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8750] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8759] policy: set 'Wired connection 2' (eno2) as default for IPv4 routing and DNS
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8761] policy: set 'Wired connection 2' (eno2) as default for IPv6 routing and DNS
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8766] device (eno1): Activation: successful, device activated.
Jun 19 19:21:58 localhost.localdomain nm-dispatcher[2302]: req:6 'up' [eno1]: new request (3 scripts)
Jun 19 19:21:58 localhost.localdomain nm-dispatcher[2302]: req:7 'connectivity-change': new request (3 scripts)
Jun 19 19:21:58 localhost.localdomain NetworkManager[2096]: <info>  [1560972118.8785] policy: set-hostname: set hostname to 'openshift-master-2.qe1.kni.lab.eng.bos.redhat.com' (from address lookup)
Jun 19 19:21:58 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2504]: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 19:21:58 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:58 root: NM dns-vip-prepender triggered by pre-upping eno2.
Jun 19 19:21:58 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com systemd-hostnamed[2218]: Changed host name to 'openshift-master-2.qe1.kni.lab.eng.bos.redhat.com'
Jun 19 19:21:58 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: req:8 'hostname': new request (3 scripts)
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2608]: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: NM dns-vip-prepender: Checking if DNS VIP is the first entry in resolv.conf
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2612]: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: NM dns-vip-prepender: Setting dhclient to prepend DNS VIP in resolv.conf
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: prepend domain-name-servers 10.19.134.14;
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2616]: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134.14'
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: NM dns-vip-prepender: Looking for 'search qe1.kni.lab.eng.bos.redhat.com' in /etc/resolv.conf to place 'nameserver 10.19.134>
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: # Generated by NetworkManager
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: # Generated by NetworkManager
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: search qe1.kni.lab.eng.bos.redhat.com
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: nameserver 10.19.134.14
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: nameserver 10.19.42.41
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: nameserver 10.11.5.19
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: <13>Jun 19 19:21:59 root: nameserver 10.5.30.160
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: search qe1.kni.lab.eng.bos.redhat.com
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: nameserver 10.19.134.14
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: nameserver 10.19.42.41
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: nameserver 10.11.5.19
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com root[2618]: nameserver 10.5.30.160
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: req:5 'dhcp4-change' [eno2]: start running ordered scripts...
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1529] device (eno2): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1532] device (eno2): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1535] manager: NetworkManager state is now CONNECTED_SITE
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1546] device (eno2): Activation: successful, device activated.
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1551] manager: NetworkManager state is now CONNECTED_GLOBAL
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com NetworkManager[2096]: <info>  [1560972119.1558] manager: startup complete
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: req:9 'up' [eno2]: new request (3 scripts)
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: req:10 'connectivity-change': new request (3 scripts)
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com systemd[1]: Started Network Manager Wait Online.
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com nm-dispatcher[2302]: req:6 'up' [eno1]: start running ordered scripts...
Jun 19 19:21:59 openshift-master-2.qe1.kni.lab.eng.bos.redhat.com systemd[1]: Reached target Network is Online.
dhellmann commented 5 years ago

Is this still an issue?

russellb commented 5 years ago

Given the age of the issue, I think this can be closed. Feel free to open if i've missed something.