search

openshift-metal3 / dev-scripts

Scripts to automate development/test setup for openshift integration with https://github.com/metal3-io/
Apache License 2.0
93 stars 185 forks source link

240 pods after installation in +-6 different states, 46GB host swapping #291

Closed jhutar closed 5 years ago

jhutar commented 5 years ago

After installation that passed, there is too many pods it too many different states like (I particularly dislike "Terminating" and "OOMKilled" :)).

$ oc --config ocp/auth/kubeconfig get pods --all-namespaces
NAMESPACE                                               NAME                                                              READY   STATUS        RESTARTS   AGE
kube-system                                             etcd-member-master-0                                              1/1     Running       0          17h
kube-system                                             etcd-member-master-1                                              1/1     Running       0          17h
kube-system                                             etcd-member-master-2                                              1/1     Running       0          17h
openshift-apiserver-operator                            openshift-apiserver-operator-7b8c99bb8b-7dnlk                     1/1     Running       1          17h
openshift-apiserver                                     apiserver-jmjrs                                                   1/1     Running       0          17h
openshift-apiserver                                     apiserver-mzkx6                                                   1/1     Running       0          17h
openshift-apiserver                                     apiserver-p2c7x                                                   1/1     Running       0          17h
openshift-authentication-operator                       openshift-authentication-operator-bb8775754-9s258                 1/1     Running       0          17h
openshift-authentication                                openshift-authentication-677f9f678-2djk7                          1/1     Running       0          17h
openshift-authentication                                openshift-authentication-677f9f678-b6p4p                          1/1     Running       0          17h
openshift-cloud-credential-operator                     cloud-credential-operator-5cf49888b5-ccktg                        1/1     Running       0          17h
openshift-cluster-machine-approver                      machine-approver-6cf997dbcc-g25cx                                 1/1     Running       0          17h
openshift-cluster-node-tuning-operator                  cluster-node-tuning-operator-cb6dd6bcb-2c72p                      1/1     Running       0          17h
openshift-cluster-node-tuning-operator                  tuned-6djpt                                                       1/1     Running       0          17h
openshift-cluster-node-tuning-operator                  tuned-dx6sl                                                       1/1     Running       0          17h
openshift-cluster-node-tuning-operator                  tuned-zwkng                                                       1/1     Running       0          17h
openshift-cluster-samples-operator                      cluster-samples-operator-749c4b7dc7-zc5hz                         1/1     Running       0          17h
openshift-cluster-storage-operator                      cluster-storage-operator-7d7fcb7b56-xzhfm                         1/1     Running       0          17h
openshift-cluster-version                               cluster-version-operator-56c74d99b9-4qxhf                         1/1     Running       0          17h
openshift-console-operator                              console-operator-589ddb9775-tll9h                                 1/1     Running       0          17h
openshift-console                                       console-595b47967-nqqkz                                           1/1     Terminating   1          17h
openshift-console                                       console-6d8db4c7df-b5gwh                                          1/1     Running       1          17h
openshift-console                                       console-6d8db4c7df-dqnn7                                          0/1     Terminating   0          17h
openshift-console                                       console-6d8db4c7df-vlkrd                                          1/1     Running       0          17h
openshift-console                                       downloads-7748c8d856-2bhqw                                        1/1     Running       0          17h
openshift-console                                       downloads-7748c8d856-wd56w                                        1/1     Running       0          17h
openshift-controller-manager-operator                   openshift-controller-manager-operator-5f78855946-xxj86            1/1     Running       1          17h
openshift-controller-manager                            controller-manager-fj5hk                                          1/1     Running       0          17h
openshift-controller-manager                            controller-manager-m5vhr                                          1/1     Running       0          17h
openshift-controller-manager                            controller-manager-tpdkm                                          1/1     Running       0          17h
openshift-dns-operator                                  dns-operator-6f9d679b9c-fqj8s                                     1/1     Running       0          17h
openshift-dns                                           dns-default-9t87q                                                 2/2     Running       0          17h
openshift-dns                                           dns-default-d942l                                                 2/2     Running       0          17h
openshift-dns                                           dns-default-pb227                                                 2/2     Running       0          17h
openshift-image-registry                                cluster-image-registry-operator-86885f6c8d-4csbf                  1/1     Running       0          17h
openshift-image-registry                                cluster-image-registry-operator-86885f6c8d-vvpzn                  1/1     Terminating   0          17h
openshift-ingress-operator                              ingress-operator-7f8dcf7bb9-hvk9r                                 1/1     Running       0          17h
openshift-ingress                                       router-default-55f4fcfd66-774t6                                   1/1     Running       0          17h
openshift-ingress                                       router-default-55f4fcfd66-k4gz2                                   0/1     Pending       0          17h
openshift-ingress                                       router-default-55f4fcfd66-phlq7                                   1/1     Terminating   0          17h
openshift-ingress                                       router-default-55f4fcfd66-z555w                                   1/1     Running       0          17h
openshift-kube-apiserver-operator                       kube-apiserver-operator-6976f454fb-r5tkj                          1/1     Running       1          17h
openshift-kube-apiserver                                installer-1-master-0                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-1-master-1                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-2-master-0                                              0/1     OOMKilled     0          17h
openshift-kube-apiserver                                installer-2-master-1                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-2-master-2                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-3-master-0                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-3-master-1                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-4-master-0                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-4-master-1                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-4-master-2                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-5-master-0                                              0/1     OOMKilled     0          17h
openshift-kube-apiserver                                installer-6-master-0                                              0/1     Completed     0          17h
openshift-kube-apiserver                                installer-6-master-1                                              1/1     Running       0          17h
openshift-kube-apiserver                                kube-apiserver-master-0                                           2/2     Running       0          17h
openshift-kube-apiserver                                kube-apiserver-master-1                                           0/2     Init:0/2      0          17h
openshift-kube-apiserver                                kube-apiserver-master-2                                           2/2     Running       0          17h
openshift-kube-apiserver                                revision-pruner-1-master-0                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-1-master-1                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-2-master-0                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-2-master-1                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-2-master-2                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-3-master-0                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-3-master-1                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-4-master-0                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-4-master-1                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-4-master-2                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-5-master-0                                        0/1     Completed     0          17h
openshift-kube-apiserver                                revision-pruner-6-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager-operator              kube-controller-manager-operator-7566b748b8-7wk4l                 1/1     Running       1          17h
openshift-kube-controller-manager                       installer-1-master-0                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-2-master-0                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-3-master-0                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-4-master-0                                              0/1     OOMKilled     0          17h
openshift-kube-controller-manager                       installer-5-master-0                                              0/1     OOMKilled     0          17h
openshift-kube-controller-manager                       installer-5-master-1                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-5-master-2                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-6-master-0                                              0/1     OOMKilled     0          17h
openshift-kube-controller-manager                       installer-6-master-1                                              0/1     Completed     0          17h
openshift-kube-controller-manager                       installer-6-master-2                                              0/1     OOMKilled     0          17h
openshift-kube-controller-manager                       kube-controller-manager-master-0                                  1/1     Running       2          17h
openshift-kube-controller-manager                       kube-controller-manager-master-1                                  1/1     Running       3          17h
openshift-kube-controller-manager                       kube-controller-manager-master-2                                  1/1     Running       0          17h
openshift-kube-controller-manager                       revision-pruner-1-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-2-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-3-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-4-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-5-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-5-master-1                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-5-master-2                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-6-master-0                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-6-master-1                                        0/1     Completed     0          17h
openshift-kube-controller-manager                       revision-pruner-6-master-2                                        0/1     Completed     0          17h
openshift-kube-scheduler-operator                       openshift-kube-scheduler-operator-cd7fd87ff-7jhlk                 1/1     Running       1          17h
openshift-kube-scheduler                                installer-1-master-0                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-1-master-1                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-1-master-2                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-2-master-0                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-2-master-1                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-3-master-0                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-3-master-1                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-3-master-2                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-4-master-0                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-4-master-1                                              0/1     Completed     0          17h
openshift-kube-scheduler                                installer-4-master-2                                              0/1     Completed     0          17h
openshift-kube-scheduler                                openshift-kube-scheduler-master-0                                 1/1     Running       0          17h
openshift-kube-scheduler                                openshift-kube-scheduler-master-1                                 1/1     Running       3          17h
openshift-kube-scheduler                                openshift-kube-scheduler-master-2                                 1/1     Running       3          17h
openshift-kube-scheduler                                revision-pruner-1-master-0                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-1-master-1                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-1-master-2                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-2-master-0                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-3-master-0                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-3-master-1                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-3-master-2                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-4-master-0                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-4-master-1                                        0/1     Completed     0          17h
openshift-kube-scheduler                                revision-pruner-4-master-2                                        0/1     Completed     0          17h
openshift-machine-api                                   cluster-autoscaler-operator-6dd695cc7d-v2rx6                      1/1     Running       1          17h
openshift-machine-api                                   clusterapi-manager-controllers-5cf47db544-vg7dr                   4/4     Running       0          17h
openshift-machine-api                                   machine-api-operator-bd5f59899-z46pj                              1/1     Running       1          17h
openshift-machine-api                                   metalkube-baremetal-operator-8f9c66c86-mpq5t                      0/1     Terminating   0          17h
openshift-machine-api                                   metalkube-baremetal-operator-8f9c66c86-scgzw                      1/1     Running       0          17h
openshift-machine-config-operator                       machine-config-operator-767fcfcf74-5w5mv                          1/1     Running       1          17h
openshift-marketplace                                   certified-operators-5997774686-hkg4h                              1/1     Running       0          17h
openshift-marketplace                                   community-operators-7fbb7c4588-bbklz                              1/1     Running       0          17h
openshift-marketplace                                   marketplace-operator-78d556c764-l2zgv                             1/1     Running       0          17h
openshift-marketplace                                   redhat-operators-6b4f995b78-5csgz                                 1/1     Running       0          17h
openshift-monitoring                                    alertmanager-main-0                                               0/3     Pending       0          17h
openshift-monitoring                                    alertmanager-main-1                                               3/3     Running       0          17h
openshift-monitoring                                    cluster-monitoring-operator-5c68c9d967-th5qk                      1/1     Running       0          17h
openshift-monitoring                                    grafana-74876d8b8d-4qwqf                                          0/2     Preempting    0          17h
openshift-monitoring                                    grafana-74876d8b8d-6h8zm                                          2/2     Running       0          17h
openshift-monitoring                                    kube-state-metrics-56d947b89d-dfk66                               3/3     Terminating   0          17h
openshift-monitoring                                    kube-state-metrics-56d947b89d-l6jlz                               3/3     Running       0          17h
openshift-monitoring                                    node-exporter-4jjkc                                               2/2     Running       0          17h
openshift-monitoring                                    node-exporter-jsrzm                                               2/2     Running       0          17h
openshift-monitoring                                    node-exporter-kv4bb                                               2/2     Running       0          17h
openshift-monitoring                                    prometheus-adapter-c76d6596f-82vjk                                1/1     Running       0          17h
openshift-monitoring                                    prometheus-adapter-c76d6596f-lcnfv                                1/1     Running       0          17h
openshift-monitoring                                    prometheus-adapter-c76d6596f-q5kwq                                1/1     Terminating   0          17h
openshift-monitoring                                    prometheus-k8s-0                                                  0/6     Pending       0          17h
openshift-monitoring                                    prometheus-k8s-1                                                  6/6     Terminating   1          17h
openshift-monitoring                                    prometheus-operator-6ff74c9976-rmqvq                              1/1     Running       0          17h
openshift-monitoring                                    prometheus-operator-6ff74c9976-x52g6                              1/1     Terminating   1          17h
openshift-monitoring                                    telemeter-client-6579d7cf8-q7mkk                                  3/3     Running       0          17h
openshift-multus                                        multus-8j6kb                                                      1/1     Running       0          17h
openshift-multus                                        multus-98kcc                                                      1/1     Running       0          17h
openshift-multus                                        multus-nzm94                                                      1/1     Running       0          17h
openshift-network-operator                              network-operator-56b8ccdcbb-xd9q5                                 1/1     Running       0          17h
openshift-operator-lifecycle-manager                    catalog-operator-6865f8bd88-hwqth                                 1/1     Running       0          17h
openshift-operator-lifecycle-manager                    olm-operator-fccbd8798-9x77k                                      1/1     Running       0          17h
openshift-operator-lifecycle-manager                    olm-operators-zhkk2                                               1/1     Running       0          17h
openshift-operator-lifecycle-manager                    packageserver-544b89c886-8ldtr                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-54569d7c8f-8j9sv                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-5486cc45fd-nqwvt                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-556df8d845-gcn7w                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-55b74df5bb-m2cqx                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-5647ff4c9b-z78cm                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-5668b7bd5-ssvrf                                     0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-5687977596-db522                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-56ccf9988b-f5rzq                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-576dddbcf6-f4fk8                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-58d68cfc8b-lpz5m                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-58f796cb5-zcfk8                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-596f49576c-zj282                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-599489cfbf-5tng4                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-5bc68c459b-mlpsg                                    1/1     Running       0          13h
openshift-operator-lifecycle-manager                    packageserver-5bc68c459b-mqxvq                                    1/1     Running       0          13h
openshift-operator-lifecycle-manager                    packageserver-5c548f4fb5-h6tpf                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-5c6c4f598b-xh7hv                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-5cc868cff6-kz2gb                                    0/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-5cdfd89cf9-r7bqn                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-5df46449fb-58pdm                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-5dfcd9fb56-fgmlc                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-6484b96c7-lxbnq                                     1/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-64cbcff7cb-chrv9                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-64ff8c5b89-vnpj4                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-658d685bdf-t4j7v                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-65fb76c755-qgh47                                    0/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-66845969c7-qg7ks                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-669b56f59-xz8xb                                     0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-6798cc6565-49dlp                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-68475586b6-tx6wm                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-687768b948-q9l9t                                    0/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-68bdb47898-lh6bm                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-68cf9fd6bf-dlbn9                                    0/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-69488c7665-zbvnt                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-6988754f9c-t7rp2                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-699795f5c6-22lxq                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-6b59bdb4d7-2mgjc                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-6d6548469d-k7jth                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-6d8446c695-z5n4h                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-6d89c547c9-m8bk8                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-6f769fdc56-nqc8g                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-74cd8b5574-cg8gp                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-76648b7798-tpfsn                                    0/1     Terminating   0          17h
openshift-operator-lifecycle-manager                    packageserver-76d4b8dbc4-frmd5                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-77678bbdc-s6pd8                                     0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7869f5c94f-vj6xn                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-78d7685975-jwg4k                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-79d9f749bd-28h7x                                    0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-79ff46ccc5-kwpwg                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-7b5bbc9db6-xhpdb                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7bbf6b9f9c-klrst                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-7bdf97c5d5-nzbfz                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7c58db8f46-kh9db                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7c64c778c7-gzlsd                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7cf476b5b4-xpnm5                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-7dcc548b54-vvt6j                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-845c6bbf44-ppzmv                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-84b6f97575-s962t                                    0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-84c9cbb78c-bjskn                                    0/1     Terminating   0          13h
openshift-operator-lifecycle-manager                    packageserver-85985d844c-prpfw                                    0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-867448d9-f8cfq                                      0/1     Terminating   0          16h
openshift-operator-lifecycle-manager                    packageserver-8b74bd6d8-f8vk9                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-b48c575c8-257jl                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-b89585667-srqhn                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-c4dd86cc9-tppd6                                     0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-c78858db-blkc8                                      0/1     Terminating   0          14h
openshift-operator-lifecycle-manager                    packageserver-c9f694678-fn8cd                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-cc84784b6-nqnwr                                     0/1     Terminating   0          15h
openshift-operator-lifecycle-manager                    packageserver-cf694654d-c724r                                     0/1     Terminating   0          17h
openshift-sdn                                           ovs-7p2qf                                                         1/1     Running       0          17h
openshift-sdn                                           ovs-k8qbx                                                         1/1     Running       0          17h
openshift-sdn                                           ovs-z7wwv                                                         1/1     Running       0          17h
openshift-sdn                                           sdn-7xlxr                                                         1/1     Running       0          17h
openshift-sdn                                           sdn-controller-jfmj5                                              1/1     Running       0          17h
openshift-sdn                                           sdn-controller-nfmkf                                              1/1     Running       0          17h
openshift-sdn                                           sdn-controller-sl2g4                                              1/1     Running       0          17h
openshift-sdn                                           sdn-sjskt                                                         1/1     Running       2          17h
openshift-sdn                                           sdn-t8bc8                                                         1/1     Running       1          17h
openshift-service-ca-operator                           openshift-service-ca-operator-7957dd76c9-p7425                    1/1     Running       0          17h
openshift-service-ca                                    apiservice-cabundle-injector-6589bb696b-phqmw                     1/1     Running       0          17h
openshift-service-ca                                    configmap-cabundle-injector-787f7f684b-z78kn                      1/1     Running       0          17h
openshift-service-ca                                    service-serving-cert-signer-58f9487f4f-5stwv                      1/1     Running       0          17h
openshift-service-catalog-apiserver-operator            openshift-service-catalog-apiserver-operator-84d5b596c7-wghlc     1/1     Running       1          17h
openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-77cf59cj8   1/1     Running       1          17h
jhutar commented 5 years ago

Logs with internal hostname replaced by "HOSTNAME" string. Let me know in next ~3 days if you would like to see the actual system.

dev-scripts-make.log 01_install_requirements-2019-04-03-094035.log 02_configure_host-2019-04-03-094759.log 03_ocp_repo_sync-2019-04-03-094937.log 04_setup_ironic-2019-04-03-095417.log 04_setup_ironic-2019-04-03-095418.log 05_build_ocp_installer-2019-04-03-101309.log 06_create_cluster-2019-04-03-101407.log

hardys commented 5 years ago

Can you confirm if you had https://github.com/openshift-metalkube/dev-scripts/pull/254 applied please?

We were definitely seeing OOMKilled prior to that, but we probably need to look at the AWS flavors if the increase from #254 wasn't enough.

Also note you can re-test this with more memory by modifying tripleo-quickstart-config/metalkube-nodes.yml - it's a tough compromize because some folks want to do minimal testing on memory contrained hosts, and some want to do more realistic tests on a box with plenty of spare resources.

jhutar commented 5 years ago

$ git log | grep d16dcdeca841b9ce291156c67a1ba47a9b2b8c98

Looks like I do not have it. Will pull, make clean and make. Thank you!
jhutar commented 5 years ago

Now while make is running...

@hardys Problem I had was that even the host itself swapped something which means that on a host with 46GB RAM I can not increase VM's memory.

hardys commented 5 years ago

Ack thanks, looking at the openshift/installer code it looks like the default instance type for AWS is m4/m5 xlarge, which means 16GB and 4vCPU

Regarding the swapping, can you perhaps check if there are any other workloads consuming resources on the host? I've been testing on a 32G host and it's not swapping, my assumption was that KSM was doing a good job of sharing duplicate pages between the VMs

jhutar commented 5 years ago

That was a clean up2date CentOS7 installation - nothing else was running on the host. Only thing I have created was:

oc --config ocp/auth/kubeconfig new-app https://github.com/OpenShiftDemos/os-sample-python.git

I'll take a look at the closer memory consumption once make finishes.

jhutar commented 5 years ago

OK, make is still running this loop:

level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
karmab commented 5 years ago

this is an issue that is now solved. i suggest you retry from scratch and feel free to increment memory and virtual cpus for the master nodes so that all pods can run

jhutar commented 5 years ago

After new installation I have:

[kni@hp-dl360gen8-01 dev-scripts]$ oc --config ocp/auth/kubeconfig get pods --all-namespaces | wc -l
3447
[kni@hp-dl360gen8-01 dev-scripts]$ oc --config ocp/auth/kubeconfig get pods --all-namespaces | grep -v Running | wc -l
3339

I assume it is not healthy state? How to debug that please?

hardys commented 5 years ago

There's not enough information here to propose any fix, so I'm going to close this. If you're still having issues please either raise another issue with more details, or jump onto slack and we can talk about the steps to debug, thanks!