100% complete, waiting on authentication, console

deanpeterson commented 5 years ago

I get the following error installing with libvirt settings:

FATAL failed to initialize the cluster: Working towards 4.2.0-0.okd-2019-08-29-022631: 100% complete, waiting on authentication, console

I am using a RHEL 8 host machine.

wking commented 5 years ago

Attach oc adm must-gather logs?

rphillips commented 5 years ago

There are some special DNS settings you need for the console. Are these settings setup?

troubleshooting

deanpeterson commented 5 years ago

There are some special DNS settings you need for the console. Are these settings setup?

troubleshooting

I followed the instructions at the time and did this:

echo server=/abecorn.enterprisewebservice/192.168.126.1 | sudo tee /etc/NetworkManager/dnsmasq.d/openshift.conf

how will adding address=/.apps.abecorn.enterprisewebservice/192.168.126.51 help?

How will I know the worker node will get 192.168.126.51 as its ip?

It failed with the same error.

deanpeterson commented 5 years ago

Attach oc adm must-gather logs?

[root@23-25-149-229-static bin]# ./oc adm must-gather --config install-dir/auth/kubeconfig
Using image: registry.svc.ci.openshift.org/origin/4.2-2019-08-30-023930@sha256:99353133df86a6baacaeaeab602f0fc9638c8abc79ae768e4d516bb2fe4a0a3a
namespace/openshift-must-gather-bkvwj created
clusterrolebinding.rbac.authorization.k8s.io/must-gather-g5xk2 created
2019/08/30 05:08:45 Finished successfully with no errors.
2019/08/30 05:08:46 Gathering data for ns/openshift-cluster-version...
2019/08/30 05:08:46     Collecting resources for namespace "openshift-cluster-version"...
2019/08/30 05:08:46     Gathering pod data for namespace "openshift-cluster-version"...
2019/08/30 05:08:46         Gathering data for pod "cluster-version-operator-6c47bf78cd-xplhp"
2019/08/30 05:08:48         Unable to gather previous container logs: previous terminated container "cluster-version-operator" in pod "cluster-version-operator-6c47bf78cd-xplhp" not found
2019/08/30 05:08:48         Skipping container endpoint collection for pod "cluster-version-operator-6c47bf78cd-xplhp" container "cluster-version-operator": No ports
2019/08/30 05:08:55 Finished successfully with no errors.
2019/08/30 05:08:55 Gathering config.openshift.io resource data...
2019/08/30 05:08:57 Gathering kubeapiserver.operator.openshift.io resource data...
2019/08/30 05:08:57 Gathering cluster operator resource data...
2019/08/30 05:08:57     Gathering related object reference information for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "authentications.operator.openshift.io/cluster" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "authentications.config.openshift.io/cluster" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "infrastructures.config.openshift.io/cluster" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "oauths.config.openshift.io/cluster" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "namespaces/openshift-config" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "namespaces/openshift-config-managed" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "namespaces/openshift-authentication" for ClusterOperator "authentication"...
2019/08/30 05:08:57     Found related object "namespaces/openshift-authentication-operator" for ClusterOperator "authentication"...
2019/08/30 05:09:21 Gathering data for ns/openshift-config...
2019/08/30 05:09:21     Collecting resources for namespace "openshift-config"...
2019/08/30 05:09:21     Gathering pod data for namespace "openshift-config"...
2019/08/30 05:09:27 Gathering data for ns/openshift-config-managed...
2019/08/30 05:09:27     Collecting resources for namespace "openshift-config-managed"...
2019/08/30 05:09:27     Gathering pod data for namespace "openshift-config-managed"...
2019/08/30 05:09:35 Gathering data for ns/openshift-authentication...
2019/08/30 05:09:36     Collecting resources for namespace "openshift-authentication"...
2019/08/30 05:09:36     Gathering pod data for namespace "openshift-authentication"...
2019/08/30 05:09:36         Gathering data for pod "oauth-openshift-75d8969cb-5k5rf"
2019/08/30 05:09:36         Skipping container data collection for pod "oauth-openshift-75d8969cb-5k5rf": Pod not running
2019/08/30 05:09:36         Gathering data for pod "oauth-openshift-75d8969cb-kqd26"
2019/08/30 05:09:36         Skipping container data collection for pod "oauth-openshift-75d8969cb-kqd26": Pod not running
2019/08/30 05:09:36         Gathering data for pod "oauth-openshift-75d8969cb-sk2g6"
2019/08/30 05:09:36         Unable to gather previous container logs: previous terminated container "oauth-openshift" in pod "oauth-openshift-75d8969cb-sk2g6" not found
2019/08/30 05:09:37         Gathering data for pod "oauth-openshift-7cf95f4dbf-4qwzk"
2019/08/30 05:09:37         Skipping container data collection for pod "oauth-openshift-7cf95f4dbf-4qwzk": Pod not running
2019/08/30 05:09:42 Gathering data for ns/openshift-authentication-operator...
2019/08/30 05:09:42     Collecting resources for namespace "openshift-authentication-operator"...
2019/08/30 05:09:42     Gathering pod data for namespace "openshift-authentication-operator"...
2019/08/30 05:09:42         Gathering data for pod "authentication-operator-679d566b79-fkgkg"
2019/08/30 05:09:43         Unable to gather previous container logs: previous terminated container "operator" in pod "authentication-operator-679d566b79-fkgkg" not found
2019/08/30 05:09:43         Skipping container endpoint collection for pod "authentication-operator-679d566b79-fkgkg" container "operator": No ports
2019/08/30 05:09:52 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:09:52 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:09:52 Gathering cluster operator resource data...
2019/08/30 05:09:52     Gathering related object reference information for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "namespaces/openshift-cloud-credential-operator" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/cloud-credential-operator-iam-ro" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-image-registry" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-image-registry-azure" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-image-registry-gcs" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-image-registry-openstack" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-ingress" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-ingress-azure" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-ingress-gcp" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-machine-api-aws" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-machine-api-azure" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-machine-api-gcp" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52     Found related object "CredentialsRequest.cloudcredential.openshift.io/openshift-machine-api-openstack" for ClusterOperator "cloud-credential"...
2019/08/30 05:09:52 Gathering data for ns/openshift-cloud-credential-operator...
2019/08/30 05:09:52     Collecting resources for namespace "openshift-cloud-credential-operator"...
2019/08/30 05:09:52     Gathering pod data for namespace "openshift-cloud-credential-operator"...
2019/08/30 05:09:52         Gathering data for pod "cloud-credential-operator-5d5b8c86d8-w7pdq"
E0830 05:09:53.545516      32 portforward.go:331] an error occurred forwarding 37587 -> 9876: error forwarding port 9876 to pod 6a530e88bfc54c37d8e9733ae10e500114fa00e3bae5aa0ce33fd0aff3c3dadc, uid : exit status 1: 2019/08/30 05:09:53 socat[18923] E connect(5, AF=2 127.0.0.1:9876, 16): Connection refused
E0830 05:09:53.693908      32 portforward.go:331] an error occurred forwarding 37587 -> 9876: error forwarding port 9876 to pod 6a530e88bfc54c37d8e9733ae10e500114fa00e3bae5aa0ce33fd0aff3c3dadc, uid : exit status 1: 2019/08/30 05:09:53 socat[18939] E connect(5, AF=2 127.0.0.1:9876, 16): Connection refused
E0830 05:09:53.772836      32 portforward.go:331] an error occurred forwarding 37587 -> 9876: error forwarding port 9876 to pod 6a530e88bfc54c37d8e9733ae10e500114fa00e3bae5aa0ce33fd0aff3c3dadc, uid : exit status 1: 2019/08/30 05:09:53 socat[18946] E connect(5, AF=2 127.0.0.1:9876, 16): Connection refused
2019/08/30 05:11:30 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:11:30 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:11:30 Gathering cluster operator resource data...
2019/08/30 05:11:30     Gathering related object reference information for ClusterOperator "cluster-autoscaler"...
2019/08/30 05:11:30     Found related object "namespaces/openshift-machine-api" for ClusterOperator "cluster-autoscaler"...
2019/08/30 05:11:30 Gathering data for ns/openshift-machine-api...
2019/08/30 05:11:30     Collecting resources for namespace "openshift-machine-api"...
2019/08/30 05:11:30     Gathering pod data for namespace "openshift-machine-api"...
2019/08/30 05:11:30         Gathering data for pod "cluster-autoscaler-operator-56fd4f8b66-zrhwt"
2019/08/30 05:11:30         Unable to gather previous container logs: previous terminated container "cluster-autoscaler-operator" in pod "cluster-autoscaler-operator-56fd4f8b66-zrhwt" not found
2019/08/30 05:11:31         Gathering data for pod "machine-api-controllers-64976d866f-txhbs"
2019/08/30 05:11:31         Unable to gather previous container logs: previous terminated container "controller-manager" in pod "machine-api-controllers-64976d866f-txhbs" not found
2019/08/30 05:11:31         Skipping container endpoint collection for pod "machine-api-controllers-64976d866f-txhbs" container "controller-manager": No ports
2019/08/30 05:11:31         Unable to gather previous container logs: previous terminated container "machine-controller" in pod "machine-api-controllers-64976d866f-txhbs" not found
2019/08/30 05:11:31         Skipping container endpoint collection for pod "machine-api-controllers-64976d866f-txhbs" container "machine-controller": No ports
2019/08/30 05:11:31         Unable to gather previous container logs: previous terminated container "nodelink-controller" in pod "machine-api-controllers-64976d866f-txhbs" not found
2019/08/30 05:11:31         Skipping container endpoint collection for pod "machine-api-controllers-64976d866f-txhbs" container "nodelink-controller": No ports
2019/08/30 05:11:31         Gathering data for pod "machine-api-operator-594fcb5d95-jzg8d"
E0830 05:11:31.767315      32 portforward.go:316] error copying from local connection to remote stream: EOF
2019/08/30 05:11:38 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:11:38 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:11:38 Gathering cluster operator resource data...
2019/08/30 05:11:38     Gathering related object reference information for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "consoles.operator.openshift.io/cluster" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "consoles.config.openshift.io/cluster" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "infrastructures.config.openshift.io/cluster" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "proxies.config.openshift.io/cluster" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "oauthclients.oauth.openshift.io/console" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "namespaces/openshift-console-operator" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "namespaces/openshift-console" for ClusterOperator "console"...
2019/08/30 05:11:38     Found related object "configmaps/console-public" for ClusterOperator "console"...
2019/08/30 05:12:11 Gathering data for ns/openshift-console-operator...
2019/08/30 05:12:11     Collecting resources for namespace "openshift-console-operator"...
2019/08/30 05:12:11     Gathering pod data for namespace "openshift-console-operator"...
2019/08/30 05:12:11         Gathering data for pod "console-operator-7cbbf95d76-jw97k"
2019/08/30 05:12:11         Skipping container data collection for pod "console-operator-7cbbf95d76-jw97k": Pod not running
2019/08/30 05:12:11         Gathering data for pod "console-operator-7cbbf95d76-l97nx"
2019/08/30 05:12:11         Unable to gather previous container logs: previous terminated container "console-operator" in pod "console-operator-7cbbf95d76-l97nx" not found
E0830 05:12:11.740802      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 1688a29a8ee99a95337095c9b1bda4e8b7f0e7df8b2e8d256e19a74f822c1497, uid : exit status 1: 2019/08/30 05:12:11 socat[23922] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:12:11.804787      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 1688a29a8ee99a95337095c9b1bda4e8b7f0e7df8b2e8d256e19a74f822c1497, uid : exit status 1: 2019/08/30 05:12:11 socat[23929] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:12:11.866803      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 1688a29a8ee99a95337095c9b1bda4e8b7f0e7df8b2e8d256e19a74f822c1497, uid : exit status 1: 2019/08/30 05:12:11 socat[23936] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
2019/08/30 05:12:17 Gathering data for ns/openshift-console...
2019/08/30 05:12:17     Collecting resources for namespace "openshift-console"...
2019/08/30 05:12:17     Gathering pod data for namespace "openshift-console"...
2019/08/30 05:12:17         Gathering data for pod "console-6d96fdb575-42cr5"
2019/08/30 05:12:17         Skipping container data collection for pod "console-6d96fdb575-42cr5": Pod not running
2019/08/30 05:12:17         Gathering data for pod "console-6d96fdb575-jfqfp"
2019/08/30 05:12:17         Skipping container data collection for pod "console-6d96fdb575-jfqfp": Pod not running
2019/08/30 05:12:17         Gathering data for pod "console-8444846b74-cfrhh"
2019/08/30 05:12:17         Skipping container data collection for pod "console-8444846b74-cfrhh": Pod not running
2019/08/30 05:12:17         Gathering data for pod "downloads-f9ff65658-b5pw7"
2019/08/30 05:12:17         Unable to gather previous container logs: previous terminated container "download-server" in pod "downloads-f9ff65658-b5pw7" not found
2019/08/30 05:12:17         Gathering data for pod "downloads-f9ff65658-lhfvx"
2019/08/30 05:12:17         Unable to gather previous container logs: previous terminated container "download-server" in pod "downloads-f9ff65658-lhfvx" not found
E0830 05:12:17.598841      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:12:22 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:12:22 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:12:22 Gathering cluster operator resource data...
2019/08/30 05:12:22     Gathering related object reference information for ClusterOperator "dns"...
2019/08/30 05:12:22     Found related object "namespaces/openshift-dns-operator" for ClusterOperator "dns"...
2019/08/30 05:12:22     Found related object "namespaces/openshift-dns" for ClusterOperator "dns"...
2019/08/30 05:12:22     Found related object "DNS.operator.openshift.io/default" for ClusterOperator "dns"...
2019/08/30 05:12:22 Gathering data for ns/openshift-dns-operator...
2019/08/30 05:12:22     Collecting resources for namespace "openshift-dns-operator"...
2019/08/30 05:12:22     Gathering pod data for namespace "openshift-dns-operator"...
2019/08/30 05:12:22         Gathering data for pod "dns-operator-7d7964d5c-92ttg"
2019/08/30 05:12:22         Unable to gather previous container logs: previous terminated container "dns-operator" in pod "dns-operator-7d7964d5c-92ttg" not found
2019/08/30 05:12:22         Skipping container endpoint collection for pod "dns-operator-7d7964d5c-92ttg" container "dns-operator": No ports
2019/08/30 05:12:33 Gathering data for ns/openshift-dns...
2019/08/30 05:12:33     Collecting resources for namespace "openshift-dns"...
2019/08/30 05:12:33     Gathering pod data for namespace "openshift-dns"...
2019/08/30 05:12:33         Gathering data for pod "dns-default-fc9v6"
2019/08/30 05:12:33         Unable to gather previous container logs: previous terminated container "dns" in pod "dns-default-fc9v6" not found
2019/08/30 05:12:40         Unable to gather previous container logs: previous terminated container "dns-node-resolver" in pod "dns-default-fc9v6" not found
2019/08/30 05:12:40         Skipping container endpoint collection for pod "dns-default-fc9v6" container "dns-node-resolver": No ports
2019/08/30 05:12:40         Gathering data for pod "dns-default-xm2qs"
2019/08/30 05:12:40         Unable to gather previous container logs: previous terminated container "dns" in pod "dns-default-xm2qs" not found
2019/08/30 05:12:48         Unable to gather previous container logs: previous terminated container "dns-node-resolver" in pod "dns-default-xm2qs" not found
2019/08/30 05:12:48         Skipping container endpoint collection for pod "dns-default-xm2qs" container "dns-node-resolver": No ports
2019/08/30 05:13:02 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:13:02 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:13:02 Gathering cluster operator resource data...
2019/08/30 05:13:02     Gathering related object reference information for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "clusterroles.rbac.authorization.k8s.io/system:registry" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "clusterrolebindings.rbac.authorization.k8s.io/registry-registry-role" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "serviceaccounts/registry" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "configmaps/serviceca" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "configmaps/image-registry-certificates" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "secrets/image-registry-private-configuration" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "images.config.openshift.io/cluster" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "daemonsets.apps/node-ca" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "services/image-registry" for ClusterOperator "image-registry"...
2019/08/30 05:13:02     Found related object "deployments.apps/image-registry" for ClusterOperator "image-registry"...
2019/08/30 05:13:21 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:13:21 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:13:21 Gathering cluster operator resource data...
2019/08/30 05:13:21     Gathering related object reference information for ClusterOperator "ingress"...
2019/08/30 05:13:21     Found related object "namespaces/openshift-ingress-operator" for ClusterOperator "ingress"...
2019/08/30 05:13:21     Found related object "namespaces/openshift-ingress" for ClusterOperator "ingress"...
2019/08/30 05:13:21     Found related object "IngressController.operator.openshift.io/default" for ClusterOperator "ingress"...
2019/08/30 05:13:21 Gathering data for ns/openshift-ingress-operator...
2019/08/30 05:13:21     Collecting resources for namespace "openshift-ingress-operator"...
2019/08/30 05:13:21     Gathering pod data for namespace "openshift-ingress-operator"...
2019/08/30 05:13:21         Gathering data for pod "ingress-operator-656758bf4b-p6r9m"
2019/08/30 05:13:21         Unable to gather previous container logs: previous terminated container "ingress-operator" in pod "ingress-operator-656758bf4b-p6r9m" not found
2019/08/30 05:13:21         Skipping container endpoint collection for pod "ingress-operator-656758bf4b-p6r9m" container "ingress-operator": No ports
2019/08/30 05:13:28 Gathering data for ns/openshift-ingress...
2019/08/30 05:13:28     Collecting resources for namespace "openshift-ingress"...
2019/08/30 05:13:28     Gathering pod data for namespace "openshift-ingress"...
2019/08/30 05:13:28         Gathering data for pod "router-default-6bd4f956f-p8hwj"
2019/08/30 05:13:28         Skipping container data collection for pod "router-default-6bd4f956f-p8hwj": Pod not running
2019/08/30 05:13:28         Gathering data for pod "router-default-6bd4f956f-pfxvd"
2019/08/30 05:13:28         Unable to gather previous container logs: previous terminated container "router" in pod "router-default-6bd4f956f-pfxvd" not found
2019/08/30 05:13:40 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:13:40 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:13:40 Gathering cluster operator resource data...
2019/08/30 05:13:41     Gathering related object reference information for ClusterOperator "insights"...
2019/08/30 05:13:41     Found related object "namespaces/openshift-insights" for ClusterOperator "insights"...
2019/08/30 05:13:41     Found related object "deployments.apps/insights-operator" for ClusterOperator "insights"...
2019/08/30 05:13:41 Gathering data for ns/openshift-insights...
2019/08/30 05:13:41     Collecting resources for namespace "openshift-insights"...
2019/08/30 05:13:41     Gathering pod data for namespace "openshift-insights"...
2019/08/30 05:13:41         Gathering data for pod "insights-operator-68d67f8466-shrbn"
2019/08/30 05:13:41         Unable to gather previous container logs: previous terminated container "operator" in pod "insights-operator-68d67f8466-shrbn" not found
2019/08/30 05:13:42         Gathering data for pod "insights-operator-68d67f8466-zg9hb"
2019/08/30 05:13:42         Skipping container data collection for pod "insights-operator-68d67f8466-zg9hb": Pod not running
2019/08/30 05:13:50 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:13:50 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:13:50 Gathering cluster operator resource data...
2019/08/30 05:13:50     Gathering related object reference information for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:50     Found related object "kubeapiservers.operator.openshift.io/cluster" for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:50     Found related object "namespaces/openshift-config" for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:50     Found related object "namespaces/openshift-config-managed" for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:50     Found related object "namespaces/openshift-kube-apiserver-operator" for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:50     Found related object "namespaces/openshift-kube-apiserver" for ClusterOperator "kube-apiserver"...
2019/08/30 05:13:59 Gathering data for ns/openshift-kube-apiserver-operator...
2019/08/30 05:13:59     Collecting resources for namespace "openshift-kube-apiserver-operator"...
2019/08/30 05:13:59     Gathering pod data for namespace "openshift-kube-apiserver-operator"...
2019/08/30 05:13:59         Gathering data for pod "kube-apiserver-operator-78476cd5fc-b89sc"
2019/08/30 05:14:07 Gathering data for ns/openshift-kube-apiserver...
2019/08/30 05:14:07     Collecting resources for namespace "openshift-kube-apiserver"...
2019/08/30 05:14:07     Gathering pod data for namespace "openshift-kube-apiserver"...
2019/08/30 05:14:07         Gathering data for pod "installer-2-ocp-n76nx-master-0"
2019/08/30 05:14:07         Skipping container data collection for pod "installer-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:07         Gathering data for pod "installer-3-ocp-n76nx-master-0"
2019/08/30 05:14:07         Skipping container data collection for pod "installer-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:07         Gathering data for pod "installer-4-ocp-n76nx-master-0"
2019/08/30 05:14:07         Skipping container data collection for pod "installer-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:07         Gathering data for pod "installer-5-ocp-n76nx-master-0"
2019/08/30 05:14:07         Skipping container data collection for pod "installer-5-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:07         Gathering data for pod "installer-6-ocp-n76nx-master-0"
2019/08/30 05:14:07         Skipping container data collection for pod "installer-6-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:07         Gathering data for pod "kube-apiserver-ocp-n76nx-master-0"
2019/08/30 05:14:08         Unable to gather previous container logs: previous terminated container "kube-apiserver-6" in pod "kube-apiserver-ocp-n76nx-master-0" not found
2019/08/30 05:14:08         Unable to gather previous container logs: previous terminated container "kube-apiserver-cert-syncer-6" in pod "kube-apiserver-ocp-n76nx-master-0" not found
2019/08/30 05:14:08         Skipping container endpoint collection for pod "kube-apiserver-ocp-n76nx-master-0" container "kube-apiserver-cert-syncer-6": No ports
2019/08/30 05:14:08         Unable to gather previous container logs: previous terminated container "kube-apiserver-insecure-readyz-6" in pod "kube-apiserver-ocp-n76nx-master-0" not found
2019/08/30 05:14:08         Unable to gather previous container logs: previous terminated container "setup" in pod "kube-apiserver-ocp-n76nx-master-0" not found
2019/08/30 05:14:08         Skipping container endpoint collection for pod "kube-apiserver-ocp-n76nx-master-0" container "setup": No ports
2019/08/30 05:14:08         Gathering data for pod "revision-pruner-2-ocp-n76nx-master-0"
2019/08/30 05:14:08         Skipping container data collection for pod "revision-pruner-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:08         Gathering data for pod "revision-pruner-3-ocp-n76nx-master-0"
2019/08/30 05:14:08         Skipping container data collection for pod "revision-pruner-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:08         Gathering data for pod "revision-pruner-4-ocp-n76nx-master-0"
2019/08/30 05:14:08         Skipping container data collection for pod "revision-pruner-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:08         Gathering data for pod "revision-pruner-5-ocp-n76nx-master-0"
2019/08/30 05:14:08         Skipping container data collection for pod "revision-pruner-5-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:08         Gathering data for pod "revision-pruner-6-ocp-n76nx-master-0"
2019/08/30 05:14:08         Skipping container data collection for pod "revision-pruner-6-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:16 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:14:16 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:14:16 Gathering cluster operator resource data...
2019/08/30 05:14:16     Gathering related object reference information for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:16     Found related object "kubecontrollermanagers.operator.openshift.io/cluster" for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:16     Found related object "namespaces/openshift-config" for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:16     Found related object "namespaces/openshift-config-managed" for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:16     Found related object "namespaces/openshift-kube-controller-manager" for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:16     Found related object "namespaces/openshift-kube-controller-manager-operator" for ClusterOperator "kube-controller-manager"...
2019/08/30 05:14:22 Gathering data for ns/openshift-kube-controller-manager...
2019/08/30 05:14:22     Collecting resources for namespace "openshift-kube-controller-manager"...
2019/08/30 05:14:22     Gathering pod data for namespace "openshift-kube-controller-manager"...
2019/08/30 05:14:22         Gathering data for pod "installer-2-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "installer-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:22         Gathering data for pod "installer-3-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "installer-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:22         Gathering data for pod "installer-4-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "installer-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:22         Gathering data for pod "kube-controller-manager-ocp-n76nx-master-0"
2019/08/30 05:14:22         Unable to gather previous container logs: previous terminated container "kube-controller-manager-cert-syncer-4" in pod "kube-controller-manager-ocp-n76nx-master-0" not found
2019/08/30 05:14:22         Skipping container endpoint collection for pod "kube-controller-manager-ocp-n76nx-master-0" container "kube-controller-manager-cert-syncer-4": No ports
2019/08/30 05:14:22         Unable to gather previous container logs: previous terminated container "wait-for-host-port" in pod "kube-controller-manager-ocp-n76nx-master-0" not found
2019/08/30 05:14:22         Skipping container endpoint collection for pod "kube-controller-manager-ocp-n76nx-master-0" container "wait-for-host-port": No ports
2019/08/30 05:14:22         Gathering data for pod "revision-pruner-2-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "revision-pruner-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:22         Gathering data for pod "revision-pruner-3-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "revision-pruner-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:22         Gathering data for pod "revision-pruner-4-ocp-n76nx-master-0"
2019/08/30 05:14:22         Skipping container data collection for pod "revision-pruner-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:30 Gathering data for ns/openshift-kube-controller-manager-operator...
2019/08/30 05:14:30     Collecting resources for namespace "openshift-kube-controller-manager-operator"...
2019/08/30 05:14:30     Gathering pod data for namespace "openshift-kube-controller-manager-operator"...
2019/08/30 05:14:30         Gathering data for pod "kube-controller-manager-operator-5995667999-567rh"
2019/08/30 05:14:37 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:14:37 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:14:37 Gathering cluster operator resource data...
2019/08/30 05:14:37     Gathering related object reference information for ClusterOperator "kube-scheduler"...
2019/08/30 05:14:37     Found related object "kubeschedulers.operator.openshift.io/cluster" for ClusterOperator "kube-scheduler"...
2019/08/30 05:14:37     Found related object "namespaces/openshift-config" for ClusterOperator "kube-scheduler"...
2019/08/30 05:14:37     Found related object "namespaces/openshift-kube-scheduler" for ClusterOperator "kube-scheduler"...
2019/08/30 05:14:37     Found related object "namespaces/openshift-kube-scheduler-operator" for ClusterOperator "kube-scheduler"...
2019/08/30 05:14:42 Gathering data for ns/openshift-kube-scheduler...
2019/08/30 05:14:42     Collecting resources for namespace "openshift-kube-scheduler"...
2019/08/30 05:14:42     Gathering pod data for namespace "openshift-kube-scheduler"...
2019/08/30 05:14:42         Gathering data for pod "installer-2-ocp-n76nx-master-0"
2019/08/30 05:14:42         Skipping container data collection for pod "installer-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:42         Gathering data for pod "installer-3-ocp-n76nx-master-0"
2019/08/30 05:14:42         Skipping container data collection for pod "installer-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:42         Gathering data for pod "installer-4-ocp-n76nx-master-0"
2019/08/30 05:14:42         Skipping container data collection for pod "installer-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:42         Gathering data for pod "installer-5-ocp-n76nx-master-0"
2019/08/30 05:14:42         Skipping container data collection for pod "installer-5-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:42         Gathering data for pod "openshift-kube-scheduler-ocp-n76nx-master-0"
2019/08/30 05:14:43         Unable to gather previous container logs: previous terminated container "wait-for-host-port" in pod "openshift-kube-scheduler-ocp-n76nx-master-0" not found
2019/08/30 05:14:43         Skipping container endpoint collection for pod "openshift-kube-scheduler-ocp-n76nx-master-0" container "wait-for-host-port": No ports
2019/08/30 05:14:43         Gathering data for pod "revision-pruner-2-ocp-n76nx-master-0"
2019/08/30 05:14:43         Skipping container data collection for pod "revision-pruner-2-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:43         Gathering data for pod "revision-pruner-3-ocp-n76nx-master-0"
2019/08/30 05:14:43         Skipping container data collection for pod "revision-pruner-3-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:43         Gathering data for pod "revision-pruner-4-ocp-n76nx-master-0"
2019/08/30 05:14:43         Skipping container data collection for pod "revision-pruner-4-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:43         Gathering data for pod "revision-pruner-5-ocp-n76nx-master-0"
2019/08/30 05:14:43         Skipping container data collection for pod "revision-pruner-5-ocp-n76nx-master-0": Pod not running
2019/08/30 05:14:49 Gathering data for ns/openshift-kube-scheduler-operator...
2019/08/30 05:14:49     Collecting resources for namespace "openshift-kube-scheduler-operator"...
2019/08/30 05:14:49     Gathering pod data for namespace "openshift-kube-scheduler-operator"...
2019/08/30 05:14:49         Gathering data for pod "openshift-kube-scheduler-operator-596c57db79-lgddk"
2019/08/30 05:14:49         Skipping container endpoint collection for pod "openshift-kube-scheduler-operator-596c57db79-lgddk" container "kube-scheduler-operator-container": No ports
2019/08/30 05:14:57 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:14:57 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:14:57 Gathering cluster operator resource data...
2019/08/30 05:14:57     Gathering related object reference information for ClusterOperator "machine-api"...
2019/08/30 05:14:57     Found related object "namespaces/openshift-machine-api" for ClusterOperator "machine-api"...
2019/08/30 05:14:57 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:14:57 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:14:57 Gathering cluster operator resource data...
2019/08/30 05:14:57     Gathering related object reference information for ClusterOperator "machine-config"...
2019/08/30 05:14:57     Found related object "namespaces/openshift-machine-config-operator" for ClusterOperator "machine-config"...
2019/08/30 05:14:57     Found related object "machineconfigpools.machineconfiguration.openshift.io/master" for ClusterOperator "machine-config"...
2019/08/30 05:14:57     Found related object "machineconfigpools.machineconfiguration.openshift.io/worker" for ClusterOperator "machine-config"...
2019/08/30 05:14:57     Found related object "controllerconfigs.machineconfiguration.openshift.io/machine-config-controller" for ClusterOperator "machine-config"...
2019/08/30 05:14:57 Gathering data for ns/openshift-machine-config-operator...
2019/08/30 05:14:57     Collecting resources for namespace "openshift-machine-config-operator"...
2019/08/30 05:14:57     Gathering pod data for namespace "openshift-machine-config-operator"...
2019/08/30 05:14:57         Gathering data for pod "etcd-quorum-guard-bf7845696-86m7t"
2019/08/30 05:14:57         Skipping container data collection for pod "etcd-quorum-guard-bf7845696-86m7t": Pod not running
2019/08/30 05:14:57         Gathering data for pod "etcd-quorum-guard-bf7845696-sn86t"
2019/08/30 05:14:57         Skipping container data collection for pod "etcd-quorum-guard-bf7845696-sn86t": Pod not running
2019/08/30 05:14:57         Gathering data for pod "etcd-quorum-guard-bf7845696-z99xx"
2019/08/30 05:14:57         Unable to gather previous container logs: previous terminated container "guard" in pod "etcd-quorum-guard-bf7845696-z99xx" not found
2019/08/30 05:14:57         Skipping container endpoint collection for pod "etcd-quorum-guard-bf7845696-z99xx" container "guard": No ports
2019/08/30 05:14:57         Gathering data for pod "machine-config-controller-854dcf874c-kn5j5"
2019/08/30 05:14:57         Skipping container endpoint collection for pod "machine-config-controller-854dcf874c-kn5j5" container "machine-config-controller": No ports
2019/08/30 05:14:57         Gathering data for pod "machine-config-daemon-r6stm"
2019/08/30 05:14:57         Unable to gather previous container logs: previous terminated container "machine-config-daemon" in pod "machine-config-daemon-r6stm" not found
2019/08/30 05:14:57         Skipping container endpoint collection for pod "machine-config-daemon-r6stm" container "machine-config-daemon": No ports
2019/08/30 05:14:57         Gathering data for pod "machine-config-daemon-vhd94"
2019/08/30 05:14:58         Unable to gather previous container logs: previous terminated container "machine-config-daemon" in pod "machine-config-daemon-vhd94" not found
2019/08/30 05:14:58         Skipping container endpoint collection for pod "machine-config-daemon-vhd94" container "machine-config-daemon": No ports
2019/08/30 05:14:58         Gathering data for pod "machine-config-operator-6fb5f4dfc6-8fxvc"
2019/08/30 05:14:58         Skipping container endpoint collection for pod "machine-config-operator-6fb5f4dfc6-8fxvc" container "machine-config-operator": No ports
2019/08/30 05:14:58         Gathering data for pod "machine-config-server-t7cct"
2019/08/30 05:14:58         Unable to gather previous container logs: previous terminated container "machine-config-server" in pod "machine-config-server-t7cct" not found
2019/08/30 05:14:58         Skipping container endpoint collection for pod "machine-config-server-t7cct" container "machine-config-server": No ports
2019/08/30 05:15:20 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:15:20 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:15:20 Gathering cluster operator resource data...
2019/08/30 05:15:20     Gathering related object reference information for ClusterOperator "marketplace"...
2019/08/30 05:15:20     Found related object "namespaces/openshift-marketplace" for ClusterOperator "marketplace"...
2019/08/30 05:15:20     Found related object "OperatorSource.operators.coreos.com" for ClusterOperator "marketplace"...
2019/08/30 05:15:20     Found related object "CatalogSourceConfig.operators.coreos.com" for ClusterOperator "marketplace"...
2019/08/30 05:15:20     Found related object "CatalogSource.operators.coreos.com" for ClusterOperator "marketplace"...
2019/08/30 05:15:20 Gathering data for ns/openshift-marketplace...
2019/08/30 05:15:20     Collecting resources for namespace "openshift-marketplace"...
2019/08/30 05:15:20     Gathering pod data for namespace "openshift-marketplace"...
2019/08/30 05:15:20         Gathering data for pod "certified-operators-549b8668cd-tzxnv"
2019/08/30 05:15:20         Unable to gather previous container logs: previous terminated container "certified-operators" in pod "certified-operators-549b8668cd-tzxnv" not found
2019/08/30 05:15:20         Gathering data for pod "community-operators-6b67cd49cb-k6l9m"
2019/08/30 05:15:20         Unable to gather previous container logs: previous terminated container "community-operators" in pod "community-operators-6b67cd49cb-k6l9m" not found
2019/08/30 05:15:20         Gathering data for pod "marketplace-operator-866f588dc6-sz5q2"
2019/08/30 05:15:20         Unable to gather previous container logs: previous terminated container "marketplace-operator" in pod "marketplace-operator-866f588dc6-sz5q2" not found
E0830 05:15:21.143711      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 588441235bd764339c5025bc56072254ce39e2fc77005a83ac9385fcfb47ca3e, uid : exit status 1: 2019/08/30 05:15:21 socat[30313] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:15:21.229785      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 588441235bd764339c5025bc56072254ce39e2fc77005a83ac9385fcfb47ca3e, uid : exit status 1: 2019/08/30 05:15:21 socat[30320] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:15:21.852169      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 588441235bd764339c5025bc56072254ce39e2fc77005a83ac9385fcfb47ca3e, uid : exit status 1: 2019/08/30 05:15:21 socat[30337] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
2019/08/30 05:15:21         Gathering data for pod "redhat-operators-79bd89d844-fxfvs"
2019/08/30 05:15:21         Unable to gather previous container logs: previous terminated container "redhat-operators" in pod "redhat-operators-79bd89d844-fxfvs" not found
2019/08/30 05:15:59 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:15:59 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:15:59 Gathering cluster operator resource data...
2019/08/30 05:15:59     Gathering related object reference information for ClusterOperator "monitoring"...
2019/08/30 05:15:59     Found related object "monitoring.operator.openshift.io/cluster" for ClusterOperator "monitoring"...
2019/08/30 05:15:59     Found related object "namespaces/openshift-monitoring" for ClusterOperator "monitoring"...
2019/08/30 05:16:14 Gathering data for ns/openshift-monitoring...
2019/08/30 05:16:14     Collecting resources for namespace "openshift-monitoring"...
2019/08/30 05:16:14     Gathering pod data for namespace "openshift-monitoring"...
2019/08/30 05:16:14         Gathering data for pod "alertmanager-main-0"
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "alertmanager" in pod "alertmanager-main-0" not found
E0830 05:16:14.183927      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 3dcc0528e6154e688c407deb83a6e0a65fa776dc587a8557ce459d597d53f671, uid : exit status 1: 2019/08/30 05:16:14 socat[87807] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:14.240374      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 3dcc0528e6154e688c407deb83a6e0a65fa776dc587a8557ce459d597d53f671, uid : exit status 1: 2019/08/30 05:16:14 socat[87814] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:14.298726      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 3dcc0528e6154e688c407deb83a6e0a65fa776dc587a8557ce459d597d53f671, uid : exit status 1: 2019/08/30 05:16:14 socat[87821] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "config-reloader" in pod "alertmanager-main-0" not found
2019/08/30 05:16:14         Skipping container endpoint collection for pod "alertmanager-main-0" container "config-reloader": No ports
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "alertmanager-proxy" in pod "alertmanager-main-0" not found
2019/08/30 05:16:14         Gathering data for pod "alertmanager-main-1"
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "alertmanager" in pod "alertmanager-main-1" not found
E0830 05:16:14.633281      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 1a9fbf4af43b59511e92618b657b249c7d0ffff086939b0a1657a13755449e6d, uid : exit status 1: 2019/08/30 05:16:14 socat[87851] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:14.687590      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 1a9fbf4af43b59511e92618b657b249c7d0ffff086939b0a1657a13755449e6d, uid : exit status 1: 2019/08/30 05:16:14 socat[87859] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:14.760667      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod 1a9fbf4af43b59511e92618b657b249c7d0ffff086939b0a1657a13755449e6d, uid : exit status 1: 2019/08/30 05:16:14 socat[87866] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "config-reloader" in pod "alertmanager-main-1" not found
2019/08/30 05:16:14         Skipping container endpoint collection for pod "alertmanager-main-1" container "config-reloader": No ports
2019/08/30 05:16:14         Unable to gather previous container logs: previous terminated container "alertmanager-proxy" in pod "alertmanager-main-1" not found
2019/08/30 05:16:15         Gathering data for pod "alertmanager-main-2"
2019/08/30 05:16:15         Unable to gather previous container logs: previous terminated container "alertmanager" in pod "alertmanager-main-2" not found
E0830 05:16:15.319140      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod febf1195979215307553fa0cb335e176d02520810a8403f7ac93e61ee4e4f400, uid : exit status 1: 2019/08/30 05:16:15 socat[87950] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:15.403760      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod febf1195979215307553fa0cb335e176d02520810a8403f7ac93e61ee4e4f400, uid : exit status 1: 2019/08/30 05:16:15 socat[87957] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
E0830 05:16:15.470558      32 portforward.go:331] an error occurred forwarding 37587 -> 9094: error forwarding port 9094 to pod febf1195979215307553fa0cb335e176d02520810a8403f7ac93e61ee4e4f400, uid : exit status 1: 2019/08/30 05:16:15 socat[87964] E connect(5, AF=2 127.0.0.1:9094, 16): Connection refused
2019/08/30 05:16:15         Unable to gather previous container logs: previous terminated container "config-reloader" in pod "alertmanager-main-2" not found
2019/08/30 05:16:15         Skipping container endpoint collection for pod "alertmanager-main-2" container "config-reloader": No ports
2019/08/30 05:16:15         Unable to gather previous container logs: previous terminated container "alertmanager-proxy" in pod "alertmanager-main-2" not found
2019/08/30 05:16:16         Gathering data for pod "cluster-monitoring-operator-7d68ff9d9b-l9b8k"
2019/08/30 05:16:16         Unable to gather previous container logs: previous terminated container "cluster-monitoring-operator" in pod "cluster-monitoring-operator-7d68ff9d9b-l9b8k" not found
2019/08/30 05:16:16         Gathering data for pod "grafana-5b4896f7bd-vvxts"
2019/08/30 05:16:16         Unable to gather previous container logs: previous terminated container "grafana" in pod "grafana-5b4896f7bd-vvxts" not found
2019/08/30 05:16:17         Unable to gather previous container logs: previous terminated container "grafana-proxy" in pod "grafana-5b4896f7bd-vvxts" not found
2019/08/30 05:16:17         Gathering data for pod "kube-state-metrics-5f9cd4b54d-h2nm6"
2019/08/30 05:16:17         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy-main" in pod "kube-state-metrics-5f9cd4b54d-h2nm6" not found
2019/08/30 05:16:17         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy-self" in pod "kube-state-metrics-5f9cd4b54d-h2nm6" not found
2019/08/30 05:16:18         Unable to gather previous container logs: previous terminated container "kube-state-metrics" in pod "kube-state-metrics-5f9cd4b54d-h2nm6" not found
2019/08/30 05:16:18         Skipping container endpoint collection for pod "kube-state-metrics-5f9cd4b54d-h2nm6" container "kube-state-metrics": No ports
2019/08/30 05:16:18         Gathering data for pod "node-exporter-jmjrz"
2019/08/30 05:16:18         Unable to gather previous container logs: previous terminated container "node-exporter" in pod "node-exporter-jmjrz" not found
2019/08/30 05:16:18         Skipping container endpoint collection for pod "node-exporter-jmjrz" container "node-exporter": No ports
2019/08/30 05:16:19         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy" in pod "node-exporter-jmjrz" not found
2019/08/30 05:16:19         Gathering data for pod "node-exporter-n4cqw"
2019/08/30 05:16:19         Unable to gather previous container logs: previous terminated container "node-exporter" in pod "node-exporter-n4cqw" not found
2019/08/30 05:16:19         Skipping container endpoint collection for pod "node-exporter-n4cqw" container "node-exporter": No ports
2019/08/30 05:16:19         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy" in pod "node-exporter-n4cqw" not found
2019/08/30 05:16:20         Gathering data for pod "openshift-state-metrics-fccc95544-mgz6j"
E0830 05:16:20.419934      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:16:20         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy-main" in pod "openshift-state-metrics-fccc95544-mgz6j" not found
2019/08/30 05:16:20         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy-self" in pod "openshift-state-metrics-fccc95544-mgz6j" not found
2019/08/30 05:16:21         Unable to gather previous container logs: previous terminated container "openshift-state-metrics" in pod "openshift-state-metrics-fccc95544-mgz6j" not found
2019/08/30 05:16:21         Skipping container endpoint collection for pod "openshift-state-metrics-fccc95544-mgz6j" container "openshift-state-metrics": No ports
2019/08/30 05:16:21         Gathering data for pod "prometheus-adapter-c59d9597b-gdpxz"
2019/08/30 05:16:21         Unable to gather previous container logs: previous terminated container "prometheus-adapter" in pod "prometheus-adapter-c59d9597b-gdpxz" not found
2019/08/30 05:16:21         Gathering data for pod "prometheus-adapter-c59d9597b-npkhv"
2019/08/30 05:16:21         Unable to gather previous container logs: previous terminated container "prometheus-adapter" in pod "prometheus-adapter-c59d9597b-npkhv" not found
2019/08/30 05:16:22         Gathering data for pod "prometheus-k8s-0"
2019/08/30 05:16:22         Skipping container endpoint collection for pod "prometheus-k8s-0" container "prometheus": No ports
2019/08/30 05:16:22         Unable to gather previous container logs: previous terminated container "prometheus-config-reloader" in pod "prometheus-k8s-0" not found
2019/08/30 05:16:22         Skipping container endpoint collection for pod "prometheus-k8s-0" container "prometheus-config-reloader": No ports
2019/08/30 05:16:23         Unable to gather previous container logs: previous terminated container "rules-configmap-reloader" in pod "prometheus-k8s-0" not found
2019/08/30 05:16:23         Skipping container endpoint collection for pod "prometheus-k8s-0" container "rules-configmap-reloader": No ports
2019/08/30 05:16:23         Unable to gather previous container logs: previous terminated container "prometheus-proxy" in pod "prometheus-k8s-0" not found
2019/08/30 05:16:23         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy" in pod "prometheus-k8s-0" not found
2019/08/30 05:16:24         Unable to gather previous container logs: previous terminated container "prom-label-proxy" in pod "prometheus-k8s-0" not found
2019/08/30 05:16:24         Skipping container endpoint collection for pod "prometheus-k8s-0" container "prom-label-proxy": No ports
2019/08/30 05:16:24         Gathering data for pod "prometheus-k8s-1"
2019/08/30 05:16:24         Skipping container endpoint collection for pod "prometheus-k8s-1" container "prometheus": No ports
2019/08/30 05:16:25         Unable to gather previous container logs: previous terminated container "prometheus-config-reloader" in pod "prometheus-k8s-1" not found
2019/08/30 05:16:25         Skipping container endpoint collection for pod "prometheus-k8s-1" container "prometheus-config-reloader": No ports
2019/08/30 05:16:25         Unable to gather previous container logs: previous terminated container "rules-configmap-reloader" in pod "prometheus-k8s-1" not found
2019/08/30 05:16:25         Skipping container endpoint collection for pod "prometheus-k8s-1" container "rules-configmap-reloader": No ports
2019/08/30 05:16:26         Unable to gather previous container logs: previous terminated container "prometheus-proxy" in pod "prometheus-k8s-1" not found
2019/08/30 05:16:27         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy" in pod "prometheus-k8s-1" not found
E0830 05:16:27.878001      32 portforward.go:316] error copying from local connection to remote stream: EOF
2019/08/30 05:16:27         Unable to gather previous container logs: previous terminated container "prom-label-proxy" in pod "prometheus-k8s-1" not found
2019/08/30 05:16:27         Skipping container endpoint collection for pod "prometheus-k8s-1" container "prom-label-proxy": No ports
2019/08/30 05:16:27         Gathering data for pod "prometheus-operator-796cfff58-c74pg"
2019/08/30 05:16:27         Skipping container data collection for pod "prometheus-operator-796cfff58-c74pg": Pod not running
2019/08/30 05:16:27         Gathering data for pod "prometheus-operator-cb87ddff6-r842w"
2019/08/30 05:16:27         Unable to gather previous container logs: previous terminated container "prometheus-operator" in pod "prometheus-operator-cb87ddff6-r842w" not found
2019/08/30 05:16:29         Gathering data for pod "telemeter-client-59655d88b5-sz7tr"
2019/08/30 05:16:29         Unable to gather previous container logs: previous terminated container "telemeter-client" in pod "telemeter-client-59655d88b5-sz7tr" not found
E0830 05:16:29.536378      32 portforward.go:316] error copying from local connection to remote stream: EOF
E0830 05:16:30.081475      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:16:30         Unable to gather previous container logs: previous terminated container "reload" in pod "telemeter-client-59655d88b5-sz7tr" not found
2019/08/30 05:16:30         Skipping container endpoint collection for pod "telemeter-client-59655d88b5-sz7tr" container "reload": No ports
2019/08/30 05:16:30         Unable to gather previous container logs: previous terminated container "kube-rbac-proxy" in pod "telemeter-client-59655d88b5-sz7tr" not found
2019/08/30 05:16:40 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:16:40 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:16:40 Gathering cluster operator resource data...
2019/08/30 05:16:40     Gathering related object reference information for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "configmaps/applied-cluster" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "customresourcedefinitions.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "namespaces/openshift-multus" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterroles.rbac.authorization.k8s.io/multus" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "serviceaccounts/multus" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterrolebindings.rbac.authorization.k8s.io/multus" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "daemonsets.apps/multus" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "services/multus-admission-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterroles.rbac.authorization.k8s.io/multus-admission-controller-webhook" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterrolebindings.rbac.authorization.k8s.io/multus-admission-controller-webhook" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "validatingwebhookconfigurations.admissionregistration.k8s.io/multus.openshift.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "configmaps/openshift-service-ca" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "daemonsets.apps/multus-admission-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "namespaces/openshift-sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "customresourcedefinitions.apiextensions.k8s.io/clusternetworks.network.openshift.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "customresourcedefinitions.apiextensions.k8s.io/hostsubnets.network.openshift.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "customresourcedefinitions.apiextensions.k8s.io/netnamespaces.network.openshift.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "customresourcedefinitions.apiextensions.k8s.io/egressnetworkpolicies.network.openshift.io" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterroles.rbac.authorization.k8s.io/openshift-sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "serviceaccounts/sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterrolebindings.rbac.authorization.k8s.io/openshift-sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "serviceaccounts/sdn-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterroles.rbac.authorization.k8s.io/openshift-sdn-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusterrolebindings.rbac.authorization.k8s.io/openshift-sdn-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "roles.rbac.authorization.k8s.io/openshift-sdn-controller-leaderelection" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "rolebindings.rbac.authorization.k8s.io/openshift-sdn-controller-leaderelection" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "clusternetworks.network.openshift.io/default" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "prometheusrules.monitoring.coreos.com/networking-rules" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "daemonsets.apps/sdn-controller" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "servicemonitors.monitoring.coreos.com/monitor-sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "services/sdn" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "roles.rbac.authorization.k8s.io/prometheus-k8s" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "rolebindings.rbac.authorization.k8s.io/prometheus-k8s" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "daemonsets.apps/ovs" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "configmaps/sdn-config" for ClusterOperator "network"...
2019/08/30 05:16:40     Found related object "daemonsets.apps/sdn" for ClusterOperator "network"...
2019/08/30 05:16:46 Gathering data for ns/openshift-multus...
2019/08/30 05:16:46     Collecting resources for namespace "openshift-multus"...
2019/08/30 05:16:46     Gathering pod data for namespace "openshift-multus"...
2019/08/30 05:16:46         Gathering data for pod "multus-admission-controller-vs9jt"
2019/08/30 05:16:46         Unable to gather previous container logs: previous terminated container "multus-admission-controller" in pod "multus-admission-controller-vs9jt" not found
2019/08/30 05:16:46         Skipping container endpoint collection for pod "multus-admission-controller-vs9jt" container "multus-admission-controller": No ports
2019/08/30 05:16:46         Gathering data for pod "multus-rxgr8"
2019/08/30 05:16:46         Unable to gather previous container logs: previous terminated container "kube-multus" in pod "multus-rxgr8" not found
2019/08/30 05:16:46         Skipping container endpoint collection for pod "multus-rxgr8" container "kube-multus": No ports
2019/08/30 05:16:46         Unable to gather previous container logs: previous terminated container "cni-plugins" in pod "multus-rxgr8" not found
2019/08/30 05:16:46         Skipping container endpoint collection for pod "multus-rxgr8" container "cni-plugins": No ports
2019/08/30 05:16:46         Gathering data for pod "multus-sq9qn"
2019/08/30 05:16:46         Unable to gather previous container logs: previous terminated container "kube-multus" in pod "multus-sq9qn" not found
2019/08/30 05:16:46         Skipping container endpoint collection for pod "multus-sq9qn" container "kube-multus": No ports
2019/08/30 05:16:46         Unable to gather previous container logs: previous terminated container "cni-plugins" in pod "multus-sq9qn" not found
2019/08/30 05:16:46         Skipping container endpoint collection for pod "multus-sq9qn" container "cni-plugins": No ports
2019/08/30 05:17:36 Gathering data for ns/openshift-sdn...
2019/08/30 05:17:36     Collecting resources for namespace "openshift-sdn"...
2019/08/30 05:17:36     Gathering pod data for namespace "openshift-sdn"...
2019/08/30 05:17:36         Gathering data for pod "ovs-d8fn2"
2019/08/30 05:17:36         Unable to gather previous container logs: previous terminated container "openvswitch" in pod "ovs-d8fn2" not found
2019/08/30 05:17:36         Skipping container endpoint collection for pod "ovs-d8fn2" container "openvswitch": No ports
2019/08/30 05:17:36         Gathering data for pod "ovs-x9dj2"
2019/08/30 05:17:36         Unable to gather previous container logs: previous terminated container "openvswitch" in pod "ovs-x9dj2" not found
2019/08/30 05:17:36         Skipping container endpoint collection for pod "ovs-x9dj2" container "openvswitch": No ports
2019/08/30 05:17:36         Gathering data for pod "sdn-c7h95"
2019/08/30 05:17:38         Unable to gather previous container logs: previous terminated container "sdn" in pod "sdn-c7h95" not found
2019/08/30 05:17:38         Unable to gather previous container logs: previous terminated container "install-cni-plugins" in pod "sdn-c7h95" not found
2019/08/30 05:17:38         Skipping container endpoint collection for pod "sdn-c7h95" container "install-cni-plugins": No ports
2019/08/30 05:17:38         Gathering data for pod "sdn-controller-gvgjv"
2019/08/30 05:17:38         Skipping container endpoint collection for pod "sdn-controller-gvgjv" container "sdn-controller": No ports
2019/08/30 05:17:38         Gathering data for pod "sdn-f5qr7"
2019/08/30 05:17:38         Unable to gather previous container logs: previous terminated container "sdn" in pod "sdn-f5qr7" not found
E0830 05:17:38.881719      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:17:39         Unable to gather previous container logs: previous terminated container "install-cni-plugins" in pod "sdn-f5qr7" not found
2019/08/30 05:17:39         Skipping container endpoint collection for pod "sdn-f5qr7" container "install-cni-plugins": No ports
2019/08/30 05:19:49 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:19:49 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:19:49 Gathering cluster operator resource data...
2019/08/30 05:19:49     Gathering related object reference information for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "namespaces/openshift-cluster-node-tuning-operator" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "Tuned.tuned.openshift.io/default" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "ServiceAccount/tuned" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "ClusterRole.rbac.authorization.k8s.io/cluster-node-tuning:tuned" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "ClusterRoleBinding.rbac.authorization.k8s.io/cluster-node-tuning:tuned" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "ConfigMap/tuned-profiles" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "ConfigMap/tuned-recommend" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49     Found related object "DaemonSet.apps/tuned" for ClusterOperator "node-tuning"...
2019/08/30 05:19:49 Gathering data for ns/openshift-cluster-node-tuning-operator...
2019/08/30 05:19:49     Collecting resources for namespace "openshift-cluster-node-tuning-operator"...
2019/08/30 05:19:49     Gathering pod data for namespace "openshift-cluster-node-tuning-operator"...
2019/08/30 05:19:49         Gathering data for pod "cluster-node-tuning-operator-6bf5fb9dd-wts7n"
2019/08/30 05:19:49         Unable to gather previous container logs: previous terminated container "cluster-node-tuning-operator" in pod "cluster-node-tuning-operator-6bf5fb9dd-wts7n" not found
2019/08/30 05:19:49         Skipping container endpoint collection for pod "cluster-node-tuning-operator-6bf5fb9dd-wts7n" container "cluster-node-tuning-operator": No ports
2019/08/30 05:19:49         Gathering data for pod "tuned-464bz"
2019/08/30 05:19:49         Unable to gather previous container logs: previous terminated container "tuned" in pod "tuned-464bz" not found
2019/08/30 05:19:49         Skipping container endpoint collection for pod "tuned-464bz" container "tuned": No ports
2019/08/30 05:19:49         Gathering data for pod "tuned-9lfl2"
2019/08/30 05:19:49         Unable to gather previous container logs: previous terminated container "tuned" in pod "tuned-9lfl2" not found
2019/08/30 05:19:49         Skipping container endpoint collection for pod "tuned-9lfl2" container "tuned": No ports
2019/08/30 05:20:23 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:20:23 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:20:23 Gathering cluster operator resource data...
2019/08/30 05:20:23     Gathering related object reference information for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "openshiftapiservers.operator.openshift.io/cluster" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "namespaces/openshift-config" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "namespaces/openshift-config-managed" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "namespaces/openshift-apiserver-operator" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "namespaces/openshift-apiserver" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.apps.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.authorization.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.build.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.image.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.oauth.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.project.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.quota.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.route.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.security.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.template.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:23     Found related object "apiservices.apiregistration.k8s.io/v1.user.openshift.io" for ClusterOperator "openshift-apiserver"...
2019/08/30 05:20:28 Gathering data for ns/openshift-apiserver-operator...
2019/08/30 05:20:28     Collecting resources for namespace "openshift-apiserver-operator"...
2019/08/30 05:20:28     Gathering pod data for namespace "openshift-apiserver-operator"...
2019/08/30 05:20:28         Gathering data for pod "openshift-apiserver-operator-7d8665c54b-hn7fh"
2019/08/30 05:20:42 Gathering data for ns/openshift-apiserver...
2019/08/30 05:20:42     Collecting resources for namespace "openshift-apiserver"...
2019/08/30 05:20:42     Gathering pod data for namespace "openshift-apiserver"...
2019/08/30 05:20:42         Gathering data for pod "apiserver-w4mmr"
2019/08/30 05:20:42         Unable to gather previous container logs: previous terminated container "openshift-apiserver" in pod "apiserver-w4mmr" not found
2019/08/30 05:20:43         Unable to gather previous container logs: previous terminated container "fix-audit-permissions" in pod "apiserver-w4mmr" not found
2019/08/30 05:20:43         Skipping container endpoint collection for pod "apiserver-w4mmr" container "fix-audit-permissions": No ports
2019/08/30 05:24:07 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:24:07 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:24:07 Gathering cluster operator resource data...
2019/08/30 05:24:07     Gathering related object reference information for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:07     Found related object "openshiftcontrollermanagers.operator.openshift.io/cluster" for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:07     Found related object "namespaces/openshift-config" for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:07     Found related object "namespaces/openshift-config-managed" for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:07     Found related object "namespaces/openshift-controller-manager-operator" for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:07     Found related object "namespaces/openshift-controller-manager" for ClusterOperator "openshift-controller-manager"...
2019/08/30 05:24:16 Gathering data for ns/openshift-controller-manager-operator...
2019/08/30 05:24:16     Collecting resources for namespace "openshift-controller-manager-operator"...
2019/08/30 05:24:16     Gathering pod data for namespace "openshift-controller-manager-operator"...
2019/08/30 05:24:16         Gathering data for pod "openshift-controller-manager-operator-5fb54c6cf5-9jk78"
2019/08/30 05:24:29 Gathering data for ns/openshift-controller-manager...
2019/08/30 05:24:29     Collecting resources for namespace "openshift-controller-manager"...
2019/08/30 05:24:29     Gathering pod data for namespace "openshift-controller-manager"...
2019/08/30 05:24:29         Gathering data for pod "controller-manager-rv4pc"
2019/08/30 05:24:29         Unable to gather previous container logs: previous terminated container "controller-manager" in pod "controller-manager-rv4pc" not found
2019/08/30 05:24:36 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:24:36 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:24:36 Gathering cluster operator resource data...
2019/08/30 05:24:36     Gathering related object reference information for ClusterOperator "openshift-samples"...
2019/08/30 05:24:36     Found related object "configs.samples.operator.openshift.io/cluster" for ClusterOperator "openshift-samples"...
2019/08/30 05:24:36     Found related object "namespaces/openshift-cluster-samples-operator" for ClusterOperator "openshift-samples"...
2019/08/30 05:24:36     Found related object "namespaces/openshift" for ClusterOperator "openshift-samples"...
2019/08/30 05:24:49 Gathering data for ns/openshift-cluster-samples-operator...
2019/08/30 05:24:50     Collecting resources for namespace "openshift-cluster-samples-operator"...
2019/08/30 05:24:50     Gathering pod data for namespace "openshift-cluster-samples-operator"...
2019/08/30 05:24:50         Gathering data for pod "cluster-samples-operator-96df65c4d-5m8pw"
2019/08/30 05:24:50         Unable to gather previous container logs: previous terminated container "cluster-samples-operator" in pod "cluster-samples-operator-96df65c4d-5m8pw" not found
E0830 05:24:50.568815      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 5c7e125bea56797b248f90f649ce2c35fd631e471f92f20c1a5ee70e36c5daef, uid : exit status 1: 2019/08/30 05:24:50 socat[49263] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:24:50.634128      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 5c7e125bea56797b248f90f649ce2c35fd631e471f92f20c1a5ee70e36c5daef, uid : exit status 1: 2019/08/30 05:24:50 socat[49270] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:24:52.104269      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod 5c7e125bea56797b248f90f649ce2c35fd631e471f92f20c1a5ee70e36c5daef, uid : exit status 1: 2019/08/30 05:24:52 socat[49288] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
2019/08/30 05:25:01 Gathering data for ns/openshift...
2019/08/30 05:25:01     Collecting resources for namespace "openshift"...
2019/08/30 05:25:01     Gathering pod data for namespace "openshift"...
2019/08/30 05:25:08 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:25:08 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:25:08 Gathering cluster operator resource data...
2019/08/30 05:25:08     Gathering related object reference information for ClusterOperator "operator-lifecycle-manager"...
2019/08/30 05:25:08     Found related object "OperatorGroup.operators.coreos.com/operator-lifecycle-manager" for ClusterOperator "operator-lifecycle-manager"...
2019/08/30 05:25:08     Found related object "ClusterServiceVersion.operators.coreos.com/operator-lifecycle-manager" for ClusterOperator "operator-lifecycle-manager"...
2019/08/30 05:25:08     Found related object "namespaces/openshift-operator-lifecycle-manager" for ClusterOperator "operator-lifecycle-manager"...
2019/08/30 05:25:37 Gathering data for ns/openshift-operator-lifecycle-manager...
2019/08/30 05:25:37     Collecting resources for namespace "openshift-operator-lifecycle-manager"...
2019/08/30 05:25:37     Gathering pod data for namespace "openshift-operator-lifecycle-manager"...
2019/08/30 05:25:37         Gathering data for pod "catalog-operator-8494bc69b7-lmkwr"
2019/08/30 05:25:50         Unable to gather previous container logs: previous terminated container "catalog-operator" in pod "catalog-operator-8494bc69b7-lmkwr" not found
E0830 05:25:51.871272      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:25:52         Gathering data for pod "olm-operator-75b7694687-t4krz"
E0830 05:25:52.394146      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:25:53         Unable to gather previous container logs: previous terminated container "olm-operator" in pod "olm-operator-75b7694687-t4krz" not found
E0830 05:25:53.759180      32 portforward.go:316] error copying from local connection to remote stream: tls: use of closed connection
2019/08/30 05:25:53         Gathering data for pod "packageserver-6bcffc6d9f-c2fcm"
2019/08/30 05:25:54         Gathering data for pod "packageserver-6bcffc6d9f-q8qfj"
2019/08/30 05:26:03 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:26:03 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:26:03 Gathering cluster operator resource data...
2019/08/30 05:26:03     Gathering related object reference information for ClusterOperator "operator-lifecycle-manager-catalog"...
2019/08/30 05:26:03     Found related object "Subscription.operators.coreos.com/operator-lifecycle-manager-catalog" for ClusterOperator "operator-lifecycle-manager-catalog"...
2019/08/30 05:26:03     Found related object "InstallPlan.operators.coreos.com/operator-lifecycle-manager-catalog" for ClusterOperator "operator-lifecycle-manager-catalog"...
2019/08/30 05:26:03     Found related object "namespaces/openshift-marketplace" for ClusterOperator "operator-lifecycle-manager-catalog"...
2019/08/30 05:26:23 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:26:23 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:26:23 Gathering cluster operator resource data...
2019/08/30 05:26:23     Gathering related object reference information for ClusterOperator "operator-lifecycle-manager-packageserver"...
2019/08/30 05:26:23     Found related object "namespaces/openshift-operator-lifecycle-manager" for ClusterOperator "operator-lifecycle-manager-packageserver"...
2019/08/30 05:26:23     Found related object "ClusterServiceVersion.operators.coreos.com/packageserver" for ClusterOperator "operator-lifecycle-manager-packageserver"...
2019/08/30 05:26:30 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:26:30 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:26:30 Gathering cluster operator resource data...
2019/08/30 05:26:30     Gathering related object reference information for ClusterOperator "service-ca"...
2019/08/30 05:26:30     Found related object "servicecas.operator.openshift.io/cluster" for ClusterOperator "service-ca"...
2019/08/30 05:26:30     Found related object "namespaces/openshift-config" for ClusterOperator "service-ca"...
2019/08/30 05:26:30     Found related object "namespaces/openshift-config-managed" for ClusterOperator "service-ca"...
2019/08/30 05:26:30     Found related object "namespaces/openshift-service-ca-operator" for ClusterOperator "service-ca"...
2019/08/30 05:26:30     Found related object "namespaces/openshift-service-ca" for ClusterOperator "service-ca"...
2019/08/30 05:26:41 Gathering data for ns/openshift-service-ca-operator...
2019/08/30 05:26:41     Collecting resources for namespace "openshift-service-ca-operator"...
2019/08/30 05:26:41     Gathering pod data for namespace "openshift-service-ca-operator"...
2019/08/30 05:26:41         Gathering data for pod "service-ca-operator-8c9b5ddf4-698zt"
2019/08/30 05:26:41         Skipping container data collection for pod "service-ca-operator-8c9b5ddf4-698zt": Pod not running
2019/08/30 05:26:41         Gathering data for pod "service-ca-operator-8c9b5ddf4-7npjs"
2019/08/30 05:26:42         Unable to gather previous container logs: previous terminated container "operator" in pod "service-ca-operator-8c9b5ddf4-7npjs" not found
2019/08/30 05:26:42         Skipping container endpoint collection for pod "service-ca-operator-8c9b5ddf4-7npjs" container "operator": No ports
2019/08/30 05:26:53 Gathering data for ns/openshift-service-ca...
2019/08/30 05:26:53     Collecting resources for namespace "openshift-service-ca"...
2019/08/30 05:26:54     Gathering pod data for namespace "openshift-service-ca"...
2019/08/30 05:26:54         Gathering data for pod "apiservice-cabundle-injector-64dbf9f-k5mjb"
2019/08/30 05:26:56         Skipping /version info gathering for pod "apiservice-cabundle-injector-64dbf9f-k5mjb". Endpoint not found...
2019/08/30 05:26:56         Gathering data for pod "configmap-cabundle-injector-9bdd4668c-792zm"
2019/08/30 05:26:57         Skipping /version info gathering for pod "configmap-cabundle-injector-9bdd4668c-792zm". Endpoint not found...
2019/08/30 05:26:57         Gathering data for pod "service-serving-cert-signer-6969d94ff7-96zwd"
2019/08/30 05:26:57         Skipping /version info gathering for pod "service-serving-cert-signer-6969d94ff7-96zwd". Endpoint not found...
2019/08/30 05:27:19 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:27:19 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:27:19 Gathering cluster operator resource data...
2019/08/30 05:27:19     Gathering related object reference information for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19     Found related object "namespaces/openshift-config" for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19     Found related object "namespaces/openshift-config-managed" for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19     Found related object "namespaces/openshift-service-catalog-apiserver-operator" for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19     Found related object "namespaces/openshift-service-catalog-apiserver" for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19     Found related object "apiservices.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io" for ClusterOperator "service-catalog-apiserver"...
2019/08/30 05:27:19 Gathering data for ns/openshift-service-catalog-apiserver-operator...
2019/08/30 05:27:19     Collecting resources for namespace "openshift-service-catalog-apiserver-operator"...
2019/08/30 05:27:19     Gathering pod data for namespace "openshift-service-catalog-apiserver-operator"...
2019/08/30 05:27:19         Gathering data for pod "openshift-service-catalog-apiserver-operator-6d77cf8ccf-kpnkc"
2019/08/30 05:27:19         Unable to gather previous container logs: previous terminated container "operator" in pod "openshift-service-catalog-apiserver-operator-6d77cf8ccf-kpnkc" not found
2019/08/30 05:27:38 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:27:38 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:27:38 Gathering cluster operator resource data...
2019/08/30 05:27:38     Gathering related object reference information for ClusterOperator "service-catalog-controller-manager"...
2019/08/30 05:27:38     Found related object "servicecatalogcontrollermanagers.operator.openshift.io/cluster" for ClusterOperator "service-catalog-controller-manager"...
2019/08/30 05:27:38     Found related object "namespaces/openshift-service-catalog-controller-manager-operator" for ClusterOperator "service-catalog-controller-manager"...
2019/08/30 05:27:38     Found related object "namespaces/openshift-service-catalog-controller-manager" for ClusterOperator "service-catalog-controller-manager"...
2019/08/30 05:28:17 Gathering data for ns/openshift-service-catalog-controller-manager-operator...
2019/08/30 05:28:17     Collecting resources for namespace "openshift-service-catalog-controller-manager-operator"...
2019/08/30 05:28:17     Gathering pod data for namespace "openshift-service-catalog-controller-manager-operator"...
2019/08/30 05:28:17         Gathering data for pod "openshift-service-catalog-controller-manager-operator-55c9rzv6f"
2019/08/30 05:28:21         Unable to gather previous container logs: previous terminated container "operator" in pod "openshift-service-catalog-controller-manager-operator-55c9rzv6f" not found
2019/08/30 05:29:14 Skipping previously-collected config.openshift.io resource data
2019/08/30 05:29:14 Skipping previously-collected operator.openshift.io resource data
2019/08/30 05:29:14 Gathering cluster operator resource data...
2019/08/30 05:29:14     Gathering related object reference information for ClusterOperator "storage"...
2019/08/30 05:29:14     Found related object "namespaces/openshift-cluster-storage-operator" for ClusterOperator "storage"...
2019/08/30 05:29:14     Found related object "infrastructures.config.openshift.io/cluster" for ClusterOperator "storage"...
2019/08/30 05:29:14 Gathering data for ns/openshift-cluster-storage-operator...
2019/08/30 05:29:14     Collecting resources for namespace "openshift-cluster-storage-operator"...
2019/08/30 05:29:14     Gathering pod data for namespace "openshift-cluster-storage-operator"...
2019/08/30 05:29:14         Gathering data for pod "cluster-storage-operator-5b46658459-8nwlq"
2019/08/30 05:29:14         Unable to gather previous container logs: previous terminated container "cluster-storage-operator" in pod "cluster-storage-operator-5b46658459-8nwlq" not found
E0830 05:29:14.649077      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod de341888257e6a02f7e97dfbd732ea8778405af747310366966be500461aee44, uid : exit status 1: 2019/08/30 05:29:14 socat[58340] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:29:14.746070      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod de341888257e6a02f7e97dfbd732ea8778405af747310366966be500461aee44, uid : exit status 1: 2019/08/30 05:29:14 socat[58347] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
E0830 05:29:14.817120      32 portforward.go:331] an error occurred forwarding 37587 -> 60000: error forwarding port 60000 to pod de341888257e6a02f7e97dfbd732ea8778405af747310366966be500461aee44, uid : exit status 1: 2019/08/30 05:29:14 socat[58354] E connect(5, AF=2 127.0.0.1:60000, 16): Connection refused
Error: [one or more errors ocurred while gathering pod-specific data for namespace: openshift-authentication

    one or more errors ocurred while gathering container data for pod oauth-openshift-75d8969cb-sk2g6:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-cloud-credential-operator

    one or more errors ocurred while gathering container data for pod cloud-credential-operator-5d5b8c86d8-w7pdq:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], one or more errors ocurred while gathering pod-specific data for namespace: openshift-machine-api

    [one or more errors ocurred while gathering container data for pod cluster-autoscaler-operator-56fd4f8b66-zrhwt:

    [unable to gather container /healthz: the server could not find the requested resource, unable to gather container /version: the server could not find the requested resource, unable to gather container /metrics: the server could not find the requested resource], one or more errors ocurred while gathering container data for pod machine-api-operator-594fcb5d95-jzg8d:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client]], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-console-operator

    one or more errors ocurred while gathering container data for pod console-operator-7cbbf95d76-l97nx:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], one or more errors ocurred while gathering pod-specific data for namespace: openshift-console

    [one or more errors ocurred while gathering container data for pod downloads-f9ff65658-b5pw7:

    [unable to gather container /healthz: Get https://localhost:37587/: tls: oversized record received with length 24932, unable to gather container /version: Get https://localhost:37587/: tls: oversized record received with length 24932, unable to gather container /metrics: Get https://localhost:37587/metrics: tls: oversized record received with length 24932], one or more errors ocurred while gathering container data for pod downloads-f9ff65658-lhfvx:

    [unable to gather container /healthz: Get https://localhost:37587/: tls: oversized record received with length 24932, unable to gather container /version: Get https://localhost:37587/: tls: oversized record received with length 24932, unable to gather container /metrics: Get https://localhost:37587/metrics: tls: oversized record received with length 24932]]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns

    [one or more errors ocurred while gathering container data for pod dns-default-fc9v6:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], one or more errors ocurred while gathering container data for pod dns-default-xm2qs:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress

    one or more errors ocurred while gathering container data for pod router-default-6bd4f956f-pfxvd:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering pod-specific data for namespace: openshift-insights

    one or more errors ocurred while gathering container data for pod insights-operator-68d67f8466-shrbn:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver-operator

    one or more errors ocurred while gathering container data for pod kube-apiserver-operator-78476cd5fc-b89sc:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-apiserver

    one or more errors ocurred while gathering container data for pod kube-apiserver-ocp-n76nx-master-0:

    [[unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client]]], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager

    one or more errors ocurred while gathering container data for pod kube-controller-manager-ocp-n76nx-master-0:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-controller-manager-operator

    one or more errors ocurred while gathering container data for pod kube-controller-manager-operator-5995667999-567rh:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-kube-scheduler

    one or more errors ocurred while gathering container data for pod openshift-kube-scheduler-ocp-n76nx-master-0:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-marketplace

    [one or more errors ocurred while gathering container data for pod certified-operators-549b8668cd-tzxnv:

    [unable to gather container /healthz: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /version: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /metrics: Get https://localhost:37587/metrics: tls: first record does not look like a TLS handshake], one or more errors ocurred while gathering container data for pod community-operators-6b67cd49cb-k6l9m:

    [unable to gather container /healthz: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /version: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /metrics: Get https://localhost:37587/metrics: tls: first record does not look like a TLS handshake], one or more errors ocurred while gathering container data for pod marketplace-operator-866f588dc6-sz5q2:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], one or more errors ocurred while gathering container data for pod redhat-operators-79bd89d844-fxfvs:

    [unable to gather container /healthz: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /version: Get https://localhost:37587/: tls: first record does not look like a TLS handshake, unable to gather container /metrics: Get https://localhost:37587/metrics: tls: first record does not look like a TLS handshake]], [the server doesn't have a resource type "monitoring", one or more errors ocurred while gathering pod-specific data for namespace: openshift-monitoring

    [one or more errors ocurred while gathering container data for pod alertmanager-main-0:

    [[unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], [unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW]], one or more errors ocurred while gathering container data for pod alertmanager-main-1:

    [[unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], [unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW]], one or more errors ocurred while gathering container data for pod alertmanager-main-2:

    [[unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], [unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW]], one or more errors ocurred while gathering container data for pod cluster-monitoring-operator-7d68ff9d9b-l9b8k:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod grafana-5b4896f7bd-vvxts:

    [[unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], [unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW]], one or more errors ocurred while gathering container data for pod kube-state-metrics-5f9cd4b54d-h2nm6:

    [[unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials], [unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials]], one or more errors ocurred while gathering container data for pod node-exporter-jmjrz:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod node-exporter-n4cqw:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod openshift-state-metrics-fccc95544-mgz6j:

    [[unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials], [unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials]], one or more errors ocurred while gathering container data for pod prometheus-adapter-c59d9597b-gdpxz:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering container data for pod prometheus-adapter-c59d9597b-npkhv:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering container data for pod prometheus-k8s-0:

    [[unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW], [unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials]], one or more errors ocurred while gathering container data for pod prometheus-k8s-1:

    [[unable to gather container /healthz: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW, unable to gather container /version: <!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
  <title>Log In</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
  <style>
    @font-face {
      font-family: "Open Sans";
      src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRgABAAAAAFeoABMAAAAAlkQAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAABGRlRNAAABqAAAABwAAAAcavCZq0dERUYAAAHEAAAAHQAAAB4AJwD1R1BPUwAAAeQAAASjAAAJni1yF0JHU1VCAAAGiAAAAIEAAACooGKInk9TLzIAAAcMAAAAYAAAAGCh3ZrDY21hcAAAB2wAAAGcAAACAv1rbL5jdnQgAAAJCAAAADIAAAA8K3MG4GZwZ20AAAk8AAAE+gAACZGLC3pBZ2FzcAAADjgAAAAIAAAACAAAABBnbHlmAAAOQAAAQG0AAHBIDuDVH2hlYWQAAE6wAAAANAAAADYHgk2EaGhlYQAATuQAAAAgAAAAJA37BfVobXR4AABPBAAAAjgAAAO8MaBM1GxvY2EAAFE8AAAB1QAAAeB9N5qybWF4cAAAUxQAAAAgAAAAIAMhAjxuYW1lAABTNAAAAeQAAARWRvKTBXBvc3QAAFUYAAAB+AAAAvgEbWOAcHJlcAAAVxAAAACQAAAAkPNEIux3ZWJmAABXoAAAAAYAAAAGxDVUvgAAAAEAAAAA0MoNVwAAAADJQhegAAAAANDkdLN42mNgZGBg4AFiMSBmYmAEwndAzALmMQAADdgBHQAAAHjarZZNTJRHGMf/uyzuFm2RtmnTj2hjKKE0tikxAbboiQCljdUF7Npiaz9MDxoTSWPSkHhAV9NDE9NYasYPGtRFUfZgEAl+tUEuHnodAoVTjxNOpgdjuv3NwKJ2K22T5skv8zLvM8/Hf+YdVhFJZerQZ4o1Nb/XoRc//7p7j6q+7N61W7V7Pv1qrzYpho/yeXnff/Mc2b2re68S/ikQUzSMCUUS3cFzp+7oTuRopC9yF+5F09EsTEXnotmS1dF0yQEYif0Sux+7H82Wzq/4LXI0/ly8Op6CL3jaD/7v6vhP8VQimUjG9yeSxLv3wIiWhQVLP2zEDVY6X3IgxClY9aOW2AlJT3SqdJ5K74aq+wJvqTK/T3V6TQ2QhEY9q6Z8Ts35jFqgFdryE9oCWyHF3+2MHYydjNsgDb3EOQiHIAOH4Qj0E28A3zPEPAvnIAuDcB4u8G4ILsIlGIYRuAKjcBXGYByukec63ICbcJu5SeJHtF5jel5VeaMaqIUNUEf++rxVA35JaIRvmD8G30Mf/ADHwcAJfE/CKTgN/fhPMD/JGCFajhylxCyDKt7XwPpIGfks+WzI14BXEhZyWXJZcllyWXJZcllyFWLbEHuadbPwjMpZWQGVIdoE0RzRnN7m70bGjdDL80E4BBk4DEdCREc0pxnWz8GqpRoL9S1Xj6/F69jDunJqqoB1nAdfyeMyzuAzBy+hSheqdBVlrIN6ampgTIYeJpat4gS+J+EUnIZ+/BdUmkClLlTq0pMq/+N3VUAle+OVWVDFUKOhRkONhhoNNRrN4DcHzaGr1UHfQmf7iutlvokczbxrgVZogy1E2gopntsZOxg7GbcRK824nbUfwkfQBTvI87gvYrn+B3h/hvxn4RxkYRDOwwXeDcFFuATDMAJXYBSuwhiMwzVqug434CbcWtzh27yz1DYFhd1biTIWVSyKeB0dVTuqdlTtqNpRtT9VFm92EG+Dt1nUMIeGDg0dGjo0dOhn0c+in0U/i34O/Rz6OfSz6OfQz6KfQz+Hfj5rjqw5subImiNrjqw5tHJo5dDKoZVDK4dWDq0cWlm0smhl0cqilUUri1YWrSxaW], [unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials]], one or more errors ocurred while gathering container data for pod prometheus-operator-cb87ddff6-r842w:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod telemeter-client-59655d88b5-sz7tr:

    [[unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], [unable to gather container /healthz: the server has asked for the client to provide credentials, unable to gather container /version: the server has asked for the client to provide credentials, unable to gather container /metrics: the server has asked for the client to provide credentials]]]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-sdn

    [one or more errors ocurred while gathering container data for pod sdn-c7h95:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod sdn-f5qr7:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client]], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver-operator

    one or more errors ocurred while gathering container data for pod openshift-apiserver-operator-7d8665c54b-hn7fh:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-apiserver

    one or more errors ocurred while gathering container data for pod apiserver-w4mmr:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"]], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-controller-manager-operator

    one or more errors ocurred while gathering container data for pod openshift-controller-manager-operator-5fb54c6cf5-9jk78:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering pod-specific data for namespace: openshift-controller-manager

    one or more errors ocurred while gathering container data for pod controller-manager-rv4pc:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"]], one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-samples-operator

    one or more errors ocurred while gathering container data for pod cluster-samples-operator-96df65c4d-5m8pw:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF], [operatorgroups.operators.coreos.com "operator-lifecycle-manager" not found, clusterserviceversions.operators.coreos.com "operator-lifecycle-manager" not found, one or more errors ocurred while gathering pod-specific data for namespace: openshift-operator-lifecycle-manager

    [one or more errors ocurred while gathering container data for pod catalog-operator-8494bc69b7-lmkwr:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod olm-operator-75b7694687-t4krz:

    [unable to gather container /healthz: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /version: Get https://localhost:37587/: http: server gave HTTP response to HTTPS client, unable to gather container /metrics: Get https://localhost:37587/metrics: http: server gave HTTP response to HTTPS client], one or more errors ocurred while gathering container data for pod packageserver-6bcffc6d9f-c2fcm:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], one or more errors ocurred while gathering container data for pod packageserver-6bcffc6d9f-q8qfj:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"]]], [subscriptions.operators.coreos.com "operator-lifecycle-manager-catalog" not found, installplans.operators.coreos.com "operator-lifecycle-manager-catalog" not found], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-service-catalog-apiserver-operator

    one or more errors ocurred while gathering container data for pod openshift-service-catalog-apiserver-operator-6d77cf8ccf-kpnkc:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], namespaces "openshift-service-catalog-apiserver" not found, apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io" not found], [one or more errors ocurred while gathering pod-specific data for namespace: openshift-service-catalog-controller-manager-operator

    one or more errors ocurred while gathering container data for pod openshift-service-catalog-controller-manager-operator-55c9rzv6f:

    [unable to gather container /healthz: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /version: forbidden: User "system:anonymous" cannot get path "/", unable to gather container /metrics: forbidden: User "system:anonymous" cannot get path "/metrics"], namespaces "openshift-service-catalog-controller-manager" not found], one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-storage-operator

    one or more errors ocurred while gathering container data for pod cluster-storage-operator-5b46658459-8nwlq:

    [unable to gather container /healthz: Get https://localhost:37587/: EOF, unable to gather container /version: Get https://localhost:37587/: EOF, unable to gather container /metrics: Get https://localhost:37587/metrics: EOF]]
2019/08/30 05:29:54 Finished successfully with no errors.
2019/08/30 05:29:54 Finished successfully with no errors.
2019/08/30 05:29:54 Finished successfully with no errors.
2019/08/30 05:29:55 Finished successfully with no errors.
2019/08/30 05:29:55 Finished successfully with no errors.
2019/08/30 05:29:55 Gathering data for ns/default...
2019/08/30 05:29:55     Collecting resources for namespace "default"...
2019/08/30 05:29:55     Gathering pod data for namespace "default"...
2019/08/30 05:30:17 Finished successfully with no errors.
2019/08/30 05:30:17 Gathering data for ns/openshift...
2019/08/30 05:30:17     Collecting resources for namespace "openshift"...
2019/08/30 05:30:17     Gathering pod data for namespace "openshift"...
2019/08/30 05:30:32 Finished successfully with no errors.
2019/08/30 05:30:33 Gathering data for ns/kube-system...
2019/08/30 05:30:33     Collecting resources for namespace "kube-system"...
2019/08/30 05:30:33     Gathering pod data for namespace "kube-system"...
2019/08/30 05:30:46 Finished successfully with no errors.
2019/08/30 05:30:46 Gathering data for ns/openshift-etcd...
2019/08/30 05:30:46     Collecting resources for namespace "openshift-etcd"...
2019/08/30 05:30:46     Gathering pod data for namespace "openshift-etcd"...
2019/08/30 05:30:46         Gathering data for pod "etcd-member-ocp-n76nx-master-0"
2019/08/30 05:30:48         Unable to gather previous container logs: previous terminated container "etcd-member" in pod "etcd-member-ocp-n76nx-master-0" not found
2019/08/30 05:30:49         Unable to gather previous container logs: previous terminated container "etcd-metrics" in pod "etcd-member-ocp-n76nx-master-0" not found
2019/08/30 05:30:49         Unable to gather previous container logs: previous terminated container "discovery" in pod "etcd-member-ocp-n76nx-master-0" not found
2019/08/30 05:30:49         Skipping container endpoint collection for pod "etcd-member-ocp-n76nx-master-0" container "discovery": No ports
2019/08/30 05:30:49         Unable to gather previous container logs: previous terminated container "certs" in pod "etcd-member-ocp-n76nx-master-0" not found
2019/08/30 05:30:49         Skipping container endpoint collection for pod "etcd-member-ocp-n76nx-master-0" container "certs": No ports
Error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-etcd

    one or more errors ocurred while gathering container data for pod etcd-member-ocp-n76nx-master-0:

    [[unable to gather container /healthz: Get https://localhost:37587/: remote error: tls: bad certificate, unable to gather container /version: Get https://localhost:37587/: remote error: tls: bad certificate, unable to gather container /metrics: Get https://localhost:37587/metrics: remote error: tls: bad certificate], [unable to gather container /healthz: Get https://localhost:37587/: remote error: tls: bad certificate, unable to gather container /version: Get https://localhost:37587/: remote error: tls: bad certificate, unable to gather container /metrics: Get https://localhost:37587/metrics: remote error: tls: bad certificate]]
2019/08/30 05:30:57 Finished successfully with no errors.
2019/08/30 05:30:57 Finished successfully with no errors.
2019/08/30 05:30:57 Finished successfully with no errors.
2019/08/30 05:31:04 Finished successfully with no errors.
2019/08/30 05:31:04 Finished successfully with no errors.
2019/08/30 05:31:04 Finished successfully with no errors.
WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=openshift-apiserver/audit.log
INFO: Audit logs for openshift-apiserver collected.
WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=kube-apiserver/audit-2019-08-30T03-31-11.121.log
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=kube-apiserver/audit-2019-08-30T04-01-53.559.log
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=kube-apiserver/audit-2019-08-30T04-33-28.495.log
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=kube-apiserver/audit-2019-08-30T05-04-53.373.log
/usr/bin/oc adm node-logs ocp-n76nx-master-0 --path=kube-apiserver/audit.log
INFO: Audit logs for kube-apiserver collected.
WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
INFO: Collecting host service logs for kubelet
INFO: Collecting host service logs for crio
INFO: Waiting for worker host service log collection to complete ...
INFO: Worker host service log collection to complete.
error: Missing or incomplete configuration info.  Please login or point to an existing, complete config file:

  1. Via the command-line flag --config
  2. Via the KUBECONFIG environment variable
  3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.3]
clusterrolebinding.rbac.authorization.k8s.io/must-gather-g5xk2 deleted
namespace/openshift-must-gather-bkvwj deleted
error: exit status 12

wking commented 5 years ago

Hmm. Looks like some of that succeeded and a bunch failed. Did it create a tarball/

wking commented 5 years ago

Also, what's the cluster ID from install-dir/metadata.json? Maybe some Telemetry is making it out.

deanpeterson commented 5 years ago

Also, what's the cluster ID from install-dir/metadata.json? Maybe some Telemetry is making it out.

{"clusterName":"ocp","clusterID":"a9c25afa-568b-4a7a-88d4-22de58a1987a","infraID":"ocp-n76nx","libvirt":{"uri":"qemu+tcp://192.168.122.1/system"}}

deanpeterson commented 5 years ago

Hmm. Looks like some of that succeeded and a bunch failed. Did it create a tarball/

Where would the tarball be? It's definitely not on the host machine where I ran the oc adm must-gather command.

wking commented 5 years ago

Tarball would have been on the installer machine. Seems like your cluster is too broken to get the tarball off the cluster and back to your installer machine (rsync: connection unexpectedly closed).

Looking at metrics from your cluster, there are all sorts of alerts firing, including KubeCPUOvercommit, KubeMemOvercommit, KubePodCrashLooping, and etcdHighCommitDurations. I suspect you have underprovisioned your control plane machines, and the cluster components there are all gasping for resources. #2258 has some discussion about possibly bumping our defaults (as we've done before in #1279 and similar).

deanpeterson commented 5 years ago

Interesting. I will modify the defaults. Just an fyi, following the Libvirt Instructions did not indicate needing to set cpu/memory. So, I am using the defaults but the machine I am using is a dual socket xeon with 24 cores and 64 GB of ram. If the defaults don't work on that machine wouldn't that tutorial fail for most everyone?

deanpeterson commented 5 years ago

What isn't entirely clear, if I'm following Libvirt Instructions, do I still have to do prerequisites like set up an http server to host ignition files and rchos images? Or is all of that unnecessary when doing the libvirt install?

simon3z commented 5 years ago

@wking I am hitting this as well with machines of 12Gb each (master and worker). This issue doesn't seem related to underprovisioned VMs (unless for some reason the system believes to be underprovisioned) as there are 4-6Gb still free when the error occurs.

I cannot provide the must-gather feedback but on the other hand this is 100% reproducible ATM.

wking commented 5 years ago

I dunno. Getting logs from the pod in the openshift-cluster-version namespace might help. oc get -o yaml clusteroperators and oc get -o yaml --all-namespaces pods might help. Logs from pods in the openshift-authentication-operator and openshift-console-operator namespaces might help. Hopefully those complete fast enough that they don't die like your must-gather.

deanpeterson commented 5 years ago

I'm thinking without better understanding of what the requirements are it could be any number of things. I am most likely going to have to do the bare metal install with kvm and libvirt from scratch rather than use the dev / libvirt tutorial and scripts. That way I'll have a much better understanding where I might be going wrong. In fact, I only did the dev / libvirt process after trying to follow a number of tutorials online about how to get the bare metal install process to work with kvm and libvirt. Then, I found the github libvirt readme thinking the process had been streamlined. I didn't realize it was an entirely different process designed just for low powered dev machines. I will say the new 4.X install UPI has quite a learning curve for less operational type app/dev - middleware folks like myself. I have installed Openshift 3.X a number of times and that was easier for me. I think the biggest hurdle is some of the assumption the 4.X documentation makes about setting up DNS requirements. I have read a number of tutorials, and they all make different recommendations and jump between running a separate VM dedicated to DNS with dnsmasq or Bind (will have to read up what the actual right way is or if any of that is in fact necessary) to the libvirt tutorial saying I just have to run some one line command to set things manually to get around dns wildcard limitations. That is all highly confusing at the moment. I am actually a new Middleware SA for Red Hat. My plan is to get this working and do a painstakingly detailed video tutorial on how to install Openshift 4.X. Eventually things will click, but right now, I am quite confused as I have yet to learn the CoreOS way.

wking commented 5 years ago

Have you tried the official 4.1 baremetal docs? Using the official flow with a stable 4.1 release is going to be more reliable than any workflow with a 4.2 nightly.

deanpeterson commented 5 years ago

Have you tried the official 4.1 baremetal docs? Using the official flow with a stable 4.1 release is going to be more reliable than any workflow with a 4.2 nightly.

I have kind of tried to do that but then I went to on-line blogs to help me understand the pre reqs a bit better and ended up confusing myself more it seems. I will see if I can figure it out just using the documentation. Being a middleware - app/dev guy some of pre reqs (especially around DNS) are still a bit fuzzy to me and I was looking for a short cut. Lesson learned.

wking commented 5 years ago

Installer-provisioning for AWS is probably the simplest approach. But there's always room for improvement; let us know if/where you get stuck in any flow.

wking commented 5 years ago

And, for what it's worth, the libvirt CI jobs are generally making it through setup (and then failing to pass the e2e suite): https://ci-search-ci-search-next.svc.ci.openshift.org/chart?name=libvirt&search=failed%20to%20initialize%20the%20cluster

deanpeterson commented 5 years ago

I was finally successful. I created a video tutorial to help anyone else trying to get through the gotchas I ran into: https://youtu.be/4mFMqNExRWk

zeenix commented 5 years ago

I was finally successful. I created a video tutorial to help anyone else trying to get through the gotchas I ran into:

Thanks for keeping us updated and also making a video. Could you please summarize what was it in the end? Did you just miss something in the docs?

deanpeterson commented 5 years ago

I was finally successful. I created a video tutorial to help anyone else trying to get through the gotchas I ran into:

Thanks for keeping us updated and also making a video. Could you please summarize what was it in the end? Did you just miss something in the docs?

It was a few things:

The part in the troubleshooting section about modifying openshift.conf to get around the dns wildcard issue should probably just go right in the main instructions section of the libvirt howto. It should be more clear generating the manifests first and then running that sed command is an important step.
It helped quite a bit to update the default vm sizing to 16 gigs and 8 vcpu.
There should be a reminder to generate an ssh key prior to generating the manifests. I was on a fresh machine and the cli would just skip past setting or asking about the ssh key when one didn't exist. Once I generated that, the cli then asked me about using it.
The error messages in the openshift_install.log while things are coming up are scary. Unless you spend some time looking at issues in github, it's easy to just think things are really messed up rather than realize the errors are continued checks to probe running components.

ghost commented 4 years ago

@deanpeterson

Great job !! but you can set dns (bind - bare metal) to resolve the * .apps.${domain} and i made some changes below

1)

[root@argon ~]# cat /etc/NetworkManager/dnsmasq.d/openshift.conf 
server=/jaguar.fsbano.com/192.168.126.1
server=/apps.jaguar.fsbano.com/172.27.15.30
[root@argon ~]#

2) git diff

[root@argon installer]# git diff
diff --git a/cmd/openshift-install/create.go b/cmd/openshift-install/create.go
index 9021025b6..679649d1d 100644
--- a/cmd/openshift-install/create.go
+++ b/cmd/openshift-install/create.go
@@ -238,7 +238,7 @@ func waitForBootstrapComplete(ctx context.Context, config *rest.Config, director

        discovery := client.Discovery()

-       apiTimeout := 30 * time.Minute
+       apiTimeout := 60 * time.Minute
        logrus.Infof("Waiting up to %v for the Kubernetes API at %s...", apiTimeout, config.Host)
        apiContext, cancel := context.WithTimeout(ctx, apiTimeout)
        defer cancel()
@@ -279,7 +279,7 @@ func waitForBootstrapComplete(ctx context.Context, config *rest.Config, director
 // and waits for the bootstrap configmap to report that bootstrapping has
 // completed.
 func waitForBootstrapConfigMap(ctx context.Context, client *kubernetes.Clientset) error {
-       timeout := 30 * time.Minute
+       timeout := 60 * time.Minute
        logrus.Infof("Waiting up to %v for bootstrapping to complete...", timeout)

        waitCtx, cancel := context.WithTimeout(ctx, timeout)
@@ -317,7 +317,7 @@ func waitForBootstrapConfigMap(ctx context.Context, client *kubernetes.Clientset
 // waitForInitializedCluster watches the ClusterVersion waiting for confirmation
 // that the cluster has been initialized.
 func waitForInitializedCluster(ctx context.Context, config *rest.Config) error {
-       timeout := 30 * time.Minute
+       timeout := 60 * time.Minute
        logrus.Infof("Waiting up to %v for the cluster at %s to initialize...", timeout, config.Host)
        cc, err := configclient.NewForConfig(config)
        if err != nil {
diff --git a/data/data/libvirt/main.tf b/data/data/libvirt/main.tf
index 9ba88c9cf..152c78dd5 100644
--- a/data/data/libvirt/main.tf
+++ b/data/data/libvirt/main.tf
@@ -54,6 +54,11 @@ resource "libvirt_network" "net" {
   dns {
     local_only = true

+    forwarders { 
+        address = "172.27.15.30"
+        domain = "apps.${var.cluster_domain}"
+    }
+
     dynamic "srvs" {
       for_each = data.libvirt_network_dns_srv_template.etcd_cluster.*.rendered
       content {
diff --git a/data/data/libvirt/variables-libvirt.tf b/data/data/libvirt/variables-libvirt.tf
index 53cf68bae..79d1018e2 100644
--- a/data/data/libvirt/variables-libvirt.tf
+++ b/data/data/libvirt/variables-libvirt.tf
@@ -32,7 +32,7 @@ variable "libvirt_master_ips" {
 variable "libvirt_master_memory" {
   type        = string
   description = "RAM in MiB allocated to masters"
-  default     = "6144"
+  default     = "16384"
 }

 # At some point this one is likely to default to the number
diff --git a/pkg/asset/machines/libvirt/machines.go b/pkg/asset/machines/libvirt/machines.go
index 2ab6d9aa2..08847ab95 100644
--- a/pkg/asset/machines/libvirt/machines.go
+++ b/pkg/asset/machines/libvirt/machines.go
@@ -63,7 +63,7 @@ func provider(clusterID string, networkInterfaceAddress string, platform *libvir
                        APIVersion: "libvirtproviderconfig.openshift.io/v1beta1",
                        Kind:       "LibvirtMachineProviderConfig",
                },
-               DomainMemory: 7168,
+               DomainMemory: 16384,
                DomainVcpu:   4,
                Ignition: &libvirtprovider.Ignition{
                        UserDataSecret: userDataSecret,
[root@argon installer]#

ghost commented 4 years ago

@deanpeterson

see - https://pbs.twimg.com/media/EFiKJ0OW4AAc4xn?format=png&name=large

Best Regards, Fabio Sbano

rthallisey commented 4 years ago

@deanpeterson If you're still having this problem, here's a workaround: https://github.com/openshift/installer/issues/1648#issuecomment-585235423

openshift / installer

100% complete, waiting on authentication, console #2287