search

gshipley / installcentos

427 stars 455 forks source link

Service catalog install failed #182

Open hostingnuggets opened 4 years ago

hostingnuggets commented 4 years ago

Hello,

I am install okd 3.11 with this script on a fresh CentOS 7.7 server and unfortunately the service catalog part of the deploy ansible script fails. Below I have pasted the two specific tasks which fail including the debug info.

The only custom settings I have for the installer script is that I have answered yes to use Let's Encrypt (the certbot part of the script went fine) and I am using export DISK="/dev/sdb".

Any hints or help is appreciated. Thanks in advance.

TASK [openshift_service_catalog : Wait for Controller Manager rollout success] *************************************************************************************************************************************************************************************************
FAILED - RETRYING: Wait for Controller Manager rollout success (5 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (4 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (3 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (2 retries left).
FAILED - RETRYING: Wait for Controller Manager rollout success (1 retries left).
fatal: [<IP_REMOVED>]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["oc", "rollout", "status", "--config=/etc/origin/master/admin.kubeconfig", "-n", "kube-service-catalog", "ds/controller-manager"], "delta": "0:00:00.203854", "end": "2020-04-17 23:19:50.143786", "msg": "non-zero return code", "rc": 1, "start": "2020-04-17 23:19:49.939932", "stderr": "error: watch closed before Until timeout", "stderr_lines": ["error: watch closed before Until timeout"], "stdout": "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...\nWaiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "stdout_lines": ["Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available...", "Waiting for daemon set \"controller-manager\" rollout to finish: 0 of 1 updated pods are available..."]}
...ignoring

TASK [openshift_service_catalog : Verify that the Catalog API Server is running] ***********************************************************************************************************************************************************************************************
FAILED - RETRYING: Verify that the Catalog API Server is running (60 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (59 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (58 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (57 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (56 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (55 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (54 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (53 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (52 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (51 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (50 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (49 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (48 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (47 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (46 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (45 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (44 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (43 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (42 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (41 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (40 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (39 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (38 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (37 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (36 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (35 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (34 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (33 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (32 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (31 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (30 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (29 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (28 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (27 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (26 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (25 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (24 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (23 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (22 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (21 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (20 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (19 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (18 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (17 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (16 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (15 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (14 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (13 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (12 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (11 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (10 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (9 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (8 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (7 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (6 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (5 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (4 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (3 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (2 retries left).
FAILED - RETRYING: Verify that the Catalog API Server is running (1 retries left).
fatal: [<IP_REMOVED>]: FAILED! => {"attempts": 60, "changed": false, "cmd": ["curl", "-k", "https://apiserver.kube-service-catalog.svc/healthz"], "delta": "0:00:01.010253", "end": "2020-04-17 23:31:06.368636", "msg": "non-zero return code", "rc": 7, "start": "2020-04-17 23:31:05.358383", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to apiserver.kube-service-catalog.svc:443; Connection refused"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "In project kube-service-catalog on server https://console-internal.mydomain.com:8443", 
        "", 
        "https://apiserver-kube-service-catalog.apps.mydomain.com (passthrough) to pod port secure (svc/apiserver)", 
        "  daemonset/apiserver manages docker.io/openshift/origin-service-catalog:v3.11", 
        "    generation #1 running for 31 minutes - 0/1 pods growing to 1", 
        "  pod/apiserver-hjbmt runs docker.io/openshift/origin-service-catalog:v3.11", 
        "", 
        "svc/controller-manager - 172.30.179.105:443 -> 6443", 
        "  daemonset/controller-manager manages docker.io/openshift/origin-service-catalog:v3.11", 
        "    generation #1 running for 31 minutes - 0/1 pods growing to 1", 
        "  pod/controller-manager-5479q runs docker.io/openshift/origin-service-catalog:v3.11", 
        "", 
        "Errors:", 
        "  * pod/controller-manager-5479q is crash-looping", 
        "", 
        "1 error, 1 warning identified, use 'oc status --suggest' to see details."
    ]
}

TASK [openshift_service_catalog : Get pods in the kube-service-catalog namespace] **********************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "NAME                       READY     STATUS             RESTARTS   AGE       IP            NODE                     NOMINATED NODE", 
        "apiserver-hjbmt            0/1       Running            8          31m       10.128.0.32   myserver.mydomain.com   <none>", 
        "controller-manager-5479q   0/1       CrashLoopBackOff   11         31m       10.128.0.33   myserver.mydomain.com   <none>"
    ]
}

TASK [openshift_service_catalog : Get events in the kube-service-catalog namespace] ********************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "LAST SEEN   FIRST SEEN   COUNT     NAME                                                  KIND        SUBOBJECT                             TYPE      REASON             SOURCE                               MESSAGE", 
        "31m         31m          1         apiserver.1606b708b62fc509                            DaemonSet                                         Normal    SuccessfulCreate   daemonset-controller                 Created pod: apiserver-hjbmt", 
        "31m         31m          1         apiserver-hjbmt.1606b70943ace0c8                      Pod         spec.containers{apiserver}            Normal    Pulling            kubelet, myserver.mydomain.com      pulling image \"docker.io/openshift/origin-service-catalog:v3.11\"", 
        "31m         31m          1         controller-manager.1606b709c9ea1b59                   DaemonSet                                         Normal    SuccessfulCreate   daemonset-controller                 Created pod: controller-manager-5479q", 
        "31m         31m          2         controller-manager-5479q.1606b709de00d7d6             Pod                                               Warning   FailedMount        kubelet, myserver.mydomain.com      MountVolume.SetUp failed for volume \"service-catalog-ssl\" : secrets \"controllermanager-ssl\" not found", 
        "31m         31m          1         apiserver-hjbmt.1606b70a9fb9cab4                      Pod         spec.containers{apiserver}            Normal    Pulled             kubelet, myserver.mydomain.com      Successfully pulled image \"docker.io/openshift/origin-service-catalog:v3.11\"", 
        "31m         31m          1         service-catalog-controller-manager.1606b70b77a69ae0   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "31m         31m          1         service-catalog-controller-manager.1606b70bda237e24   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "31m         31m          1         service-catalog-controller-manager.1606b70f719db525   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "30m         31m          4         controller-manager-5479q.1606b70b6b94652c             Pod         spec.containers{controller-manager}   Normal    Created            kubelet, myserver.mydomain.com      Created container", 
        "30m         31m          4         controller-manager-5479q.1606b70b769e9857             Pod         spec.containers{controller-manager}   Normal    Started            kubelet, myserver.mydomain.com      Started container", 
        "30m         30m          1         service-catalog-controller-manager.1606b7179303ccc6   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "29m         31m          5         controller-manager-5479q.1606b70b462e62a6             Pod         spec.containers{controller-manager}   Normal    Pulled             kubelet, myserver.mydomain.com      Container image \"docker.io/openshift/origin-service-catalog:v3.11\" already present on machine", 
        "29m         29m          1         service-catalog-controller-manager.1606b721912d165d   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "28m         31m          4         apiserver-hjbmt.1606b70fe7f2776a                      Pod         spec.containers{apiserver}            Normal    Pulled             kubelet, myserver.mydomain.com      Container image \"docker.io/openshift/origin-service-catalog:v3.11\" already present on machine", 
        "28m         31m          5         apiserver-hjbmt.1606b70acc234e25                      Pod         spec.containers{apiserver}            Normal    Created            kubelet, myserver.mydomain.com      Created container", 
        "28m         31m          5         apiserver-hjbmt.1606b70ada41daa4                      Pod         spec.containers{apiserver}            Normal    Started            kubelet, myserver.mydomain.com      Started container", 
        "28m         28m          1         service-catalog-controller-manager.1606b73739e3ccde   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "25m         25m          1         service-catalog-controller-manager.1606b75cbc43ac23   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "20m         20m          1         service-catalog-controller-manager.1606b7a3f3664094   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "16m         30m          61        apiserver-hjbmt.1606b7150294a040                      Pod         spec.containers{apiserver}            Warning   BackOff            kubelet, myserver.mydomain.com      Back-off restarting failed container", 
        "15m         15m          1         service-catalog-controller-manager.1606b7ebe9dfac99   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "10m         10m          1         service-catalog-controller-manager.1606b832ad46ce74   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "5m          5m           1         service-catalog-controller-manager.1606b879b2aab021   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader", 
        "1m          11m          121       apiserver-hjbmt.1606b821a488e639                      Pod         spec.containers{apiserver}            Warning   Unhealthy          kubelet, myserver.mydomain.com      Readiness probe failed: HTTP probe failed with statuscode: 500", 
        "1m          31m          140       controller-manager-5479q.1606b70c2c1048b1             Pod         spec.containers{controller-manager}   Warning   BackOff            kubelet, myserver.mydomain.com      Back-off restarting failed container", 
        "15s         15s          1         service-catalog-controller-manager.1606b8c003642347   ConfigMap                                         Normal    LeaderElection     service-catalog-controller-manager   controller-manager-5479q-external-service-catalog-controller became leader"
    ]
}

TASK [openshift_service_catalog : Get pod logs] ********************************************************************************************************************************************************************************************************************************
changed: [<IP_REMOVED>]

TASK [openshift_service_catalog : debug] ***************************************************************************************************************************************************************************************************************************************
ok: [<IP_REMOVED>] => {
    "msg": [
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45198]", 
        "I0417 21:30:52.331698       1 run_server.go:134] etcd checker called", 
        "I0417 21:30:53.267983       1 wrap.go:42] GET /healthz: (79.085µs) 200 [[kube-probe/1.11+] 10.128.0.1:45242]", 
        "E0417 21:30:53.331836       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:30:53.332025       1 wrap.go:42] GET /healthz/ready: (1.003896742s) 500", 
        "goroutine 4841 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420439d50, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420439d50, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4204814c0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6d60, 0xc4204a7000, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37b00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6d60, 0xc420a37a00)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6d60, 0xc420a37a00, 0xc4201d9980)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45230]", 
        "I0417 21:30:57.327797       1 run_server.go:134] etcd checker called", 
        "E0417 21:30:58.328094       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:30:58.328288       1 wrap.go:42] GET /healthz/ready: (1.000566658s) 500", 
        "goroutine 4861 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc420752000, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc420752000, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc4203ec0e0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4209f2298, 0xc420b34ec0, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4600)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4209f2298, 0xc4200d4500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4209f2298, 0xc4200d4500, 0xc420bd7380)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45286]", 
        "I0417 21:31:02.327443       1 run_server.go:134] etcd checker called", 
        "I0417 21:31:03.271530       1 wrap.go:42] GET /healthz: (3.603614ms) 200 [[kube-probe/1.11+] 10.128.0.1:45368]", 
        "E0417 21:31:03.327854       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:31:03.328053       1 wrap.go:42] GET /healthz/ready: (1.000644294s) 500", 
        "goroutine 4880 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4202f1420, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4202f1420, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc420481ee0, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6e90, 0xc4204a7a00, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f500)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6e90, 0xc42002f400)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6e90, 0xc42002f400, 0xc4210a89c0)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45356]", 
        "I0417 21:31:07.329046       1 run_server.go:134] etcd checker called", 
        "E0417 21:31:08.329250       1 run_server.go:143] etcd failed to reach any server", 
        "I0417 21:31:08.329370       1 wrap.go:42] GET /healthz/ready: (1.001460942s) 500", 
        "goroutine 4948 [running]:", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4202f1c00, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xd2", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4202f1c00, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc420510a20, 0x1f4)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:192 +0xac", 
        "net/http.Error(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420bc0fc0, 0x34, 0x1f4)", 
        "\t/usr/local/go/src/net/http/server.go:1959 +0xda", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz.handleRootHealthz.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:130 +0x4c4", 
        "net/http.HandlerFunc.ServeHTTP(0xc42022df40, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42088c9c0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x514", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x1a820d2, 0x19, 0xc42062b290, 0xc42010f3b0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb5c0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc", 
        "net/http.HandlerFunc.ServeHTTP(0xc42026fef0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce4", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb600, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16300)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3", 
        "net/http.HandlerFunc.ServeHTTP(0xc4205eb640, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x178", 
        "net/http.HandlerFunc.ServeHTTP(0xc42008d0e0, 0x7f676e03dbb0, 0xc4200e6ff8, 0xc420d16200)", 
        "\t/usr/local/go/src/net/http/server.go:1947 +0x44", 
        "github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc4204c61c0, 0x1fa8700, 0xc4200e6ff8, 0xc420d16200, 0xc4210a92c0)", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d", 
        "created by github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP", 
        "\t/tmp/openshift/build-rpms/rpm/BUILD/origin-service-catalog-3.11.0/_output/local/go/src/github.com/kubernetes-incubator/service-catalog/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5", 
        "", 
        "logging error output: \"[-]etcd failed: reason withheld\\nhealthz check failed\\n\"", 
        " [[kube-probe/1.11+] 10.128.0.1:45422]"
    ]
}

TASK [openshift_service_catalog : Report errors] *******************************************************************************************************************************************************************************************************************************
fatal: [<IP_REMOVED>]: FAILED! => {"changed": false, "msg": "Catalog install failed."}
    to retry, use: --limit @/home/user/installcentos/openshift-ansible/playbooks/deploy_cluster.retry

The /var/log/messages log file contains these errors repeated every 5 seconds:


Apr 18 11:19:30 myserver origin-node: I0418 11:19:30.480421    2281 prober.go:111] Readiness probe for "apiserver-66bsx_kube-service-catalog(819aff89-8150-11ea-89a7-a0369fa3e320):apiserver" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:30 myserver origin-node: I0418 11:19:30.995262    2281 prober.go:111] Readiness probe for "controller-manager-jpssz_kube-service-catalog(818589f3-8150-11ea-89a7-a0369fa3e320):controller-manager" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:35 myserver origin-node: I0418 11:19:35.483845    2281 prober.go:111] Readiness probe for "apiserver-66bsx_kube-service-catalog(819aff89-8150-11ea-89a7-a0369fa3e320):apiserver" failed (failure): HTTP probe failed with statuscode: 500
Apr 18 11:19:35 myserver origin-node: I0418 11:19:35.993962    2281 prober.go:111] Readiness probe for "controller-manager-jpssz_kube-service-catalog(818589f3-8150-11ea-89a7-a0369fa3e320):controller-manager" failed (failure): HTTP probe failed with statuscode: 500

Then the /usr/local/bin/master-logs api api command shows the following errors repeated also quite often:

E0418 11:15:33.399732       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:15:43.417797       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:15:53.436591       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:03.455844       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:13.474989       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:23.493941       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:33.512922       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:43.531735       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:16:53.551345       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:03.569631       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:13.596194       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
E0418 11:17:23.615032       1 memcache.go:147] couldn't get resource list for servicecatalog.k8s.io/v1beta1: the server is currently unable to handle the request
I0418 11:17:27.656105       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0418 11:17:27.658974       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0418 11:17:27.658985       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0418 11:17:28.650920       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io
E0418 11:17:28.651017       1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0418 11:17:28.651025       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue.
hostingnuggets commented 4 years ago

I re-installed the whole server again and now it went through the service catalog... I can't really explain why. Leaving this issue open for a few days just for reference and in case any one knows what that was about.