openshift / origin

Conformance test suite for OpenShift
http://www.openshift.org
Apache License 2.0
8.5k stars 4.71k forks source link

If an aggregated API server is down, kubectl may take up to 10 minutes to retrieve server resources #17159

Closed smarterclayton closed 6 years ago

smarterclayton commented 7 years ago

Scenario:

  1. Aggregated API is down
  2. Try to run kubectl
  3. Kubectl tries to discover down API
  4. Kubectl hangs until i/o timeout

Solution, do not fail this way.

smarterclayton commented 7 years ago

This is must fix for shipping aggregated apis - this wedged me while trying to debug a cluster @openshift/sig-master

stevekuznetsov commented 7 years ago
max log level on oc call ``` # oc get pods --loglevel=10000 I1102 21:07:04.076594 104483 loader.go:357] Config loaded from file /root/.kube/config I1102 21:07:04.078157 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/api I1102 21:07:04.090077 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/api 200 OK in 11 milliseconds I1102 21:07:04.090102 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.090106 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.090110 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.090114 104483 round_trippers.go:414] Content-Length: 138 I1102 21:07:04.090117 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.090145 104483 request.go:994] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"api.ci.openshift.org"}]} I1102 21:07:04.090405 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis I1102 21:07:04.091441 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis 200 OK in 1 milliseconds I1102 21:07:04.091456 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.091461 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.091465 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.091468 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.091550 104483 request.go:994] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"apps","versions":[{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"},{"groupVersion":"authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v2alpha1","version":"v2alpha1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"authorization.openshift.io","versions":[{"groupVersion":"authorization.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authorization.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"settings.k8s.io","versions":[{"groupVersion":"settings.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"settings.k8s.io/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":null},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"apps.openshift.io","versions":[{"groupVersion":"apps.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"build.openshift.io","versions":[{"groupVersion":"build.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"build.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"image.openshift.io","versions":[{"groupVersion":"image.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"image.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"network.openshift.io","versions":[{"groupVersion":"network.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"network.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"oauth.openshift.io","versions":[{"groupVersion":"oauth.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"oauth.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"project.openshift.io","versions":[{"groupVersion":"project.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"project.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"quota.openshift.io","versions":[{"groupVersion":"quota.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"quota.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"route.openshift.io","versions":[{"groupVersion":"route.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"route.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"security.openshift.io","versions":[{"groupVersion":"security.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"security.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"template.openshift.io","versions":[{"groupVersion":"template.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"template.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"user.openshift.io","versions":[{"groupVersion":"user.openshift.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"user.openshift.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"prow.k8s.io","versions":[{"groupVersion":"prow.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"prow.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"metrics","versions":[{"groupVersion":"metrics/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"metrics/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":null}]} I1102 21:07:04.092497 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/apiregistration.k8s.io/v1beta1 I1102 21:07:04.093339 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/apiregistration.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.093352 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.093356 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.093360 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.093363 104483 round_trippers.go:414] Content-Length: 378 I1102 21:07:04.093367 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.093380 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apiregistration.k8s.io/v1beta1","resources":[{"name":"apiservices","singularName":"","namespaced":false,"kind":"APIService","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"apiservices/status","singularName":"","namespaced":false,"kind":"APIService","verbs":["update"]}]} I1102 21:07:04.093869 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/extensions/v1beta1 I1102 21:07:04.094914 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/extensions/v1beta1 200 OK in 1 milliseconds I1102 21:07:04.094927 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.094931 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.094935 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.094939 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.094942 104483 round_trippers.go:414] Content-Length: 2457 I1102 21:07:04.094971 104483 request.go:994] Response Body: {"kind":"APIResourceList","groupVersion":"extensions/v1beta1","resources":[{"name":"daemonsets","singularName":"","namespaced":true,"kind":"DaemonSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ds"]},{"name":"daemonsets/status","singularName":"","namespaced":true,"kind":"DaemonSet","verbs":["get","patch","update"]},{"name":"deployments","singularName":"","namespaced":true,"kind":"Deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"]},{"name":"deployments/rollback","singularName":"","namespaced":true,"kind":"DeploymentRollback","verbs":["create"]},{"name":"deployments/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"deployments/status","singularName":"","namespaced":true,"kind":"Deployment","verbs":["get","patch","update"]},{"name":"ingresses","singularName":"","namespaced":true,"kind":"Ingress","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ing"]},{"name":"ingresses/status","singularName":"","namespaced":true,"kind":"Ingress","verbs":["get","patch","update"]},{"name":"networkpolicies","singularName":"","namespaced":true,"kind":"NetworkPolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"podsecuritypolicies","singularName":"","namespaced":false,"kind":"PodSecurityPolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["psp"]},{"name":"replicasets","singularName":"","namespaced":true,"kind":"ReplicaSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rs"],"categories":["all"]},{"name":"replicasets/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"replicasets/status","singularName":"","namespaced":true,"kind":"ReplicaSet","verbs":["get","patch","update"]},{"name":"replicationcontrollers","singularName":"","namespaced":true,"kind":"ReplicationControllerDummy","verbs":[]},{"name":"replicationcontrollers/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"thirdpartyresources","singularName":"","namespaced":false,"kind":"ThirdPartyResource","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.095429 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/apps/v1beta1 I1102 21:07:04.096110 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/apps/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.096120 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.096124 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.096127 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.096131 104483 round_trippers.go:414] Content-Length: 1154 I1102 21:07:04.096135 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.096160 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apps/v1beta1","resources":[{"name":"controllerrevisions","singularName":"","namespaced":true,"kind":"ControllerRevision","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"deployments","singularName":"","namespaced":true,"kind":"Deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"]},{"name":"deployments/rollback","singularName":"","namespaced":true,"kind":"DeploymentRollback","verbs":["create"]},{"name":"deployments/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"deployments/status","singularName":"","namespaced":true,"kind":"Deployment","verbs":["get","patch","update"]},{"name":"statefulsets","singularName":"","namespaced":true,"kind":"StatefulSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sts"],"categories":["all"]},{"name":"statefulsets/status","singularName":"","namespaced":true,"kind":"StatefulSet","verbs":["get","patch","update"]}]} I1102 21:07:04.096500 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/authentication.k8s.io/v1 I1102 21:07:04.097111 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/authentication.k8s.io/v1 200 OK in 0 milliseconds I1102 21:07:04.097124 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.097128 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.097132 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.097136 104483 round_trippers.go:414] Content-Length: 202 I1102 21:07:04.097139 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.097156 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"authentication.k8s.io/v1","resources":[{"name":"tokenreviews","singularName":"","namespaced":false,"kind":"TokenReview","verbs":["create"]}]} I1102 21:07:04.097416 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/authentication.k8s.io/v1beta1 I1102 21:07:04.098064 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/authentication.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.098078 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.098082 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.098086 104483 round_trippers.go:414] Content-Length: 207 I1102 21:07:04.098089 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.098093 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.098104 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"authentication.k8s.io/v1beta1","resources":[{"name":"tokenreviews","singularName":"","namespaced":false,"kind":"TokenReview","verbs":["create"]}]} I1102 21:07:04.098407 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/authorization.k8s.io/v1 I1102 21:07:04.099021 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/authorization.k8s.io/v1 200 OK in 0 milliseconds I1102 21:07:04.099031 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.099035 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.099038 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.099042 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.099045 104483 round_trippers.go:414] Content-Length: 468 I1102 21:07:04.099057 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"authorization.k8s.io/v1","resources":[{"name":"localsubjectaccessreviews","singularName":"","namespaced":true,"kind":"LocalSubjectAccessReview","verbs":["create"]},{"name":"selfsubjectaccessreviews","singularName":"","namespaced":false,"kind":"SelfSubjectAccessReview","verbs":["create"]},{"name":"subjectaccessreviews","singularName":"","namespaced":false,"kind":"SubjectAccessReview","verbs":["create"]}]} I1102 21:07:04.099338 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/authorization.k8s.io/v1beta1 I1102 21:07:04.100043 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/authorization.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.100097 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.100102 104483 round_trippers.go:414] Content-Length: 473 I1102 21:07:04.100106 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.100109 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.100113 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.100126 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"authorization.k8s.io/v1beta1","resources":[{"name":"localsubjectaccessreviews","singularName":"","namespaced":true,"kind":"LocalSubjectAccessReview","verbs":["create"]},{"name":"selfsubjectaccessreviews","singularName":"","namespaced":false,"kind":"SelfSubjectAccessReview","verbs":["create"]},{"name":"subjectaccessreviews","singularName":"","namespaced":false,"kind":"SubjectAccessReview","verbs":["create"]}]} I1102 21:07:04.100443 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/autoscaling/v1 I1102 21:07:04.101152 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/autoscaling/v1 200 OK in 0 milliseconds I1102 21:07:04.101164 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.101168 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.101172 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.101176 104483 round_trippers.go:414] Content-Length: 468 I1102 21:07:04.101179 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.101191 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"autoscaling/v1","resources":[{"name":"horizontalpodautoscalers","singularName":"","namespaced":true,"kind":"HorizontalPodAutoscaler","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["hpa"],"categories":["all"]},{"name":"horizontalpodautoscalers/status","singularName":"","namespaced":true,"kind":"HorizontalPodAutoscaler","verbs":["get","patch","update"]}]} I1102 21:07:04.101474 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/batch/v1 I1102 21:07:04.102175 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/batch/v1 200 OK in 0 milliseconds I1102 21:07:04.102188 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.102196 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.102201 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.102208 104483 round_trippers.go:414] Content-Length: 361 I1102 21:07:04.102211 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.102227 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"batch/v1","resources":[{"name":"jobs","singularName":"","namespaced":true,"kind":"Job","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"jobs/status","singularName":"","namespaced":true,"kind":"Job","verbs":["get","patch","update"]}]} I1102 21:07:04.102554 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/batch/v2alpha1 I1102 21:07:04.103331 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/batch/v2alpha1 200 OK in 0 milliseconds I1102 21:07:04.103342 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.103346 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.103350 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.103353 104483 round_trippers.go:414] Content-Length: 695 I1102 21:07:04.103357 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.103370 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"batch/v2alpha1","resources":[{"name":"cronjobs","singularName":"","namespaced":true,"kind":"CronJob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"cronjobs/status","singularName":"","namespaced":true,"kind":"CronJob","verbs":["get","patch","update"]},{"name":"scheduledjobs","singularName":"","namespaced":true,"kind":"ScheduledJob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"scheduledjobs/status","singularName":"","namespaced":true,"kind":"ScheduledJob","verbs":["get","patch","update"]}]} I1102 21:07:04.103706 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/certificates.k8s.io/v1beta1 I1102 21:07:04.104340 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/certificates.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.104354 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.104358 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.104362 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.104366 104483 round_trippers.go:414] Content-Length: 594 I1102 21:07:04.104369 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.104390 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"certificates.k8s.io/v1beta1","resources":[{"name":"certificatesigningrequests","singularName":"","namespaced":false,"kind":"CertificateSigningRequest","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["csr"]},{"name":"certificatesigningrequests/approval","singularName":"","namespaced":false,"kind":"CertificateSigningRequest","verbs":["update"]},{"name":"certificatesigningrequests/status","singularName":"","namespaced":false,"kind":"CertificateSigningRequest","verbs":["update"]}]} I1102 21:07:04.104735 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/networking.k8s.io/v1 I1102 21:07:04.105406 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/networking.k8s.io/v1 200 OK in 0 milliseconds I1102 21:07:04.105419 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.105423 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.105427 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.105431 104483 round_trippers.go:414] Content-Length: 268 I1102 21:07:04.105434 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.105454 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"networking.k8s.io/v1","resources":[{"name":"networkpolicies","singularName":"","namespaced":true,"kind":"NetworkPolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.105741 104483 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" -H "Accept: application/json, */*" https://internal-master.ci.openshift.org:8443/apis/policy/v1beta1 I1102 21:07:04.106471 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/policy/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.106484 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.106488 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.106492 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.106496 104483 round_trippers.go:414] Content-Length: 431 I1102 21:07:04.106499 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.106512 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"policy/v1beta1","resources":[{"name":"poddisruptionbudgets","singularName":"","namespaced":true,"kind":"PodDisruptionBudget","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pdb"]},{"name":"poddisruptionbudgets/status","singularName":"","namespaced":true,"kind":"PodDisruptionBudget","verbs":["get","patch","update"]}]} I1102 21:07:04.106934 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/authorization.openshift.io/v1 I1102 21:07:04.107665 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/authorization.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.107693 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.107698 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.107702 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.107705 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.107709 104483 round_trippers.go:414] Content-Length: 1578 I1102 21:07:04.107735 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"authorization.openshift.io/v1","resources":[{"name":"clusterrolebindings","singularName":"","namespaced":false,"kind":"ClusterRoleBinding","verbs":["create","delete","get","list","patch","update"]},{"name":"clusterroles","singularName":"","namespaced":false,"kind":"ClusterRole","verbs":["create","delete","get","list","patch","update"]},{"name":"localresourceaccessreviews","singularName":"","namespaced":true,"kind":"LocalResourceAccessReview","verbs":["create"]},{"name":"localsubjectaccessreviews","singularName":"","namespaced":true,"kind":"LocalSubjectAccessReview","verbs":["create"]},{"name":"resourceaccessreviews","singularName":"","namespaced":false,"kind":"ResourceAccessReview","verbs":["create"]},{"name":"rolebindingrestrictions","singularName":"","namespaced":true,"kind":"RoleBindingRestriction","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"rolebindings","singularName":"","namespaced":true,"kind":"RoleBinding","verbs":["create","delete","get","list","patch","update"]},{"name":"roles","singularName":"","namespaced":true,"kind":"Role","verbs":["create","delete","get","list","patch","update"]},{"name":"selfsubjectrulesreviews","singularName":"","namespaced":true,"kind":"SelfSubjectRulesReview","verbs":["create"]},{"name":"subjectaccessreviews","singularName":"","namespaced":false,"kind":"SubjectAccessReview","verbs":["create"]},{"name":"subjectrulesreviews","singularName":"","namespaced":true,"kind":"SubjectRulesReview","verbs":["create"]}]} I1102 21:07:04.108090 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/rbac.authorization.k8s.io/v1beta1 I1102 21:07:04.108690 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/rbac.authorization.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.108702 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.108706 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.108710 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.108716 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.108719 104483 round_trippers.go:414] Content-Length: 776 I1102 21:07:04.108735 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"rbac.authorization.k8s.io/v1beta1","resources":[{"name":"clusterrolebindings","singularName":"","namespaced":false,"kind":"ClusterRoleBinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"clusterroles","singularName":"","namespaced":false,"kind":"ClusterRole","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"rolebindings","singularName":"","namespaced":true,"kind":"RoleBinding","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"roles","singularName":"","namespaced":true,"kind":"Role","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.109039 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/settings.k8s.io/v1alpha1 I1102 21:07:04.109620 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/settings.k8s.io/v1alpha1 200 OK in 0 milliseconds I1102 21:07:04.109631 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.109635 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.109639 104483 round_trippers.go:414] Content-Length: 263 I1102 21:07:04.109642 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.109646 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.109657 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"settings.k8s.io/v1alpha1","resources":[{"name":"podpresets","singularName":"","namespaced":true,"kind":"PodPreset","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.109938 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/storage.k8s.io/v1 I1102 21:07:04.110556 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/storage.k8s.io/v1 200 OK in 0 milliseconds I1102 21:07:04.110570 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.110579 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.110583 104483 round_trippers.go:414] Content-Length: 284 I1102 21:07:04.110586 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.110590 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.110603 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"storage.k8s.io/v1","resources":[{"name":"storageclasses","singularName":"","namespaced":false,"kind":"StorageClass","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sc"]}]} I1102 21:07:04.110887 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/storage.k8s.io/v1beta1 I1102 21:07:04.111554 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/storage.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.111566 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.111571 104483 round_trippers.go:414] Content-Length: 289 I1102 21:07:04.111575 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.111578 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.111582 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.111597 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"storage.k8s.io/v1beta1","resources":[{"name":"storageclasses","singularName":"","namespaced":false,"kind":"StorageClass","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sc"]}]} I1102 21:07:04.111900 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/apiextensions.k8s.io/v1beta1 I1102 21:07:04.112498 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/apiextensions.k8s.io/v1beta1 200 OK in 0 milliseconds I1102 21:07:04.112509 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.112513 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.112516 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.112520 104483 round_trippers.go:414] Content-Length: 453 I1102 21:07:04.112523 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.112559 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apiextensions.k8s.io/v1beta1","resources":[{"name":"customresourcedefinitions","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["crd"]},{"name":"customresourcedefinitions/status","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["update"]}]} I1102 21:07:04.112852 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/apps.openshift.io/v1 I1102 21:07:04.113471 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/apps.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.113481 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.113485 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.113488 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.113492 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.113496 104483 round_trippers.go:414] Content-Length: 922 I1102 21:07:04.113512 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apps.openshift.io/v1","resources":[{"name":"deploymentconfigs","singularName":"","namespaced":true,"kind":"DeploymentConfig","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["dc"],"categories":["all"]},{"name":"deploymentconfigs/instantiate","singularName":"","namespaced":true,"kind":"DeploymentRequest","verbs":["create"]},{"name":"deploymentconfigs/log","singularName":"","namespaced":true,"kind":"DeploymentLog","verbs":["get"]},{"name":"deploymentconfigs/rollback","singularName":"","namespaced":true,"kind":"DeploymentConfigRollback","verbs":["create"]},{"name":"deploymentconfigs/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"deploymentconfigs/status","singularName":"","namespaced":true,"kind":"DeploymentConfig","verbs":["get","patch","update"]}]} I1102 21:07:04.113843 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/build.openshift.io/v1 I1102 21:07:04.114434 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/build.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.114444 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.114447 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.114451 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.114455 104483 round_trippers.go:414] Content-Length: 1101 I1102 21:07:04.114458 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.114476 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"build.openshift.io/v1","resources":[{"name":"buildconfigs","singularName":"","namespaced":true,"kind":"BuildConfig","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["bc"],"categories":["all"]},{"name":"buildconfigs/instantiate","singularName":"","namespaced":true,"kind":"BuildRequest","verbs":["create"]},{"name":"buildconfigs/instantiatebinary","singularName":"","namespaced":true,"kind":"BinaryBuildRequestOptions","verbs":[]},{"name":"buildconfigs/webhooks","singularName":"","namespaced":true,"kind":"Build","verbs":[]},{"name":"builds","singularName":"","namespaced":true,"kind":"Build","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"builds/clone","singularName":"","namespaced":true,"kind":"BuildRequest","verbs":["create"]},{"name":"builds/details","singularName":"","namespaced":true,"kind":"Build","verbs":["update"]},{"name":"builds/log","singularName":"","namespaced":true,"kind":"BuildLog","verbs":["get"]}]} I1102 21:07:04.114813 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/image.openshift.io/v1 I1102 21:07:04.115492 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/image.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.115503 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.115507 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.115511 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.115514 104483 round_trippers.go:414] Content-Length: 1327 I1102 21:07:04.115518 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.115564 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"image.openshift.io/v1","resources":[{"name":"images","singularName":"","namespaced":false,"kind":"Image","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"imagesignatures","singularName":"","namespaced":false,"kind":"ImageSignature","verbs":["create","delete"]},{"name":"imagestreamimages","singularName":"","namespaced":true,"kind":"ImageStreamImage","verbs":["get"],"shortNames":["isimage"]},{"name":"imagestreamimports","singularName":"","namespaced":true,"kind":"ImageStreamImport","verbs":["create"]},{"name":"imagestreammappings","singularName":"","namespaced":true,"kind":"ImageStreamMapping","verbs":["create"]},{"name":"imagestreams","singularName":"","namespaced":true,"kind":"ImageStream","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["is"],"categories":["all"]},{"name":"imagestreams/secrets","singularName":"","namespaced":true,"kind":"SecretList","verbs":["get"]},{"name":"imagestreams/status","singularName":"","namespaced":true,"kind":"ImageStream","verbs":["get","patch","update"]},{"name":"imagestreamtags","singularName":"","namespaced":true,"kind":"ImageStreamTag","verbs":["create","delete","get","list","patch","update"],"shortNames":["istag"]}]} I1102 21:07:04.115898 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/network.openshift.io/v1 I1102 21:07:04.116526 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/network.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.116555 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.116559 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.116563 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.116566 104483 round_trippers.go:414] Content-Length: 790 I1102 21:07:04.116570 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.116583 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"network.openshift.io/v1","resources":[{"name":"clusternetworks","singularName":"","namespaced":false,"kind":"ClusterNetwork","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"egressnetworkpolicies","singularName":"","namespaced":true,"kind":"EgressNetworkPolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"hostsubnets","singularName":"","namespaced":false,"kind":"HostSubnet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"netnamespaces","singularName":"","namespaced":false,"kind":"NetNamespace","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.116898 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/oauth.openshift.io/v1 I1102 21:07:04.117482 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/oauth.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.117492 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.117496 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.117500 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.117503 104483 round_trippers.go:414] Content-Length: 818 I1102 21:07:04.117507 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.117520 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"oauth.openshift.io/v1","resources":[{"name":"oauthaccesstokens","singularName":"","namespaced":false,"kind":"OAuthAccessToken","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthauthorizetokens","singularName":"","namespaced":false,"kind":"OAuthAuthorizeToken","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthclientauthorizations","singularName":"","namespaced":false,"kind":"OAuthClientAuthorization","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthclients","singularName":"","namespaced":false,"kind":"OAuthClient","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.117860 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/project.openshift.io/v1 I1102 21:07:04.118455 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/project.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.118465 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.118469 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.118473 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.118476 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.118479 104483 round_trippers.go:414] Content-Length: 354 I1102 21:07:04.118496 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"project.openshift.io/v1","resources":[{"name":"projectrequests","singularName":"","namespaced":false,"kind":"ProjectRequest","verbs":["create","list"]},{"name":"projects","singularName":"","namespaced":false,"kind":"Project","verbs":["create","delete","get","list","patch","update","watch"]}]} I1102 21:07:04.118819 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/quota.openshift.io/v1 I1102 21:07:04.119422 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/quota.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.119435 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.119440 104483 round_trippers.go:414] Content-Length: 589 I1102 21:07:04.119443 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.119447 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.119450 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.119469 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"quota.openshift.io/v1","resources":[{"name":"appliedclusterresourcequotas","singularName":"","namespaced":true,"kind":"AppliedClusterResourceQuota","verbs":["get","list"]},{"name":"clusterresourcequotas","singularName":"","namespaced":false,"kind":"ClusterResourceQuota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["clusterquota"]},{"name":"clusterresourcequotas/status","singularName":"","namespaced":false,"kind":"ClusterResourceQuota","verbs":["get","patch","update"]}]} I1102 21:07:04.119819 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/route.openshift.io/v1 I1102 21:07:04.120467 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/route.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.120481 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.120485 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.120489 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.120492 104483 round_trippers.go:414] Content-Length: 382 I1102 21:07:04.120497 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.120510 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"route.openshift.io/v1","resources":[{"name":"routes","singularName":"","namespaced":true,"kind":"Route","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"routes/status","singularName":"","namespaced":true,"kind":"Route","verbs":["get","patch","update"]}]} I1102 21:07:04.120877 104483 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" -H "Accept: application/json, */*" https://internal-master.ci.openshift.org:8443/apis/security.openshift.io/v1 I1102 21:07:04.121525 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/security.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.121538 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.121542 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.121546 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.121549 104483 round_trippers.go:414] Content-Length: 726 I1102 21:07:04.121553 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.121580 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"security.openshift.io/v1","resources":[{"name":"podsecuritypolicyreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicyReview","verbs":["create"]},{"name":"podsecuritypolicyselfsubjectreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicySelfSubjectReview","verbs":["create"]},{"name":"podsecuritypolicysubjectreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicySubjectReview","verbs":["create"]},{"name":"securitycontextconstraints","singularName":"","namespaced":false,"kind":"SecurityContextConstraints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["scc"]}]} I1102 21:07:04.121935 104483 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" -H "Accept: application/json, */*" https://internal-master.ci.openshift.org:8443/apis/template.openshift.io/v1 I1102 21:07:04.122499 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/template.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.122509 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.122513 104483 round_trippers.go:414] Content-Length: 860 I1102 21:07:04.122516 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.122520 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.122523 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.122541 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"template.openshift.io/v1","resources":[{"name":"brokertemplateinstances","singularName":"","namespaced":false,"kind":"BrokerTemplateInstance","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"processedtemplates","singularName":"","namespaced":true,"kind":"Template","verbs":["create"]},{"name":"templateinstances","singularName":"","namespaced":true,"kind":"TemplateInstance","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"templateinstances/status","singularName":"","namespaced":true,"kind":"TemplateInstance","verbs":["get","patch","update"]},{"name":"templates","singularName":"","namespaced":true,"kind":"Template","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.122858 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/user.openshift.io/v1 I1102 21:07:04.123460 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/user.openshift.io/v1 200 OK in 0 milliseconds I1102 21:07:04.123471 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.123475 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.123479 104483 round_trippers.go:414] Content-Length: 716 I1102 21:07:04.123482 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.123486 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.123513 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"user.openshift.io/v1","resources":[{"name":"groups","singularName":"","namespaced":false,"kind":"Group","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"identities","singularName":"","namespaced":false,"kind":"Identity","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"useridentitymappings","singularName":"","namespaced":false,"kind":"UserIdentityMapping","verbs":["create","delete","get","patch","update"]},{"name":"users","singularName":"","namespaced":false,"kind":"User","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:04.123857 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/prow.k8s.io/v1 I1102 21:07:04.124462 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/prow.k8s.io/v1 200 OK in 0 milliseconds I1102 21:07:04.124472 104483 round_trippers.go:411] Response Headers: I1102 21:07:04.124476 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:04 GMT I1102 21:07:04.124480 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:04.124484 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:04.124487 104483 round_trippers.go:414] Content-Length: 276 I1102 21:07:04.124499 104483 request.go:994] Response Body: {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"prow.k8s.io/v1","resources":[{"name":"prowjobs","singularName":"prowjob","namespaced":true,"kind":"ProwJob","verbs":["delete","deletecollection","get","list","patch","create","update","watch"],"shortNames":["pj"]}]} I1102 21:07:04.124823 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:07:34.126194 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30001 milliseconds I1102 21:07:34.126221 104483 round_trippers.go:411] Response Headers: I1102 21:07:34.126226 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:34.126230 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:07:34.126234 104483 round_trippers.go:414] Content-Length: 115 I1102 21:07:34.126238 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:34 GMT I1102 21:07:34.126261 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:07:34.126480 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:07:34.126512 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:07:34.126613 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/api/v1 I1102 21:07:34.128273 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/api/v1 200 OK in 1 milliseconds I1102 21:07:34.128297 104483 round_trippers.go:411] Response Headers: I1102 21:07:34.128302 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:34.128307 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:34.128310 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:34 GMT I1102 21:07:34.128388 104483 request.go:994] Response Body: {"kind":"APIResourceList","groupVersion":"v1","resources":[{"name":"bindings","singularName":"","namespaced":true,"kind":"Binding","verbs":["create"]},{"name":"componentstatuses","singularName":"","namespaced":false,"kind":"ComponentStatus","verbs":["get","list"],"shortNames":["cs"]},{"name":"configmaps","singularName":"","namespaced":true,"kind":"ConfigMap","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cm"]},{"name":"endpoints","singularName":"","namespaced":true,"kind":"Endpoints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ep"]},{"name":"events","singularName":"","namespaced":true,"kind":"Event","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ev"]},{"name":"limitranges","singularName":"","namespaced":true,"kind":"LimitRange","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["limits"]},{"name":"namespaces","singularName":"","namespaced":false,"kind":"Namespace","verbs":["create","delete","get","list","patch","update","watch"],"shortNames":["ns"]},{"name":"namespaces/finalize","singularName":"","namespaced":false,"kind":"Namespace","verbs":["update"]},{"name":"namespaces/status","singularName":"","namespaced":false,"kind":"Namespace","verbs":["get","patch","update"]},{"name":"nodes","singularName":"","namespaced":false,"kind":"Node","verbs":["create","delete","deletecollection","get","list","patch","proxy","update","watch"],"shortNames":["no"]},{"name":"nodes/proxy","singularName":"","namespaced":false,"kind":"Node","verbs":[]},{"name":"nodes/status","singularName":"","namespaced":false,"kind":"Node","verbs":["get","patch","update"]},{"name":"persistentvolumeclaims","singularName":"","namespaced":true,"kind":"PersistentVolumeClaim","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pvc"]},{"name":"persistentvolumeclaims/status","singularName":"","namespaced":true,"kind":"PersistentVolumeClaim","verbs":["get","patch","update"]},{"name":"persistentvolumes","singularName":"","namespaced":false,"kind":"PersistentVolume","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["pv"]},{"name":"persistentvolumes/status","singularName":"","namespaced":false,"kind":"PersistentVolume","verbs":["get","patch","update"]},{"name":"pods","singularName":"","namespaced":true,"kind":"Pod","verbs":["create","delete","deletecollection","get","list","patch","proxy","update","watch"],"shortNames":["po"],"categories":["all"]},{"name":"pods/attach","singularName":"","namespaced":true,"kind":"Pod","verbs":[]},{"name":"pods/binding","singularName":"","namespaced":true,"kind":"Binding","verbs":["create"]},{"name":"pods/eviction","singularName":"","namespaced":true,"kind":"Eviction","verbs":["create"]},{"name":"pods/exec","singularName":"","namespaced":true,"kind":"Pod","verbs":[]},{"name":"pods/log","singularName":"","namespaced":true,"kind":"Pod","verbs":["get"]},{"name":"pods/portforward","singularName":"","namespaced":true,"kind":"Pod","verbs":[]},{"name":"pods/proxy","singularName":"","namespaced":true,"kind":"Pod","verbs":[]},{"name":"pods/status","singularName":"","namespaced":true,"kind":"Pod","verbs":["get","patch","update"]},{"name":"podtemplates","singularName":"","namespaced":true,"kind":"PodTemplate","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"replicationcontrollers","singularName":"","namespaced":true,"kind":"ReplicationController","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"]},{"name":"replicationcontrollers/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"replicationcontrollers/status","singularName":"","namespaced":true,"kind":"ReplicationController","verbs":["get","patch","update"]},{"name":"resourcequotas","singularName":"","namespaced":true,"kind":"ResourceQuota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["quota"]},{"name":"resourcequotas/status","singularName":"","namespaced":true,"kind":"ResourceQuota","verbs":["get","patch","update"]},{"name":"secrets","singularName":"","namespaced":true,"kind":"Secret","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"securitycontextconstraints","singularName":"","namespaced":false,"kind":"SecurityContextConstraints","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["scc"]},{"name":"serviceaccounts","singularName":"","namespaced":true,"kind":"ServiceAccount","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sa"]},{"name":"services","singularName":"","namespaced":true,"kind":"Service","verbs":["create","delete","get","list","patch","proxy","update","watch"]},{"name":"services/proxy","singularName":"","namespaced":true,"kind":"Service","verbs":[]},{"name":"services/status","singularName":"","namespaced":true,"kind":"Service","verbs":["get","patch","update"]}]} I1102 21:07:34.128792 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/oapi/v1 I1102 21:07:34.130155 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/oapi/v1 200 OK in 1 milliseconds I1102 21:07:34.130168 104483 round_trippers.go:411] Response Headers: I1102 21:07:34.130172 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:07:34 GMT I1102 21:07:34.130176 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:07:34.130180 104483 round_trippers.go:414] Content-Type: application/json I1102 21:07:34.130284 104483 request.go:994] Response Body: {"kind":"APIResourceList","groupVersion":"v1","resources":[{"name":"appliedclusterresourcequotas","singularName":"","namespaced":true,"kind":"AppliedClusterResourceQuota","verbs":["get","list"]},{"name":"buildconfigs","singularName":"","namespaced":true,"kind":"BuildConfig","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["bc"],"categories":["all"]},{"name":"buildconfigs/instantiate","singularName":"","namespaced":true,"kind":"BuildRequest","verbs":["create"]},{"name":"buildconfigs/instantiatebinary","singularName":"","namespaced":true,"kind":"BinaryBuildRequestOptions","verbs":[]},{"name":"buildconfigs/webhooks","singularName":"","namespaced":true,"kind":"Build","verbs":[]},{"name":"builds","singularName":"","namespaced":true,"kind":"Build","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"builds/clone","singularName":"","namespaced":true,"kind":"BuildRequest","verbs":["create"]},{"name":"builds/details","singularName":"","namespaced":true,"kind":"Build","verbs":["update"]},{"name":"builds/log","singularName":"","namespaced":true,"kind":"BuildLog","verbs":["get"]},{"name":"clusternetworks","singularName":"","namespaced":false,"kind":"ClusterNetwork","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"clusterresourcequotas","singularName":"","namespaced":false,"kind":"ClusterResourceQuota","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["clusterquota"]},{"name":"clusterresourcequotas/status","singularName":"","namespaced":false,"kind":"ClusterResourceQuota","verbs":["get","patch","update"]},{"name":"clusterrolebindings","singularName":"","namespaced":false,"kind":"ClusterRoleBinding","verbs":["create","delete","get","list","patch","update"]},{"name":"clusterroles","singularName":"","namespaced":false,"kind":"ClusterRole","verbs":["create","delete","get","list","patch","update"]},{"name":"deploymentconfigrollbacks","singularName":"","namespaced":true,"kind":"DeploymentConfigRollback","verbs":["create"]},{"name":"deploymentconfigs","singularName":"","namespaced":true,"kind":"DeploymentConfig","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["dc"],"categories":["all"]},{"name":"deploymentconfigs/instantiate","singularName":"","namespaced":true,"kind":"DeploymentRequest","verbs":["create"]},{"name":"deploymentconfigs/log","singularName":"","namespaced":true,"kind":"DeploymentLog","verbs":["get"]},{"name":"deploymentconfigs/rollback","singularName":"","namespaced":true,"kind":"DeploymentConfigRollback","verbs":["create"]},{"name":"deploymentconfigs/scale","singularName":"","namespaced":true,"kind":"Scale","verbs":["get","patch","update"]},{"name":"deploymentconfigs/status","singularName":"","namespaced":true,"kind":"DeploymentConfig","verbs":["get","patch","update"]},{"name":"egressnetworkpolicies","singularName":"","namespaced":true,"kind":"EgressNetworkPolicy","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"groups","singularName":"","namespaced":false,"kind":"Group","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"hostsubnets","singularName":"","namespaced":false,"kind":"HostSubnet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"identities","singularName":"","namespaced":false,"kind":"Identity","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"images","singularName":"","namespaced":false,"kind":"Image","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"imagesignatures","singularName":"","namespaced":false,"kind":"ImageSignature","verbs":["create","delete"]},{"name":"imagestreamimages","singularName":"","namespaced":true,"kind":"ImageStreamImage","verbs":["get"],"shortNames":["isimage"]},{"name":"imagestreamimports","singularName":"","namespaced":true,"kind":"ImageStreamImport","verbs":["create"]},{"name":"imagestreammappings","singularName":"","namespaced":true,"kind":"ImageStreamMapping","verbs":["create"]},{"name":"imagestreams","singularName":"","namespaced":true,"kind":"ImageStream","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["is"],"categories":["all"]},{"name":"imagestreams/secrets","singularName":"","namespaced":true,"kind":"SecretList","verbs":["get"]},{"name":"imagestreams/status","singularName":"","namespaced":true,"kind":"ImageStream","verbs":["get","patch","update"]},{"name":"imagestreamtags","singularName":"","namespaced":true,"kind":"ImageStreamTag","verbs":["create","delete","get","list","patch","update"],"shortNames":["istag"]},{"name":"localresourceaccessreviews","singularName":"","namespaced":true,"kind":"LocalResourceAccessReview","verbs":["create"]},{"name":"localsubjectaccessreviews","singularName":"","namespaced":true,"kind":"LocalSubjectAccessReview","verbs":["create"]},{"name":"netnamespaces","singularName":"","namespaced":false,"kind":"NetNamespace","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthaccesstokens","singularName":"","namespaced":false,"kind":"OAuthAccessToken","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthauthorizetokens","singularName":"","namespaced":false,"kind":"OAuthAuthorizeToken","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthclientauthorizations","singularName":"","namespaced":false,"kind":"OAuthClientAuthorization","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"oauthclients","singularName":"","namespaced":false,"kind":"OAuthClient","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"podsecuritypolicyreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicyReview","verbs":["create"]},{"name":"podsecuritypolicyselfsubjectreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicySelfSubjectReview","verbs":["create"]},{"name":"podsecuritypolicysubjectreviews","singularName":"","namespaced":true,"kind":"PodSecurityPolicySubjectReview","verbs":["create"]},{"name":"processedtemplates","singularName":"","namespaced":true,"kind":"Template","verbs":["create"]},{"name":"projectrequests","singularName":"","namespaced":false,"kind":"ProjectRequest","verbs":["create","list"]},{"name":"projects","singularName":"","namespaced":false,"kind":"Project","verbs":["create","delete","get","list","patch","update","watch"]},{"name":"resourceaccessreviews","singularName":"","namespaced":true,"kind":"ResourceAccessReview","verbs":["create"]},{"name":"rolebindingrestrictions","singularName":"","namespaced":true,"kind":"RoleBindingRestriction","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"rolebindings","singularName":"","namespaced":true,"kind":"RoleBinding","verbs":["create","delete","get","list","patch","update"]},{"name":"roles","singularName":"","namespaced":true,"kind":"Role","verbs":["create","delete","get","list","patch","update"]},{"name":"routes","singularName":"","namespaced":true,"kind":"Route","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]},{"name":"routes/status","singularName":"","namespaced":true,"kind":"Route","verbs":["get","patch","update"]},{"name":"selfsubjectrulesreviews","singularName":"","namespaced":true,"kind":"SelfSubjectRulesReview","verbs":["create"]},{"name":"subjectaccessreviews","singularName":"","namespaced":true,"kind":"SubjectAccessReview","verbs":["create"]},{"name":"subjectrulesreviews","singularName":"","namespaced":true,"kind":"SubjectRulesReview","verbs":["create"]},{"name":"templates","singularName":"","namespaced":true,"kind":"Template","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"useridentitymappings","singularName":"","namespaced":false,"kind":"UserIdentityMapping","verbs":["create","delete","get","patch","update"]},{"name":"users","singularName":"","namespaced":false,"kind":"User","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]}]} I1102 21:07:34.133184 104483 cached_discovery.go:119] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/servergroups.json I1102 21:07:34.133260 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiregistration.k8s.io/v1beta1/serverresources.json I1102 21:07:34.133416 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/extensions/v1beta1/serverresources.json I1102 21:07:34.133519 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps/v1beta1/serverresources.json I1102 21:07:34.133614 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1/serverresources.json I1102 21:07:34.133664 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1beta1/serverresources.json I1102 21:07:34.133752 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1/serverresources.json I1102 21:07:34.134817 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1beta1/serverresources.json I1102 21:07:34.134894 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/autoscaling/v1/serverresources.json I1102 21:07:34.134956 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v1/serverresources.json I1102 21:07:34.135095 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v2alpha1/serverresources.json I1102 21:07:34.135167 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/certificates.k8s.io/v1beta1/serverresources.json I1102 21:07:34.135226 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/networking.k8s.io/v1/serverresources.json I1102 21:07:34.135324 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/policy/v1beta1/serverresources.json I1102 21:07:34.135427 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.openshift.io/v1/serverresources.json I1102 21:07:34.135501 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json I1102 21:07:34.135555 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/settings.k8s.io/v1alpha1/serverresources.json I1102 21:07:34.135602 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1/serverresources.json I1102 21:07:34.135646 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1beta1/serverresources.json I1102 21:07:34.135709 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiextensions.k8s.io/v1beta1/serverresources.json I1102 21:07:34.135784 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps.openshift.io/v1/serverresources.json I1102 21:07:34.135862 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/build.openshift.io/v1/serverresources.json I1102 21:07:34.135957 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/image.openshift.io/v1/serverresources.json I1102 21:07:34.136031 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/network.openshift.io/v1/serverresources.json I1102 21:07:34.136100 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/oauth.openshift.io/v1/serverresources.json I1102 21:07:34.136147 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/project.openshift.io/v1/serverresources.json I1102 21:07:34.136224 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/quota.openshift.io/v1/serverresources.json I1102 21:07:34.136273 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/route.openshift.io/v1/serverresources.json I1102 21:07:34.136332 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/security.openshift.io/v1/serverresources.json I1102 21:07:34.136393 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/template.openshift.io/v1/serverresources.json I1102 21:07:34.136456 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/user.openshift.io/v1/serverresources.json I1102 21:07:34.136501 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/prow.k8s.io/v1/serverresources.json I1102 21:07:34.136563 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:08:04.138071 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30001 milliseconds I1102 21:08:04.138098 104483 round_trippers.go:411] Response Headers: I1102 21:08:04.138103 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:08:04 GMT I1102 21:08:04.138108 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:08:04.138111 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:08:04.138115 104483 round_trippers.go:414] Content-Length: 115 I1102 21:08:04.138136 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:08:04.138282 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:08:04.138297 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:08:04.138668 104483 cached_discovery.go:119] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/servergroups.json I1102 21:08:04.138783 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiregistration.k8s.io/v1beta1/serverresources.json I1102 21:08:04.138920 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/extensions/v1beta1/serverresources.json I1102 21:08:04.139004 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps/v1beta1/serverresources.json I1102 21:08:04.139044 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1/serverresources.json I1102 21:08:04.139103 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1beta1/serverresources.json I1102 21:08:04.139151 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1/serverresources.json I1102 21:08:04.139212 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1beta1/serverresources.json I1102 21:08:04.139257 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/autoscaling/v1/serverresources.json I1102 21:08:04.139305 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v1/serverresources.json I1102 21:08:04.139393 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v2alpha1/serverresources.json I1102 21:08:04.139478 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/certificates.k8s.io/v1beta1/serverresources.json I1102 21:08:04.139526 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/networking.k8s.io/v1/serverresources.json I1102 21:08:04.139574 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/policy/v1beta1/serverresources.json I1102 21:08:04.139667 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.openshift.io/v1/serverresources.json I1102 21:08:04.139753 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json I1102 21:08:04.139793 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/settings.k8s.io/v1alpha1/serverresources.json I1102 21:08:04.139838 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1/serverresources.json I1102 21:08:04.139881 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1beta1/serverresources.json I1102 21:08:04.139932 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiextensions.k8s.io/v1beta1/serverresources.json I1102 21:08:04.140015 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps.openshift.io/v1/serverresources.json I1102 21:08:04.140105 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/build.openshift.io/v1/serverresources.json I1102 21:08:04.140186 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/image.openshift.io/v1/serverresources.json I1102 21:08:04.140279 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/network.openshift.io/v1/serverresources.json I1102 21:08:04.140355 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/oauth.openshift.io/v1/serverresources.json I1102 21:08:04.140446 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/project.openshift.io/v1/serverresources.json I1102 21:08:04.140505 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/quota.openshift.io/v1/serverresources.json I1102 21:08:04.140566 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/route.openshift.io/v1/serverresources.json I1102 21:08:04.140622 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/security.openshift.io/v1/serverresources.json I1102 21:08:04.140702 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/template.openshift.io/v1/serverresources.json I1102 21:08:04.140777 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/user.openshift.io/v1/serverresources.json I1102 21:08:04.140841 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/prow.k8s.io/v1/serverresources.json I1102 21:08:04.140916 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:08:34.142415 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30001 milliseconds I1102 21:08:34.142446 104483 round_trippers.go:411] Response Headers: I1102 21:08:34.142452 104483 round_trippers.go:414] Content-Length: 115 I1102 21:08:34.142456 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:08:34 GMT I1102 21:08:34.142460 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:08:34.142464 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:08:34.142515 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:08:34.142698 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:08:34.142714 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:08:34.143083 104483 cached_discovery.go:119] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/servergroups.json I1102 21:08:34.143166 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiregistration.k8s.io/v1beta1/serverresources.json I1102 21:08:34.143308 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/extensions/v1beta1/serverresources.json I1102 21:08:34.143423 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps/v1beta1/serverresources.json I1102 21:08:34.143470 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1/serverresources.json I1102 21:08:34.143537 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1beta1/serverresources.json I1102 21:08:34.143600 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1/serverresources.json I1102 21:08:34.143661 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1beta1/serverresources.json I1102 21:08:34.143800 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/autoscaling/v1/serverresources.json I1102 21:08:34.144439 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v1/serverresources.json I1102 21:08:34.144690 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v2alpha1/serverresources.json I1102 21:08:34.144810 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/certificates.k8s.io/v1beta1/serverresources.json I1102 21:08:34.144958 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/networking.k8s.io/v1/serverresources.json I1102 21:08:34.145129 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/policy/v1beta1/serverresources.json I1102 21:08:34.145433 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.openshift.io/v1/serverresources.json I1102 21:08:34.145589 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json I1102 21:08:34.145706 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/settings.k8s.io/v1alpha1/serverresources.json I1102 21:08:34.145873 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1/serverresources.json I1102 21:08:34.146022 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1beta1/serverresources.json I1102 21:08:34.146132 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiextensions.k8s.io/v1beta1/serverresources.json I1102 21:08:34.146318 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps.openshift.io/v1/serverresources.json I1102 21:08:34.146589 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/build.openshift.io/v1/serverresources.json I1102 21:08:34.146866 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/image.openshift.io/v1/serverresources.json I1102 21:08:34.147010 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/network.openshift.io/v1/serverresources.json I1102 21:08:34.147144 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/oauth.openshift.io/v1/serverresources.json I1102 21:08:34.147248 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/project.openshift.io/v1/serverresources.json I1102 21:08:34.147411 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/quota.openshift.io/v1/serverresources.json I1102 21:08:34.147590 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/route.openshift.io/v1/serverresources.json I1102 21:08:34.147770 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/security.openshift.io/v1/serverresources.json I1102 21:08:34.147896 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/template.openshift.io/v1/serverresources.json I1102 21:08:34.148007 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/user.openshift.io/v1/serverresources.json I1102 21:08:34.148100 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/prow.k8s.io/v1/serverresources.json I1102 21:08:34.148200 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:09:04.152320 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30004 milliseconds I1102 21:09:04.152354 104483 round_trippers.go:411] Response Headers: I1102 21:09:04.152359 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:09:04.152363 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:09:04.152367 104483 round_trippers.go:414] Content-Length: 115 I1102 21:09:04.152371 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:09:04 GMT I1102 21:09:04.152393 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:09:04.152574 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:09:04.152589 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:09:04.153272 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/v1/serverresources.json I1102 21:09:04.157876 104483 cached_discovery.go:119] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/servergroups.json I1102 21:09:04.157963 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiregistration.k8s.io/v1beta1/serverresources.json I1102 21:09:04.158115 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/extensions/v1beta1/serverresources.json I1102 21:09:04.158204 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps/v1beta1/serverresources.json I1102 21:09:04.158249 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1/serverresources.json I1102 21:09:04.158291 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1beta1/serverresources.json I1102 21:09:04.158342 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1/serverresources.json I1102 21:09:04.158397 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1beta1/serverresources.json I1102 21:09:04.158449 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/autoscaling/v1/serverresources.json I1102 21:09:04.158500 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v1/serverresources.json I1102 21:09:04.158568 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v2alpha1/serverresources.json I1102 21:09:04.158627 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/certificates.k8s.io/v1beta1/serverresources.json I1102 21:09:04.158668 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/networking.k8s.io/v1/serverresources.json I1102 21:09:04.158763 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/policy/v1beta1/serverresources.json I1102 21:09:04.158858 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.openshift.io/v1/serverresources.json I1102 21:09:04.158919 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json I1102 21:09:04.158958 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/settings.k8s.io/v1alpha1/serverresources.json I1102 21:09:04.158999 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1/serverresources.json I1102 21:09:04.159039 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1beta1/serverresources.json I1102 21:09:04.159097 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiextensions.k8s.io/v1beta1/serverresources.json I1102 21:09:04.159177 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps.openshift.io/v1/serverresources.json I1102 21:09:04.159252 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/build.openshift.io/v1/serverresources.json I1102 21:09:04.159336 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/image.openshift.io/v1/serverresources.json I1102 21:09:04.159395 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/network.openshift.io/v1/serverresources.json I1102 21:09:04.159455 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/oauth.openshift.io/v1/serverresources.json I1102 21:09:04.159502 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/project.openshift.io/v1/serverresources.json I1102 21:09:04.159566 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/quota.openshift.io/v1/serverresources.json I1102 21:09:04.159617 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/route.openshift.io/v1/serverresources.json I1102 21:09:04.159686 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/security.openshift.io/v1/serverresources.json I1102 21:09:04.159820 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/template.openshift.io/v1/serverresources.json I1102 21:09:04.159911 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/user.openshift.io/v1/serverresources.json I1102 21:09:04.159961 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/prow.k8s.io/v1/serverresources.json I1102 21:09:04.160043 104483 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" -H "Accept: application/json, */*" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:09:34.161483 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30001 milliseconds I1102 21:09:34.161508 104483 round_trippers.go:411] Response Headers: I1102 21:09:34.161513 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:09:34.161517 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:09:34.161521 104483 round_trippers.go:414] Content-Length: 115 I1102 21:09:34.161525 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:09:34 GMT I1102 21:09:34.161553 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:09:34.161728 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:09:34.161743 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:09:34.162487 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/v1/serverresources.json I1102 21:09:34.163054 104483 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" -H "Accept: application/json" https://internal-master.ci.openshift.org:8443/api/v1/namespaces/default/pods I1102 21:09:34.166386 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/api/v1/namespaces/default/pods 200 OK in 3 milliseconds I1102 21:09:34.166400 104483 round_trippers.go:411] Response Headers: I1102 21:09:34.166410 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:09:34 GMT I1102 21:09:34.166414 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:09:34.166418 104483 round_trippers.go:414] Content-Type: application/json I1102 21:09:34.166516 104483 request.go:994] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/default/pods","resourceVersion":"11633652"},"items":[{"metadata":{"name":"docker-registry-2-qrd58","generateName":"docker-registry-2-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/docker-registry-2-qrd58","uid":"42885f39-c00c-11e7-88ab-42010a800002","resourceVersion":"11633633","creationTimestamp":"2017-11-02T20:27:34Z","labels":{"deployment":"docker-registry-2","deploymentconfig":"docker-registry","docker-registry":"default"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"docker-registry-2\",\"uid\":\"980dceb2-91d0-11e7-bd36-42010a800002\",\"apiVersion\":\"v1\",\"resourceVersion\":\"11615005\"}}\n","openshift.io/deployment-config.latest-version":"2","openshift.io/deployment-config.name":"docker-registry","openshift.io/deployment.name":"docker-registry-2","openshift.io/scc":"hostnetwork"},"ownerReferences":[{"apiVersion":"v1","kind":"ReplicationController","name":"docker-registry-2","uid":"980dceb2-91d0-11e7-bd36-42010a800002","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"registry-storage","emptyDir":{"sizeLimit":"0"}},{"name":"registry-certificates","secret":{"secretName":"registry-tls","defaultMode":420}},{"name":"docker-config","secret":{"secretName":"registry-config","defaultMode":420}},{"name":"registry-token-p27q0","secret":{"secretName":"registry-token-p27q0","defaultMode":420}}],"containers":[{"name":"registry","image":"openshift/origin-docker-registry:v3.7.0-alpha.1","ports":[{"containerPort":5000,"protocol":"TCP"}],"env":[{"name":"REGISTRY_HTTP_ADDR","value":":5000"},{"name":"REGISTRY_HTTP_NET","value":"tcp"},{"name":"REGISTRY_HTTP_SECRET","value":"978QSoiTGzaQe1J4ISPLfSWVtTpXPb4a1fVFxZ8H7WM="},{"name":"REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA","value":"false"},{"name":"REGISTRY_HTTP_TLS_KEY","value":"/etc/secrets/tls.key"},{"name":"OPENSHIFT_DEFAULT_REGISTRY","value":"docker-registry.default.svc:5000"},{"name":"REGISTRY_CONFIGURATION_PATH","value":"/etc/registry/config.yml"},{"name":"REGISTRY_HTTP_TLS_CERTIFICATE","value":"/etc/secrets/tls.crt"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"registry-storage","mountPath":"/registry"},{"name":"registry-certificates","mountPath":"/etc/secrets"},{"name":"docker-config","mountPath":"/etc/registry"},{"name":"registry-token-p27q0","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":5000,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":5000,"scheme":"HTTPS"},"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c5,c0"},"runAsUser":1000020000}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"role":"infra"},"serviceAccountName":"registry","serviceAccount":"registry","nodeName":"origin-ci-ig-m-11v4","securityContext":{"seLinuxOptions":{"level":"s0:c5,c0"},"supplementalGroups":[1000020000],"fsGroup":1000020000},"imagePullSecrets":[{"name":"registry-dockercfg-sh4kb"}],"schedulerName":"default-scheduler"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:39Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:39Z","reason":"ContainersNotReady","message":"containers with unready status: [registry]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:39Z"}],"hostIP":"10.128.0.2","podIP":"172.16.6.78","startTime":"2017-11-02T20:30:39Z","containerStatuses":[{"name":"registry","state":{"running":{"startedAt":"2017-11-02T21:09:26Z"}},"lastState":{"terminated":{"exitCode":2,"reason":"Error","startedAt":"2017-11-02T21:03:45Z","finishedAt":"2017-11-02T21:04:15Z","containerID":"docker://eb767307bb979a1a6f51e3292ae43114382aafb27197273e583715e806327918"}},"ready":false,"restartCount":16,"image":"docker.io/openshift/origin-docker-registry:v3.7.0-alpha.1","imageID":"docker-pullable://docker.io/openshift/origin-docker-registry@sha256:f72d40df9618108174c9b510e72625a903ba35edae6114d691d320a24e81c717","containerID":"docker://94fc83f5952997419d0a60d7486a8e426e7ca19a297c6742ec68b8654f42fd1a"}],"qosClass":"Burstable"}},{"metadata":{"name":"registry-console-1-lkfh6","generateName":"registry-console-1-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/registry-console-1-lkfh6","uid":"59e4993f-c00f-11e7-88ab-42010a800002","resourceVersion":"11630173","creationTimestamp":"2017-11-02T20:49:42Z","labels":{"app":"registry-console","deployment":"registry-console-1","deploymentconfig":"registry-console","name":"registry-console"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"registry-console-1\",\"uid\":\"cb7736e2-8d96-11e7-a412-42010a800002\",\"apiVersion\":\"v1\",\"resourceVersion\":\"11624813\"}}\n","openshift.io/deployment-config.latest-version":"1","openshift.io/deployment-config.name":"registry-console","openshift.io/deployment.name":"registry-console-1","openshift.io/generated-by":"OpenShiftNewApp","openshift.io/scc":"restricted"},"ownerReferences":[{"apiVersion":"v1","kind":"ReplicationController","name":"registry-console-1","uid":"cb7736e2-8d96-11e7-a412-42010a800002","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-1p9pr","secret":{"secretName":"default-token-1p9pr","defaultMode":420}}],"containers":[{"name":"registry-console","image":"cockpit/kubernetes:latest","ports":[{"containerPort":9090,"protocol":"TCP"}],"env":[{"name":"OPENSHIFT_OAUTH_PROVIDER_URL","value":"https://api.ci.openshift.org"},{"name":"OPENSHIFT_OAUTH_CLIENT_ID","value":"cockpit-oauth-client"},{"name":"KUBERNETES_INSECURE","value":"false"},{"name":"COCKPIT_KUBE_INSECURE","value":"false"},{"name":"REGISTRY_ONLY","value":"true"},{"name":"REGISTRY_HOST","value":"docker-registry-default.svc.ci.openshift.org"}],"resources":{},"volumeMounts":[{"name":"default-token-1p9pr","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/ping","port":9090,"scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/ping","port":9090,"scheme":"HTTP"},"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c5,c0"},"runAsUser":1000020000}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"origin-ci-ig-n-kwtq","securityContext":{"seLinuxOptions":{"level":"s0:c5,c0"},"fsGroup":1000020000},"imagePullSecrets":[{"name":"default-dockercfg-ndbw3"}],"schedulerName":"default-scheduler"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:49:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:50:12Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:49:42Z"}],"hostIP":"10.128.0.10","podIP":"172.16.15.134","startTime":"2017-11-02T20:49:42Z","containerStatuses":[{"name":"registry-console","state":{"running":{"startedAt":"2017-11-02T20:50:05Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/cockpit/kubernetes:latest","imageID":"docker-pullable://docker.io/cockpit/kubernetes@sha256:a4b418f80f5654021ff63526bbb9644f23a5cc56b6eb50d684f21c1aaf324256","containerID":"docker://aa30cd680ddced1d71b0fb1097d33e768663cd211c882b57edcd1674cb5a3635"}],"qosClass":"BestEffort"}},{"metadata":{"name":"router-1-wf6r0","generateName":"router-1-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/router-1-wf6r0","uid":"428d308f-c00c-11e7-88ab-42010a800002","resourceVersion":"11625382","creationTimestamp":"2017-11-02T20:27:34Z","labels":{"deployment":"router-1","deploymentconfig":"router","router":"router"},"annotations":{"kubernetes.io/created-by":"{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"router-1\",\"uid\":\"7d8e58fc-8d96-11e7-a412-42010a800002\",\"apiVersion\":\"v1\",\"resourceVersion\":\"11613102\"}}\n","openshift.io/deployment-config.latest-version":"1","openshift.io/deployment-config.name":"router","openshift.io/deployment.name":"router-1","openshift.io/scc":"hostnetwork"},"ownerReferences":[{"apiVersion":"v1","kind":"ReplicationController","name":"router-1","uid":"7d8e58fc-8d96-11e7-a412-42010a800002","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"server-certificate","secret":{"secretName":"router-certs","defaultMode":420}},{"name":"router-token-zfhpp","secret":{"secretName":"router-token-zfhpp","defaultMode":420}}],"containers":[{"name":"router","image":"openshift/origin-haproxy-router:v3.7.0-alpha.1","ports":[{"hostPort":80,"containerPort":80,"protocol":"TCP"},{"hostPort":443,"containerPort":443,"protocol":"TCP"},{"name":"stats","hostPort":1936,"containerPort":1936,"protocol":"TCP"}],"env":[{"name":"DEFAULT_CERTIFICATE_DIR","value":"/etc/pki/tls/private"},{"name":"DEFAULT_CERTIFICATE_PATH","value":"/etc/pki/tls/private/tls.crt"},{"name":"ROUTER_CIPHERS"},{"name":"ROUTER_EXTERNAL_HOST_HOSTNAME"},{"name":"ROUTER_EXTERNAL_HOST_HTTPS_VSERVER"},{"name":"ROUTER_EXTERNAL_HOST_HTTP_VSERVER"},{"name":"ROUTER_EXTERNAL_HOST_INSECURE","value":"false"},{"name":"ROUTER_EXTERNAL_HOST_INTERNAL_ADDRESS"},{"name":"ROUTER_EXTERNAL_HOST_PARTITION_PATH"},{"name":"ROUTER_EXTERNAL_HOST_PASSWORD"},{"name":"ROUTER_EXTERNAL_HOST_PRIVKEY","value":"/etc/secret-volume/router.pem"},{"name":"ROUTER_EXTERNAL_HOST_USERNAME"},{"name":"ROUTER_EXTERNAL_HOST_VXLAN_GW_CIDR"},{"name":"ROUTER_LISTEN_ADDR","value":"0.0.0.0:1936"},{"name":"ROUTER_METRICS_TYPE","value":"haproxy"},{"name":"ROUTER_SERVICE_HTTPS_PORT","value":"443"},{"name":"ROUTER_SERVICE_HTTP_PORT","value":"80"},{"name":"ROUTER_SERVICE_NAME","value":"router"},{"name":"ROUTER_SERVICE_NAMESPACE","value":"default"},{"name":"ROUTER_SUBDOMAIN"},{"name":"STATS_PASSWORD","value":"H4zShpMOPR"},{"name":"STATS_PORT","value":"1936"},{"name":"STATS_USERNAME","value":"admin"}],"resources":{"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"server-certificate","readOnly":true,"mountPath":"/etc/pki/tls/private"},{"name":"router-token-zfhpp","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":1936,"host":"localhost","scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":1936,"host":"localhost","scheme":"HTTP"},"initialDelaySeconds":10,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c5,c0"},"runAsUser":1000020000}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"role":"infra"},"serviceAccountName":"router","serviceAccount":"router","nodeName":"origin-ci-ig-m-11v4","hostNetwork":true,"securityContext":{"seLinuxOptions":{"level":"s0:c5,c0"},"supplementalGroups":[1000020000],"fsGroup":1000020000},"imagePullSecrets":[{"name":"router-dockercfg-v1723"}],"schedulerName":"default-scheduler"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:39Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:59Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2017-11-02T20:30:39Z"}],"hostIP":"10.128.0.2","podIP":"10.128.0.2","startTime":"2017-11-02T20:30:39Z","containerStatuses":[{"name":"router","state":{"running":{"startedAt":"2017-11-02T20:30:41Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/openshift/origin-haproxy-router:v3.7.0-alpha.1","imageID":"docker-pullable://docker.io/openshift/origin-haproxy-router@sha256:9e6c9d202cfdd96c8cdd962724e1c3574e291f942574f2e86569663233e87708","containerID":"docker://81bd2f5d2177245379e883b11c25bf19cbe3b851e3c750611d57ae311ef14f20"}],"qosClass":"Burstable"}}]} I1102 21:09:34.169926 104483 cached_discovery.go:119] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/servergroups.json I1102 21:09:34.170005 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiregistration.k8s.io/v1beta1/serverresources.json I1102 21:09:34.170147 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/extensions/v1beta1/serverresources.json I1102 21:09:34.170228 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps/v1beta1/serverresources.json I1102 21:09:34.170266 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1/serverresources.json I1102 21:09:34.170304 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authentication.k8s.io/v1beta1/serverresources.json I1102 21:09:34.170435 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1/serverresources.json I1102 21:09:34.170489 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.k8s.io/v1beta1/serverresources.json I1102 21:09:34.170547 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/autoscaling/v1/serverresources.json I1102 21:09:34.170597 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v1/serverresources.json I1102 21:09:34.170656 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/batch/v2alpha1/serverresources.json I1102 21:09:34.170720 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/certificates.k8s.io/v1beta1/serverresources.json I1102 21:09:34.170766 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/networking.k8s.io/v1/serverresources.json I1102 21:09:34.170812 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/policy/v1beta1/serverresources.json I1102 21:09:34.170895 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/authorization.openshift.io/v1/serverresources.json I1102 21:09:34.170989 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/rbac.authorization.k8s.io/v1beta1/serverresources.json I1102 21:09:34.171040 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/settings.k8s.io/v1alpha1/serverresources.json I1102 21:09:34.171084 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1/serverresources.json I1102 21:09:34.171136 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/storage.k8s.io/v1beta1/serverresources.json I1102 21:09:34.171285 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apiextensions.k8s.io/v1beta1/serverresources.json I1102 21:09:34.171353 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/apps.openshift.io/v1/serverresources.json I1102 21:09:34.171441 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/build.openshift.io/v1/serverresources.json I1102 21:09:34.171535 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/image.openshift.io/v1/serverresources.json I1102 21:09:34.171594 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/network.openshift.io/v1/serverresources.json I1102 21:09:34.171652 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/oauth.openshift.io/v1/serverresources.json I1102 21:09:34.171711 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/project.openshift.io/v1/serverresources.json I1102 21:09:34.171846 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/quota.openshift.io/v1/serverresources.json I1102 21:09:34.171899 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/route.openshift.io/v1/serverresources.json I1102 21:09:34.171962 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/security.openshift.io/v1/serverresources.json I1102 21:09:34.172031 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/template.openshift.io/v1/serverresources.json I1102 21:09:34.172091 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/user.openshift.io/v1/serverresources.json I1102 21:09:34.172133 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/prow.k8s.io/v1/serverresources.json I1102 21:09:34.172187 104483 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/v1.7.6+a08f5eeb62 (linux/amd64) kubernetes/c84beff" https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 I1102 21:10:04.173788 104483 round_trippers.go:405] GET https://internal-master.ci.openshift.org:8443/apis/metrics/v1alpha1 503 Service Unavailable in 30001 milliseconds I1102 21:10:04.173814 104483 round_trippers.go:411] Response Headers: I1102 21:10:04.173820 104483 round_trippers.go:414] Cache-Control: no-store I1102 21:10:04.173824 104483 round_trippers.go:414] Content-Type: text/plain; charset=utf-8 I1102 21:10:04.173827 104483 round_trippers.go:414] Content-Length: 115 I1102 21:10:04.173831 104483 round_trippers.go:414] Date: Thu, 02 Nov 2017 21:10:04 GMT I1102 21:10:04.173854 104483 request.go:994] Response Body: Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1' I1102 21:10:04.173984 104483 request.go:1196] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'dial tcp 172.30.176.73:443: i/o timeout' Trying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'' I1102 21:10:04.173998 104483 cached_discovery.go:79] skipped caching discovery info due to an error on the server ("Error: 'dial tcp 172.30.176.73:443: i/o timeout'\nTrying to reach: 'https://172.30.176.73:443/apis/metrics/v1alpha1'") has prevented the request from succeeding I1102 21:10:04.174655 104483 cached_discovery.go:72] returning cached discovery info from /root/.kube/internal_master.ci.openshift.org_8443/v1/serverresources.json NAME READY STATUS RESTARTS AGE docker-registry-2-qrd58 0/1 Running 16 42m registry-console-1-lkfh6 1/1 Running 0 20m router-1-wf6r0 1/1 Running 0 42m ```
stevekuznetsov commented 7 years ago

/cc @deads2k @sttts

smarterclayton commented 7 years ago

Was metrics server, which was down because it was on a node with no network connectivity.

sttts commented 7 years ago

Ouch.

sttts commented 7 years ago

Do we know which discovery endpoint blocked? I would guess one of the /group/version/serverresources.json ?

deads2k commented 7 years ago

For kubectl, I think we try to solve the "slow apiserver problem" client-side with dialer timeouts and tls connection timeouts. In the aggregator, I don't think we set an explicit timeout and expect clients to decide when to give up.

Once the last endpoint is removed the status controller will pull that APIService out of rotation, but a dangling, bad endpoint could still wedge.

Do you want the aggregator to be forcing the "how slow can you be" decision for all clients globally or would you expect clients to choose how long they'd like to wait?

deads2k commented 7 years ago

kubectl can't tell the difference between this and a slow API server. I'll see if I can find a timeout on the discovery request code. It'll be smaller than other options.

sttts commented 7 years ago

If the metrics server service has had no running pod, why didn't the connect fail right away?

sttts commented 7 years ago

I.e. either we have a retry loop somewhere or this sounds like a tcp problem.

deads2k commented 7 years ago

If the metrics server service has had no running pod, why didn't the connect fail right away?

Dial trying to route to an IP that didn't exist and some part of the networking stack didn't fail quickly? The pod still shows as existing when the node dies so the endpoint is still listed.

smarterclayton commented 7 years ago

I'd like a discovery bypass flag or something in kubectl that allows you to not rely on discovery (i.e. bypass all the smarts). The smarts are what broke this - kubectl delete apiservice should have worked without needing to get metrics discovery.

liggitt commented 7 years ago

kubectl delete apiservice requires discovery, right? there's no knowledge of which API group or version that is in otherwise

smarterclayton commented 7 years ago

But if you can discover an API group that has apiservice, i shouldn't have to block on discovering all other groups.

smarterclayton commented 7 years ago

Discovery needs to either be AP or CP, I think it needs to be AP because otherwise kuebctl dies. Or it has a --skip-discovery or --best-effort-discovery which lets you opt into AP

liggitt commented 7 years ago

But if you can discover an API group that has apiservice, i shouldn't have to block on discovering all other groups.

definitely agree with that

sttts commented 7 years ago

@liggitt didn't you have a lazy download PR for discovery?

liggitt commented 7 years ago

@liggitt didn't you have a lazy download PR for discovery?

you're thinking of https://github.com/kubernetes/kubernetes/pull/53303 which was for specific run/create commands, not commands where we rely on the full discovery tree for resource resolution

stevekuznetsov commented 7 years ago

@sttts @deads2k see full client logs in this comment: https://github.com/openshift/origin/issues/17159#issuecomment-341559718 (click to expand)

sttts commented 7 years ago

@stevekuznetsov thanks for the logs. Next to @smarterclayton suggestion to have a ignore-discovery option, I still think https://github.com/openshift/origin/issues/17159#issuecomment-341716620 is the main issue: why didn't the Dial fail quickly? Do we have a test cluster to check what happens to a tcp connection attempt from the master to a just failed pod IP?

smarterclayton commented 7 years ago

Dial failed because packets from the aggregator to that pod were being blackholed - in that case only dial timeout matters until the OS gives up.

On Fri, Nov 3, 2017 at 2:32 PM, Dr. Stefan Schimanski < notifications@github.com> wrote:

@stevekuznetsov https://github.com/stevekuznetsov thanks for the logs. Next to @smarterclayton https://github.com/smarterclayton suggestion to have a ignore-discovery option, I still think #17159 (comment) https://github.com/openshift/origin/issues/17159#issuecomment-341716620 is the main issue: why didn't the Dial fail quickly? Do we have a test cluster to check what happens to a tcp connection attempt from the master to a just failed pod IP?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openshift/origin/issues/17159#issuecomment-341790191, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p5sbPLigEtF4aEqgJSJ9Tm_OlgWuks5sy1wzgaJpZM4QQWEx .

smarterclayton commented 6 years ago

Did we mostly fix this?

sttts commented 6 years ago

Looks like we are hitting this again, maybe in a slightly different form on free-int: we get 429s and kubectl retries 10 times. Digging...

sttts commented 6 years ago

Verified: service catalog was returning 429 for unknown reason (saw some 1m timeout, but not more otherwise). Removing the APIService helped. /cc @mwoodson

sttts commented 6 years ago

/cc @juanvallejo

sttts commented 6 years ago

Related: https://github.com/kubernetes/kubernetes/pull/60434

juanvallejo commented 6 years ago

@sttts will update https://github.com/kubernetes/kubernetes/pull/60434 to reduce the number of retries, if that seems like the better approach

soltysh commented 6 years ago

Alternative approach in https://github.com/kubernetes/kubernetes/pull/62733