kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Multiple TLS handshake errors #278

Closed surajssd closed 4 years ago

surajssd commented 4 years ago

I have a single master-worker cluster:

$ kubectl get nodes
NAME                               STATUS   ROLES    AGE   VERSION
suraj-foobar-controller-0      Ready    <none>   67m   v1.18.0
suraj-foobar-pool-1-worker-0   Ready    <none>   68m   v1.18.0

I see multiple errors in apiserver pod: http: TLS handshake error from 127.0.0.1:40608: read tcp 127.53.210.227:7443->127.0.0.1:40608: read: connection reset by peer

This hasn't caused any issues with proper running of the cluster, but not sure where these errors are coming from?

logs

```console $ kubectl -n kube-system logs kube-apiserver-764dc4d7f9-vrqkr kube-apiserver + cat /run/kube-apiserver/address + exec /usr/local/bin/kube-apiserver --advertise-address=10.88.81.1 --allow-privileged=true --anonymous-auth=false --authorization-mode=RBAC --bind-address=127.53.210.227 --client-ca-file=/etc/kubernetes/secrets/ca.crt --cloud-provider= --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,Priority,PodSecurityPolicy --etcd-cafile=/etc/kubernetes/secrets/etcd-client-ca.crt --etcd-certfile=/etc/kubernetes/secrets/etcd-client.crt --etcd-keyfile=/etc/kubernetes/secrets/etcd-client.key --etcd-servers=https://suraj-foobar-etcd0.dev.example.com.:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/secrets/apiserver.crt --kubelet-client-key=/etc/kubernetes/secrets/apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=7443 --service-account-key-file=/etc/kubernetes/secrets/service-account.pub --service-ip-range=10.3.0.0/16 --tls-cert-file=/etc/kubernetes/secrets/apiserver.crt --tls-private-key-file=/etc/kubernetes/secrets/apiserver.key --proxy-client-cert-file=/etc/kubernetes/secrets/aggregation-client.crt --proxy-client-key-file=/etc/kubernetes/secrets/aggregation-client.key --requestheader-client-ca-file=/etc/kubernetes/secrets/aggregation-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --storage-backend=etcd3 Flag --insecure-port has been deprecated, This flag will be removed in a future version. I0408 05:14:59.206052 1 server.go:656] external host was not specified, using 10.88.81.1 I0408 05:14:59.206436 1 server.go:153] Version: v1.18.0 I0408 05:14:59.664343 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0408 05:14:59.664355 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I0408 05:14:59.664976 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0408 05:14:59.664986 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I0408 05:14:59.666492 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.666524 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.674420 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.674439 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.680169 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.680193 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.706320 1 master.go:270] Using reconciler: lease I0408 05:14:59.706688 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.706702 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.714846 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.714877 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.721095 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.721117 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.727063 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.727105 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.735771 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.735794 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.743129 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.743154 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.751005 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.751029 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.758772 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.758923 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.764425 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.764442 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.771068 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.771097 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.793468 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.793492 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.802422 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.802448 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.812330 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.812366 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.820745 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.820787 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.834320 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.834346 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.843060 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.843078 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.849410 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.849428 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.855339 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.855355 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.861934 1 rest.go:113] the default service ipfamily for this cluster is: IPv4 I0408 05:14:59.979606 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.979633 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:14:59.993879 1 client.go:361] parsed scheme: "endpoint" I0408 05:14:59.993903 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.003891 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.003914 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.010149 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.010172 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.016579 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.016603 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.022730 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.022753 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.030319 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.030338 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.036233 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.036270 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.041587 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.041610 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.047554 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.047576 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.054316 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.054337 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.067551 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.067571 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.074210 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.074235 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.080566 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.080589 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.087517 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.087542 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.098095 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.098126 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.106447 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.106469 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.112795 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.112816 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.119642 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.119661 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.127437 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.127458 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.133260 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.133282 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.140002 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.140018 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.146627 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.146648 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.152492 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.152511 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.162500 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.162521 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.168745 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.168769 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.175849 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.175872 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.184452 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.184579 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.196725 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.196764 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.203385 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.203412 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.208751 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.208774 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.216547 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.216834 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.224729 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.224753 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.231748 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.231766 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.237902 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.237921 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.244766 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.244784 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.251621 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.251645 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.268124 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.268148 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.274773 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.274818 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.280735 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.280760 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.288059 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.288084 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.300944 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.300968 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.310686 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.310709 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.317517 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.317541 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] W0408 05:15:00.418345 1 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources. W0408 05:15:00.427627 1 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources. W0408 05:15:00.438617 1 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources. W0408 05:15:00.455327 1 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. W0408 05:15:00.458553 1 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. W0408 05:15:00.470275 1 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources. W0408 05:15:00.481958 1 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources. W0408 05:15:00.481973 1 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources. I0408 05:15:00.487819 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0408 05:15:00.487832 1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota. I0408 05:15:00.489132 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.489148 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.494806 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.494828 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:00.663581 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:00.663600 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:02.039480 1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/secrets/aggregation-ca.crt I0408 05:15:02.039484 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/secrets/ca.crt I0408 05:15:02.039685 1 dynamic_serving_content.go:130] Starting serving-cert::/etc/kubernetes/secrets/apiserver.crt::/etc/kubernetes/secrets/apiserver.key I0408 05:15:02.040031 1 secure_serving.go:178] Serving securely on 127.53.210.227:7443 I0408 05:15:02.040063 1 controller.go:81] Starting OpenAPI AggregationController I0408 05:15:02.040112 1 apiservice_controller.go:94] Starting APIServiceRegistrationController I0408 05:15:02.040117 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0408 05:15:02.040129 1 available_controller.go:387] Starting AvailableConditionController I0408 05:15:02.040138 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I0408 05:15:02.040196 1 tlsconfig.go:240] Starting DynamicServingCertificateController I0408 05:15:02.040360 1 crd_finalizer.go:266] Starting CRDFinalizer I0408 05:15:02.040381 1 controller.go:86] Starting OpenAPI controller I0408 05:15:02.040402 1 customresource_discovery_controller.go:209] Starting DiscoveryController I0408 05:15:02.040439 1 naming_controller.go:291] Starting NamingConditionController I0408 05:15:02.040455 1 establishing_controller.go:76] Starting EstablishingController I0408 05:15:02.040468 1 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController I0408 05:15:02.040479 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I0408 05:15:02.040365 1 log.go:172] http: TLS handshake error from 127.0.0.1:42445: EOF I0408 05:15:02.040505 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0408 05:15:02.040512 1 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller I0408 05:15:02.040532 1 log.go:172] http: TLS handshake error from 127.0.0.1:43125: EOF I0408 05:15:02.040871 1 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/secrets/ca.crt I0408 05:15:02.040899 1 dynamic_cafile_content.go:167] Starting request-header::/etc/kubernetes/secrets/aggregation-ca.crt I0408 05:15:02.040955 1 autoregister_controller.go:141] Starting autoregister controller I0408 05:15:02.040963 1 cache.go:32] Waiting for caches to sync for autoregister controller I0408 05:15:02.052871 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0408 05:15:02.052882 1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister I0408 05:15:02.052888 1 shared_informer.go:230] Caches are synced for crd-autoregister E0408 05:15:02.054983 1 controller.go:152] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service I0408 05:15:02.140250 1 cache.go:39] Caches are synced for AvailableConditionController controller I0408 05:15:02.140250 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0408 05:15:02.140610 1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller I0408 05:15:02.141067 1 cache.go:39] Caches are synced for autoregister controller I0408 05:15:03.039506 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue). I0408 05:15:03.039549 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue). I0408 05:15:03.042077 1 storage_scheduling.go:143] all system priority classes are created successfully or already exist. I0408 05:15:16.919426 1 log.go:172] http: TLS handshake error from 127.0.0.1:35839: EOF I0408 05:15:17.245009 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io I0408 05:15:17.552637 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.552663 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.585070 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.585110 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.607712 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.607733 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.627336 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.627354 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.648108 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.648176 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.667447 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.667469 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.686675 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.686694 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.705730 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.705765 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.726969 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.726990 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.747730 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.747747 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.767574 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.767591 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:17.787341 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:17.787360 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:19.538750 1 controller.go:606] quota admission added evaluator for: endpoints I0408 05:15:19.640405 1 controller.go:606] quota admission added evaluator for: events.events.k8s.io I0408 05:15:20.989981 1 controller.go:606] quota admission added evaluator for: serviceaccounts I0408 05:15:22.865448 1 log.go:172] http: TLS handshake error from 127.0.0.1:33968: read tcp 127.53.210.227:7443->127.0.0.1:33968: read: connection reset by peer I0408 05:15:32.865296 1 log.go:172] http: TLS handshake error from 127.0.0.1:34058: read tcp 127.53.210.227:7443->127.0.0.1:34058: read: connection reset by peer I0408 05:15:35.866871 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:35.866892 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:35.886328 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:35.886346 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:36.391459 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io I0408 05:15:38.758085 1 client.go:361] parsed scheme: "endpoint" I0408 05:15:38.758102 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:15:42.865067 1 log.go:172] http: TLS handshake error from 127.0.0.1:34152: read tcp 127.53.210.227:7443->127.0.0.1:34152: read: connection reset by peer I0408 05:15:44.378378 1 controller.go:606] quota admission added evaluator for: daemonsets.apps I0408 05:15:44.381159 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps I0408 05:15:44.629420 1 controller.go:606] quota admission added evaluator for: deployments.apps I0408 05:15:44.632763 1 controller.go:606] quota admission added evaluator for: replicasets.apps I0408 05:15:52.865531 1 log.go:172] http: TLS handshake error from 127.0.0.1:34300: read tcp 127.53.210.227:7443->127.0.0.1:34300: read: connection reset by peer I0408 05:16:02.023306 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:02.023323 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:02.865567 1 log.go:172] http: TLS handshake error from 127.0.0.1:34398: read tcp 127.53.210.227:7443->127.0.0.1:34398: read: connection reset by peer I0408 05:16:09.012541 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.012559 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.032189 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.032206 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.054189 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.054211 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.075513 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.075544 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.096471 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.096487 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.132160 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.132182 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.175297 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.175322 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.422166 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.422187 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:09.453577 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:09.453615 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:12.096048 1 controller.go:606] quota admission added evaluator for: blockdevices.openebs.io I0408 05:16:12.865367 1 log.go:172] http: TLS handshake error from 127.0.0.1:34510: read tcp 127.53.210.227:7443->127.0.0.1:34510: read: connection reset by peer I0408 05:16:22.865547 1 log.go:172] http: TLS handshake error from 127.0.0.1:34608: read tcp 127.53.210.227:7443->127.0.0.1:34608: read: connection reset by peer I0408 05:16:24.401327 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.401364 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:24.439324 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.439341 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:24.446652 1 controller.go:606] quota admission added evaluator for: runtasks.openebs.io I0408 05:16:24.648961 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.648984 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:24.915071 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.915087 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:24.938036 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.938060 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:24.971711 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:24.971742 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:25.007766 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:25.007786 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:25.028569 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:25.028587 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:25.065967 1 controller.go:606] quota admission added evaluator for: blockdeviceclaims.openebs.io W0408 05:16:25.346603 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured I0408 05:16:29.894213 1 trace.go:116] Trace[727182442]: "Create" url:/api/v1/namespaces/monitoring/secrets,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:28.56313859 +0000 UTC m=+89.401633123) (total time: 1.331048554s): Trace[727182442]: [1.299427631s] [1.299427631s] About to convert to expected version I0408 05:16:31.946612 1 trace.go:116] Trace[418691452]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:31.337152278 +0000 UTC m=+92.175646807) (total time: 609.436655ms): Trace[418691452]: [577.560042ms] [577.560042ms] About to convert to expected version I0408 05:16:32.452652 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io I0408 05:16:32.865335 1 log.go:172] http: TLS handshake error from 127.0.0.1:34718: read tcp 127.53.210.227:7443->127.0.0.1:34718: read: connection reset by peer I0408 05:16:33.796927 1 trace.go:116] Trace[573073756]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:33.185708391 +0000 UTC m=+94.024202924) (total time: 611.194678ms): Trace[573073756]: [576.401408ms] [576.401408ms] About to convert to expected version I0408 05:16:35.671758 1 trace.go:116] Trace[1349558842]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:35.060976715 +0000 UTC m=+95.899471248) (total time: 610.755623ms): Trace[1349558842]: [563.791049ms] [563.791049ms] About to convert to expected version I0408 05:16:37.514503 1 trace.go:116] Trace[910747285]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:36.911589436 +0000 UTC m=+97.750083968) (total time: 602.889817ms): Trace[910747285]: [570.385613ms] [570.385613ms] About to convert to expected version I0408 05:16:39.664015 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:39.664033 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:39.684174 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:39.684191 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:39.703989 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:39.704007 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:39.723742 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:39.723774 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:39.743394 1 client.go:361] parsed scheme: "endpoint" I0408 05:16:39.743411 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:16:39.810613 1 trace.go:116] Trace[1234884377]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:38.961273827 +0000 UTC m=+99.799768357) (total time: 849.319869ms): Trace[1234884377]: [814.572598ms] [814.572598ms] About to convert to expected version I0408 05:16:41.898551 1 trace.go:116] Trace[1634392470]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:41.031506984 +0000 UTC m=+101.870001515) (total time: 867.020234ms): Trace[1634392470]: [834.79019ms] [834.79019ms] About to convert to expected version I0408 05:16:42.402636 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io I0408 05:16:42.865507 1 log.go:172] http: TLS handshake error from 127.0.0.1:34820: read tcp 127.53.210.227:7443->127.0.0.1:34820: read: connection reset by peer I0408 05:16:43.994706 1 trace.go:116] Trace[1682148197]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:16:43.117623917 +0000 UTC m=+103.956118445) (total time: 877.056138ms): Trace[1682148197]: [830.027877ms] [830.027877ms] About to convert to expected version I0408 05:16:44.481517 1 controller.go:606] quota admission added evaluator for: jobs.batch I0408 05:16:52.865286 1 log.go:172] http: TLS handshake error from 127.0.0.1:34912: read tcp 127.53.210.227:7443->127.0.0.1:34912: read: connection reset by peer I0408 05:17:02.865348 1 log.go:172] http: TLS handshake error from 127.0.0.1:35018: read tcp 127.53.210.227:7443->127.0.0.1:35018: read: connection reset by peer I0408 05:17:07.188902 1 controller.go:606] quota admission added evaluator for: alertmanagers.monitoring.coreos.com I0408 05:17:07.687137 1 controller.go:606] quota admission added evaluator for: prometheuses.monitoring.coreos.com I0408 05:17:12.865415 1 log.go:172] http: TLS handshake error from 127.0.0.1:35196: read tcp 127.53.210.227:7443->127.0.0.1:35196: read: connection reset by peer W0408 05:17:16.633495 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured W0408 05:17:16.663645 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured I0408 05:17:16.683008 1 client.go:361] parsed scheme: "endpoint" I0408 05:17:16.683026 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] W0408 05:17:16.687386 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured W0408 05:17:16.701003 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured I0408 05:17:16.742084 1 client.go:361] parsed scheme: "endpoint" I0408 05:17:16.742101 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:17:16.765184 1 client.go:361] parsed scheme: "endpoint" I0408 05:17:16.765200 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:17:16.787802 1 client.go:361] parsed scheme: "endpoint" I0408 05:17:16.787820 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:17:18.861756 1 controller.go:606] quota admission added evaluator for: statefulsets.apps I0408 05:17:22.865475 1 log.go:172] http: TLS handshake error from 127.0.0.1:35304: read tcp 127.53.210.227:7443->127.0.0.1:35304: read: connection reset by peer I0408 05:17:27.768749 1 client.go:361] parsed scheme: "endpoint" I0408 05:17:27.768767 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:17:32.865459 1 log.go:172] http: TLS handshake error from 127.0.0.1:35410: read tcp 127.53.210.227:7443->127.0.0.1:35410: read: connection reset by peer I0408 05:17:37.934479 1 trace.go:116] Trace[802231174]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:2322bfd8-07d9-4482-903e-41d51361e857 (started: 2020-04-08 05:17:07.9341333 +0000 UTC m=+128.772627843) (total time: 30.000278106s): Trace[802231174]: [30.000278106s] [30.000278106s] END W0408 05:17:37.934564 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded E0408 05:17:37.934579 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.934678 1 trace.go:116] Trace[1035214315]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:74376417-d9c2-4db5-bdf5-c045b19e5ca3 (started: 2020-04-08 05:17:07.934537808 +0000 UTC m=+128.773032338) (total time: 30.000113493s): Trace[1035214315]: [30.000113493s] [30.000113493s] END W0408 05:17:37.934707 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded E0408 05:17:37.934721 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.935021 1 controller.go:606] quota admission added evaluator for: prometheusrules.monitoring.coreos.com I0408 05:17:37.935105 1 trace.go:116] Trace[84699097]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:94c48810-7ced-456c-949f-b4039231d4d5 (started: 2020-04-08 05:17:07.934894984 +0000 UTC m=+128.773389517) (total time: 30.000185131s): Trace[84699097]: [30.000185131s] [30.000185131s] END W0408 05:17:37.935131 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.935152 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.939444 1 trace.go:116] Trace[1586755820]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:2ff3a188-eea5-4e67-af6b-1bbe947376d0 (started: 2020-04-08 05:17:07.938992155 +0000 UTC m=+128.777486688) (total time: 30.000424006s): Trace[1586755820]: [30.000424006s] [30.000424006s] END W0408 05:17:37.939711 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout E0408 05:17:37.939731 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout I0408 05:17:37.940744 1 trace.go:116] Trace[1081119297]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:f0652134-780f-4a50-a917-64ccd3c5c538 (started: 2020-04-08 05:17:07.940473996 +0000 UTC m=+128.778968526) (total time: 30.000245086s): Trace[1081119297]: [30.000245086s] [30.000245086s] END W0408 05:17:37.940775 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout E0408 05:17:37.940792 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout I0408 05:17:37.941523 1 trace.go:116] Trace[320375726]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:458f88d0-b602-469c-8a91-820bf7354b2d (started: 2020-04-08 05:17:07.941318327 +0000 UTC m=+128.779812858) (total time: 30.000180644s): Trace[320375726]: [30.000180644s] [30.000180644s] END W0408 05:17:37.941652 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout E0408 05:17:37.941741 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout I0408 05:17:37.941887 1 trace.go:116] Trace[798926643]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:edf94123-caeb-4652-b7a9-fe670bb79e9a (started: 2020-04-08 05:17:07.938992216 +0000 UTC m=+128.777486747) (total time: 30.002871601s): Trace[798926643]: [30.002871601s] [30.002871601s] END W0408 05:17:37.942017 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded E0408 05:17:37.942191 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.942376 1 trace.go:116] Trace[645368282]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:32938834-ca96-4f2b-96a1-8ae9e3290740 (started: 2020-04-08 05:17:07.939924369 +0000 UTC m=+128.778418903) (total time: 30.002403513s): Trace[645368282]: [30.002403513s] [30.002403513s] END W0408 05:17:37.942465 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded E0408 05:17:37.942581 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.942663 1 trace.go:116] Trace[542877932]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:7af38562-44b1-4825-90f0-1a9f579a4381 (started: 2020-04-08 05:17:07.942335095 +0000 UTC m=+128.780829625) (total time: 30.000307976s): Trace[542877932]: [30.000307976s] [30.000307976s] END I0408 05:17:37.942793 1 trace.go:116] Trace[1347248626]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:f3cc295b-2909-4771-8b0d-eb0a7a7c6740 (started: 2020-04-08 05:17:07.940454996 +0000 UTC m=+128.778949525) (total time: 30.002313627s): Trace[1347248626]: [30.002313627s] [30.002313627s] END W0408 05:17:37.942887 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded W0408 05:17:37.942718 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout E0408 05:17:37.943052 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout I0408 05:17:37.941771 1 trace.go:116] Trace[1077898496]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:73306c4e-d037-45ee-8e28-304b9a7a7c82 (started: 2020-04-08 05:17:07.941222762 +0000 UTC m=+128.779717292) (total time: 30.000507108s): Trace[1077898496]: [30.000507108s] [30.000507108s] END W0408 05:17:37.943193 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.943214 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.943154 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.943559 1 trace.go:116] Trace[746335237]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:baf6d144-4d5a-4d91-9a5b-24ab93f941df (started: 2020-04-08 05:17:07.94132703 +0000 UTC m=+128.779821560) (total time: 30.002203806s): Trace[746335237]: [30.002203806s] [30.002203806s] END W0408 05:17:37.943639 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout E0408 05:17:37.943846 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: dial tcp 10.3.255.139:443: i/o timeout I0408 05:17:37.945023 1 trace.go:116] Trace[1424602815]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:f4c53b7a-39fe-44f8-922a-f806b902c09f (started: 2020-04-08 05:17:07.944638471 +0000 UTC m=+128.783133020) (total time: 30.000354736s): Trace[1424602815]: [30.000354736s] [30.000354736s] END W0408 05:17:37.945049 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded E0408 05:17:37.945074 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: context deadline exceeded I0408 05:17:37.945329 1 trace.go:116] Trace[991191818]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:5a551209-8a6e-413c-a50b-9aab9099e2df (started: 2020-04-08 05:17:07.944787059 +0000 UTC m=+128.783281588) (total time: 30.000513891s): Trace[991191818]: [30.000513891s] [30.000513891s] END W0408 05:17:37.945359 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.945390 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.945663 1 trace.go:116] Trace[71412614]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:e39f072b-34a7-41c8-8827-998c30207c48 (started: 2020-04-08 05:17:07.944812285 +0000 UTC m=+128.783306824) (total time: 30.000824672s): Trace[71412614]: [30.000824672s] [30.000824672s] END W0408 05:17:37.945711 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.945911 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.946117 1 trace.go:116] Trace[756352742]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:c89e3597-9478-4a9d-a0a7-db11be14fa93 (started: 2020-04-08 05:17:07.944801268 +0000 UTC m=+128.783295804) (total time: 30.001295831s): Trace[756352742]: [30.001295831s] [30.001295831s] END W0408 05:17:37.946147 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.946262 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.946469 1 trace.go:116] Trace[162667212]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:a711519f-9a9b-4022-88ae-e9bcb9ace55f (started: 2020-04-08 05:17:07.945137536 +0000 UTC m=+128.783632066) (total time: 30.001305772s): Trace[162667212]: [30.001305772s] [30.001305772s] END I0408 05:17:37.946665 1 trace.go:116] Trace[1574412488]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938628349 +0000 UTC m=+128.777122880) (total time: 30.008005932s): Trace[1574412488]: [30.007660527s] [30.007452644s] Object stored in database I0408 05:17:37.947188 1 trace.go:116] Trace[530881967]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:e6a08456-6129-422c-bc04-cb9dafee29d6 (started: 2020-04-08 05:17:07.946426165 +0000 UTC m=+128.784920696) (total time: 30.000735429s): Trace[530881967]: [30.000735429s] [30.000735429s] END W0408 05:17:37.947237 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.947346 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.947574 1 trace.go:116] Trace[1727127758]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:29d2e6c6-7616-4c11-bc5e-a7ef9810ec37 (started: 2020-04-08 05:17:07.945996107 +0000 UTC m=+128.784490642) (total time: 30.001552607s): Trace[1727127758]: [30.001552607s] [30.001552607s] END W0408 05:17:37.947657 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.947726 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.948259 1 trace.go:116] Trace[944346619]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:44f489d2-1685-47d7-be85-141a44a985a7 (started: 2020-04-08 05:17:07.946109854 +0000 UTC m=+128.784604390) (total time: 30.002117028s): Trace[944346619]: [30.002117028s] [30.002117028s] END W0408 05:17:37.948286 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.948413 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.948260 1 trace.go:116] Trace[1586294310]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:9171e56c-4602-4b39-b1a8-f4ec0983a1f4 (started: 2020-04-08 05:17:07.946015648 +0000 UTC m=+128.784510177) (total time: 30.002213709s): Trace[1586294310]: [30.002213709s] [30.002213709s] END W0408 05:17:37.948434 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.948528 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.948871 1 trace.go:116] Trace[1575537303]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:51f609b6-6bf9-4ded-beea-30b31dcca1cb (started: 2020-04-08 05:17:07.946127247 +0000 UTC m=+128.784621786) (total time: 30.002719917s): Trace[1575537303]: [30.002719917s] [30.002719917s] END I0408 05:17:37.949086 1 trace.go:116] Trace[1148103056]: "Call mutating webhook" configuration:prometheus-operator-admission,webhook:prometheusrulemutate.monitoring.coreos.com,resource:monitoring.coreos.com/v1, Resource=prometheusrules,subresource:,operation:CREATE,UID:0c765de4-44bf-47c0-94aa-6dbb3a6d9758 (started: 2020-04-08 05:17:07.946200101 +0000 UTC m=+128.784694630) (total time: 30.002862433s): Trace[1148103056]: [30.002862433s] [30.002862433s] END W0408 05:17:37.949107 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.949121 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) W0408 05:17:37.948960 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.949308 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.949542 1 trace.go:116] Trace[832598906]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.932408412 +0000 UTC m=+128.770902945) (total time: 30.017019037s): Trace[832598906]: [30.016689358s] [30.015201824s] Object stored in database W0408 05:17:37.946532 1 dispatcher.go:169] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) E0408 05:17:37.949921 1 dispatcher.go:170] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": Post https://prometheus-operator-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:17:37.950860 1 trace.go:116] Trace[1407841807]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.934572032 +0000 UTC m=+128.773066562) (total time: 30.016258901s): Trace[1407841807]: [30.015847963s] [30.015684664s] Object stored in database I0408 05:17:37.953652 1 trace.go:116] Trace[436216336]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938079136 +0000 UTC m=+128.776573671) (total time: 30.015547566s): Trace[436216336]: [30.014933846s] [30.01268799s] Object stored in database I0408 05:17:37.953832 1 trace.go:116] Trace[1918859283]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.934074204 +0000 UTC m=+128.772568732) (total time: 30.019730196s): Trace[1918859283]: [30.018713815s] [30.018411428s] Object stored in database I0408 05:17:37.954489 1 trace.go:116] Trace[286654168]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938069286 +0000 UTC m=+128.776563814) (total time: 30.016283133s): Trace[286654168]: [30.015782405s] [30.01352723s] Object stored in database I0408 05:17:37.959150 1 trace.go:116] Trace[1096468044]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938078949 +0000 UTC m=+128.776573481) (total time: 30.021036247s): Trace[1096468044]: [30.019961453s] [30.016893295s] Object stored in database I0408 05:17:37.967920 1 trace.go:116] Trace[457087988]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938093161 +0000 UTC m=+128.776587695) (total time: 30.029797604s): Trace[457087988]: [30.023011993s] [30.022316954s] Object stored in database I0408 05:17:37.970297 1 trace.go:116] Trace[231206081]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.944882253 +0000 UTC m=+128.783376790) (total time: 30.025379712s): Trace[231206081]: [30.025090577s] [30.024134747s] Object stored in database I0408 05:17:37.971632 1 trace.go:116] Trace[702480119]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.938082523 +0000 UTC m=+128.776577062) (total time: 30.033518241s): Trace[702480119]: [30.033151877s] [30.031441485s] Object stored in database I0408 05:17:37.975817 1 trace.go:116] Trace[1534260385]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.940896876 +0000 UTC m=+128.779391410) (total time: 30.034889216s): Trace[1534260385]: [30.034735142s] [30.034559165s] Object stored in database I0408 05:17:37.976745 1 trace.go:116] Trace[1232085435]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.944472476 +0000 UTC m=+128.782967009) (total time: 30.032251914s): Trace[1232085435]: [30.032132905s] [30.031978364s] Object stored in database I0408 05:17:37.976852 1 trace.go:116] Trace[400386114]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.944348658 +0000 UTC m=+128.782843189) (total time: 30.032482344s): Trace[400386114]: [30.032323324s] [30.032159539s] Object stored in database I0408 05:17:37.977595 1 trace.go:116] Trace[477744363]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.941065116 +0000 UTC m=+128.779559655) (total time: 30.036504699s): Trace[477744363]: [30.036337631s] [30.036197538s] Object stored in database I0408 05:17:37.977738 1 trace.go:116] Trace[906985910]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.944442363 +0000 UTC m=+128.782936893) (total time: 30.033225119s): Trace[906985910]: [30.033037865s] [30.032825241s] Object stored in database I0408 05:17:37.978088 1 trace.go:116] Trace[1819796857]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.941080464 +0000 UTC m=+128.779574998) (total time: 30.036988193s): Trace[1819796857]: [30.036027546s] [30.034924957s] Object stored in database I0408 05:17:37.980455 1 trace.go:116] Trace[241319564]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.941938147 +0000 UTC m=+128.780432683) (total time: 30.038470274s): Trace[241319564]: [30.036628791s] [30.033949929s] Object stored in database I0408 05:17:37.982271 1 trace.go:116] Trace[1959864866]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.945625074 +0000 UTC m=+128.784119603) (total time: 30.03660212s): Trace[1959864866]: [30.036504205s] [30.036199654s] Object stored in database I0408 05:17:37.983457 1 trace.go:116] Trace[1489326673]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.945857343 +0000 UTC m=+128.784351880) (total time: 30.03757505s): Trace[1489326673]: [30.036374517s] [30.035944006s] Object stored in database I0408 05:17:37.983914 1 trace.go:116] Trace[242155925]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.945694051 +0000 UTC m=+128.784188581) (total time: 30.038202587s): Trace[242155925]: [30.038090714s] [30.037920376s] Object stored in database I0408 05:17:37.984191 1 trace.go:116] Trace[43912923]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.945780102 +0000 UTC m=+128.784274636) (total time: 30.038397657s): Trace[43912923]: [30.038320462s] [30.038128712s] Object stored in database I0408 05:17:37.984317 1 trace.go:116] Trace[601028256]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.945821894 +0000 UTC m=+128.784316431) (total time: 30.038480891s): Trace[601028256]: [30.038347614s] [30.038105039s] Object stored in database I0408 05:17:37.985621 1 trace.go:116] Trace[977006847]: "Create" url:/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules,user-agent:Go-http-client/2.0,client:127.0.0.1 (started: 2020-04-08 05:17:07.944510974 +0000 UTC m=+128.783005507) (total time: 30.041093466s): Trace[977006847]: [30.040968301s] [30.04049782s] Object stored in database W0408 05:17:37.985689 1 cacher.go:166] Terminating all watchers from cacher *unstructured.Unstructured I0408 05:17:38.697960 1 controller.go:606] quota admission added evaluator for: servicemonitors.monitoring.coreos.com I0408 05:17:42.865385 1 log.go:172] http: TLS handshake error from 127.0.0.1:35502: read tcp 127.53.210.227:7443->127.0.0.1:35502: read: connection reset by peer I0408 05:17:48.143656 1 trace.go:116] Trace[341883503]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:46.503779219 +0000 UTC m=+167.342273748) (total time: 1.639853389s): Trace[341883503]: [1.604406858s] [1.604406858s] About to convert to expected version I0408 05:17:51.084630 1 trace.go:116] Trace[1685880709]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:50.117741473 +0000 UTC m=+170.956236000) (total time: 966.862797ms): Trace[1685880709]: [931.326591ms] [931.326591ms] About to convert to expected version I0408 05:17:52.865299 1 log.go:172] http: TLS handshake error from 127.0.0.1:35594: read tcp 127.53.210.227:7443->127.0.0.1:35594: read: connection reset by peer I0408 05:17:53.143972 1 trace.go:116] Trace[2104050981]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:52.299159574 +0000 UTC m=+173.137654102) (total time: 844.790715ms): Trace[2104050981]: [808.155849ms] [808.155849ms] About to convert to expected version I0408 05:17:54.974123 1 trace.go:116] Trace[1505289044]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:54.345349946 +0000 UTC m=+175.183844476) (total time: 628.749476ms): Trace[1505289044]: [592.012664ms] [592.012664ms] About to convert to expected version I0408 05:17:56.755353 1 trace.go:116] Trace[313577719]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:56.160236293 +0000 UTC m=+176.998730823) (total time: 595.094039ms): Trace[313577719]: [558.063812ms] [558.063812ms] About to convert to expected version I0408 05:17:58.810044 1 trace.go:116] Trace[1075408235]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:17:57.98681665 +0000 UTC m=+178.825311178) (total time: 823.208731ms): Trace[1075408235]: [782.850055ms] [782.850055ms] About to convert to expected version I0408 05:18:00.687697 1 trace.go:116] Trace[1924627199]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:18:00.011711363 +0000 UTC m=+180.850205892) (total time: 675.962025ms): Trace[1924627199]: [642.066177ms] [642.066177ms] About to convert to expected version I0408 05:18:02.865420 1 log.go:172] http: TLS handshake error from 127.0.0.1:35700: read tcp 127.53.210.227:7443->127.0.0.1:35700: read: connection reset by peer I0408 05:18:03.957512 1 controller.go:606] quota admission added evaluator for: cstorvolumes.openebs.io I0408 05:18:03.977294 1 controller.go:606] quota admission added evaluator for: cstorvolumereplicas.openebs.io I0408 05:18:05.951176 1 trace.go:116] Trace[341259801]: "Update" url:/api/v1/namespaces/monitoring/secrets/sh.helm.release.v1.prometheus-operator.v1,user-agent:lokoctl/v0.0.0 (linux/amd64) kubernetes/$Format,client:127.0.0.1 (started: 2020-04-08 05:18:04.612518274 +0000 UTC m=+185.451012802) (total time: 1.33861128s): Trace[341259801]: [1.302855359s] [1.302855359s] About to convert to expected version I0408 05:18:11.320815 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:11.320832 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:11.351449 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:11.351466 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:11.384998 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:11.385027 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:11.413703 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:11.413720 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:12.865290 1 log.go:172] http: TLS handshake error from 127.0.0.1:35806: read tcp 127.53.210.227:7443->127.0.0.1:35806: read: connection reset by peer I0408 05:18:22.865161 1 log.go:172] http: TLS handshake error from 127.0.0.1:35910: read tcp 127.53.210.227:7443->127.0.0.1:35910: read: connection reset by peer I0408 05:18:32.865410 1 log.go:172] http: TLS handshake error from 127.0.0.1:36048: read tcp 127.53.210.227:7443->127.0.0.1:36048: read: connection reset by peer I0408 05:18:42.171653 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:42.171671 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:42.223985 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:42.224005 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:42.246027 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:42.246045 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:42.277211 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:42.277228 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:42.299407 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:42.299423 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:42.865413 1 log.go:172] http: TLS handshake error from 127.0.0.1:36222: read tcp 127.53.210.227:7443->127.0.0.1:36222: read: connection reset by peer I0408 05:18:44.083568 1 client.go:361] parsed scheme: "endpoint" I0408 05:18:44.083600 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://suraj-foobar-etcd0.dev.example.com.:2379 0 }] I0408 05:18:52.865259 1 log.go:172] http: TLS handshake error from 127.0.0.1:36320: read tcp 127.53.210.227:7443->127.0.0.1:36320: read: connection reset by peer I0408 05:19:02.865465 1 log.go:172] http: TLS handshake error from 127.0.0.1:36452: read tcp 127.53.210.227:7443->127.0.0.1:36452: read: connection reset by peer I0408 05:19:12.865101 1 log.go:172] http: TLS handshake error from 127.0.0.1:36554: read tcp 127.53.210.227:7443->127.0.0.1:36554: read: connection reset by peer I0408 05:19:22.865450 1 log.go:172] http: TLS handshake error from 127.0.0.1:36646: read tcp 127.53.210.227:7443->127.0.0.1:36646: read: connection reset by peer I0408 05:19:31.207421 1 controller.go:606] quota admission added evaluator for: ingresses.extensions I0408 05:19:31.223470 1 controller.go:606] quota admission added evaluator for: certificates.cert-manager.io I0408 05:19:31.439533 1 controller.go:606] quota admission added evaluator for: certificaterequests.cert-manager.io I0408 05:19:32.865459 1 log.go:172] http: TLS handshake error from 127.0.0.1:36756: read tcp 127.53.210.227:7443->127.0.0.1:36756: read: connection reset by peer I0408 05:19:35.024398 1 trace.go:116] Trace[402101241]: "Call mutating webhook" configuration:cert-manager-webhook,webhook:webhook.cert-manager.io,resource:cert-manager.io/v1alpha2, Resource=clusterissuers,subresource:status,operation:UPDATE,UID:e8d2ba9a-fe55-45e7-9613-3269040a08a1 (started: 2020-04-08 05:19:05.024106823 +0000 UTC m=+245.862601385) (total time: 30.000256277s): Trace[402101241]: [30.000256277s] [30.000256277s] END W0408 05:19:35.024460 1 dispatcher.go:181] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0408 05:19:35.032929 1 trace.go:116] Trace[117035085]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-04-08 05:19:05.023760575 +0000 UTC m=+245.862255105) (total time: 30.009153249s): Trace[117035085]: [30.008290038s] [30.008208965s] Transaction prepared I0408 05:19:35.033086 1 trace.go:116] Trace[2032282136]: "Update" url:/apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-staging/status,user-agent:controller/v0.0.0 (linux/amd64) kubernetes/$Format/jetstack-cert-manager/v0.12.0,client:127.0.0.1 (started: 2020-04-08 05:19:05.023394011 +0000 UTC m=+245.861888545) (total time: 30.009675483s): Trace[2032282136]: [30.009554945s] [30.009231471s] Object stored in database I0408 05:19:35.225447 1 trace.go:116] Trace[425051311]: "Call mutating webhook" configuration:cert-manager-webhook,webhook:webhook.cert-manager.io,resource:cert-manager.io/v1alpha2, Resource=clusterissuers,subresource:status,operation:UPDATE,UID:21d8086f-bee7-4eb3-95bc-13f51b9aa19a (started: 2020-04-08 05:19:05.225296479 +0000 UTC m=+246.063791017) (total time: 30.000130577s): Trace[425051311]: [30.000130577s] [30.000130577s] END W0408 05:19:35.225476 1 dispatcher.go:181] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: context deadline exceeded I0408 05:19:35.230493 1 trace.go:116] Trace[1970496907]: "GuaranteedUpdate etcd3" type:*unstructured.Unstructured (started: 2020-04-08 05:19:05.225016344 +0000 UTC m=+246.063510874) (total time: 30.005464227s): Trace[1970496907]: [30.004782874s] [30.004737443s] Transaction prepared I0408 05:19:35.230601 1 trace.go:116] Trace[1082075287]: "Update" url:/apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-production/status,user-agent:controller/v0.0.0 (linux/amd64) kubernetes/$Format/jetstack-cert-manager/v0.12.0,client:127.0.0.1 (started: 2020-04-08 05:19:05.22477214 +0000 UTC m=+246.063266669) (total time: 30.005816203s): Trace[1082075287]: [30.00573547s] [30.00552368s] Object stored in database I0408 05:19:35.236789 1 controller.go:606] quota admission added evaluator for: orders.acme.cert-manager.io I0408 05:19:36.267214 1 controller.go:606] quota admission added evaluator for: challenges.acme.cert-manager.io I0408 05:19:42.865439 1 log.go:172] http: TLS handshake error from 127.0.0.1:36852: read tcp 127.53.210.227:7443->127.0.0.1:36852: read: connection reset by peer I0408 05:19:52.865464 1 log.go:172] http: TLS handshake error from 127.0.0.1:36944: read tcp 127.53.210.227:7443->127.0.0.1:36944: read: connection reset by peer I0408 05:20:02.865487 1 log.go:172] http: TLS handshake error from 127.0.0.1:37036: read tcp 127.53.210.227:7443->127.0.0.1:37036: read: connection reset by peer I0408 05:20:12.865310 1 log.go:172] http: TLS handshake error from 127.0.0.1:37128: read tcp 127.53.210.227:7443->127.0.0.1:37128: read: connection reset by peer I0408 05:20:22.865542 1 log.go:172] http: TLS handshake error from 127.0.0.1:37224: read tcp 127.53.210.227:7443->127.0.0.1:37224: read: connection reset by peer I0408 05:20:32.865622 1 log.go:172] http: TLS handshake error from 127.0.0.1:37316: read tcp 127.53.210.227:7443->127.0.0.1:37316: read: connection reset by peer I0408 05:20:42.865461 1 log.go:172] http: TLS handshake error from 127.0.0.1:37408: read tcp 127.53.210.227:7443->127.0.0.1:37408: read: connection reset by peer I0408 05:20:52.865352 1 log.go:172] http: TLS handshake error from 127.0.0.1:37500: read tcp 127.53.210.227:7443->127.0.0.1:37500: read: connection reset by peer I0408 05:21:02.866893 1 log.go:172] http: TLS handshake error from 127.0.0.1:37592: read tcp 127.53.210.227:7443->127.0.0.1:37592: read: connection reset by peer I0408 05:21:12.865419 1 log.go:172] http: TLS handshake error from 127.0.0.1:37684: read tcp 127.53.210.227:7443->127.0.0.1:37684: read: connection reset by peer I0408 05:21:22.865234 1 log.go:172] http: TLS handshake error from 127.0.0.1:37776: read tcp 127.53.210.227:7443->127.0.0.1:37776: read: connection reset by peer I0408 05:21:32.865537 1 log.go:172] http: TLS handshake error from 127.0.0.1:37868: read tcp 127.53.210.227:7443->127.0.0.1:37868: read: connection reset by peer I0408 05:21:42.865288 1 log.go:172] http: TLS handshake error from 127.0.0.1:37960: read tcp 127.53.210.227:7443->127.0.0.1:37960: read: connection reset by peer I0408 05:21:52.865330 1 log.go:172] http: TLS handshake error from 127.0.0.1:38052: read tcp 127.53.210.227:7443->127.0.0.1:38052: read: connection reset by peer I0408 05:22:02.865394 1 log.go:172] http: TLS handshake error from 127.0.0.1:38144: read tcp 127.53.210.227:7443->127.0.0.1:38144: read: connection reset by peer I0408 05:22:12.865381 1 log.go:172] http: TLS handshake error from 127.0.0.1:38236: read tcp 127.53.210.227:7443->127.0.0.1:38236: read: connection reset by peer I0408 05:22:22.865307 1 log.go:172] http: TLS handshake error from 127.0.0.1:38328: read tcp 127.53.210.227:7443->127.0.0.1:38328: read: connection reset by peer I0408 05:22:32.865369 1 log.go:172] http: TLS handshake error from 127.0.0.1:38420: read tcp 127.53.210.227:7443->127.0.0.1:38420: read: connection reset by peer I0408 05:22:42.865392 1 log.go:172] http: TLS handshake error from 127.0.0.1:38512: read tcp 127.53.210.227:7443->127.0.0.1:38512: read: connection reset by peer I0408 05:22:52.865470 1 log.go:172] http: TLS handshake error from 127.0.0.1:38604: read tcp 127.53.210.227:7443->127.0.0.1:38604: read: connection reset by peer I0408 05:23:02.865211 1 log.go:172] http: TLS handshake error from 127.0.0.1:38696: read tcp 127.53.210.227:7443->127.0.0.1:38696: read: connection reset by peer I0408 05:23:12.865354 1 log.go:172] http: TLS handshake error from 127.0.0.1:38788: read tcp 127.53.210.227:7443->127.0.0.1:38788: read: connection reset by peer I0408 05:23:22.865185 1 log.go:172] http: TLS handshake error from 127.0.0.1:38880: read tcp 127.53.210.227:7443->127.0.0.1:38880: read: connection reset by peer I0408 05:23:32.865453 1 log.go:172] http: TLS handshake error from 127.0.0.1:38972: read tcp 127.53.210.227:7443->127.0.0.1:38972: read: connection reset by peer I0408 05:23:42.865295 1 log.go:172] http: TLS handshake error from 127.0.0.1:39064: read tcp 127.53.210.227:7443->127.0.0.1:39064: read: connection reset by peer I0408 05:23:52.865497 1 log.go:172] http: TLS handshake error from 127.0.0.1:39160: read tcp 127.53.210.227:7443->127.0.0.1:39160: read: connection reset by peer I0408 05:24:02.865475 1 log.go:172] http: TLS handshake error from 127.0.0.1:39252: read tcp 127.53.210.227:7443->127.0.0.1:39252: read: connection reset by peer I0408 05:24:12.865331 1 log.go:172] http: TLS handshake error from 127.0.0.1:39344: read tcp 127.53.210.227:7443->127.0.0.1:39344: read: connection reset by peer I0408 05:24:22.865465 1 log.go:172] http: TLS handshake error from 127.0.0.1:39436: read tcp 127.53.210.227:7443->127.0.0.1:39436: read: connection reset by peer I0408 05:24:32.865398 1 log.go:172] http: TLS handshake error from 127.0.0.1:39528: read tcp 127.53.210.227:7443->127.0.0.1:39528: read: connection reset by peer I0408 05:24:42.865538 1 log.go:172] http: TLS handshake error from 127.0.0.1:39620: read tcp 127.53.210.227:7443->127.0.0.1:39620: read: connection reset by peer I0408 05:24:52.865575 1 log.go:172] http: TLS handshake error from 127.0.0.1:39712: read tcp 127.53.210.227:7443->127.0.0.1:39712: read: connection reset by peer I0408 05:25:02.865498 1 log.go:172] http: TLS handshake error from 127.0.0.1:39804: read tcp 127.53.210.227:7443->127.0.0.1:39804: read: connection reset by peer I0408 05:25:12.865435 1 log.go:172] http: TLS handshake error from 127.0.0.1:39896: read tcp 127.53.210.227:7443->127.0.0.1:39896: read: connection reset by peer I0408 05:25:22.865564 1 log.go:172] http: TLS handshake error from 127.0.0.1:39988: read tcp 127.53.210.227:7443->127.0.0.1:39988: read: connection reset by peer I0408 05:25:32.865367 1 log.go:172] http: TLS handshake error from 127.0.0.1:40080: read tcp 127.53.210.227:7443->127.0.0.1:40080: read: connection reset by peer I0408 05:25:42.865418 1 log.go:172] http: TLS handshake error from 127.0.0.1:40172: read tcp 127.53.210.227:7443->127.0.0.1:40172: read: connection reset by peer I0408 05:25:52.865513 1 log.go:172] http: TLS handshake error from 127.0.0.1:40264: read tcp 127.53.210.227:7443->127.0.0.1:40264: read: connection reset by peer I0408 05:26:02.865355 1 log.go:172] http: TLS handshake error from 127.0.0.1:40356: read tcp 127.53.210.227:7443->127.0.0.1:40356: read: connection reset by peer I0408 05:26:12.865227 1 log.go:172] http: TLS handshake error from 127.0.0.1:40448: read tcp 127.53.210.227:7443->127.0.0.1:40448: read: connection reset by peer I0408 05:26:22.865330 1 log.go:172] http: TLS handshake error from 127.0.0.1:40540: read tcp 127.53.210.227:7443->127.0.0.1:40540: read: connection reset by peer I0408 05:26:32.865540 1 log.go:172] http: TLS handshake error from 127.0.0.1:40632: read tcp 127.53.210.227:7443->127.0.0.1:40632: read: connection reset by peer I0408 05:26:42.865542 1 log.go:172] http: TLS handshake error from 127.0.0.1:40724: read tcp 127.53.210.227:7443->127.0.0.1:40724: read: connection reset by peer I0408 05:26:52.865571 1 log.go:172] http: TLS handshake error from 127.0.0.1:40816: read tcp 127.53.210.227:7443->127.0.0.1:40816: read: connection reset by peer I0408 05:27:02.865462 1 log.go:172] http: TLS handshake error from 127.0.0.1:40908: read tcp 127.53.210.227:7443->127.0.0.1:40908: read: connection reset by peer I0408 05:27:12.865235 1 log.go:172] http: TLS handshake error from 127.0.0.1:41000: read tcp 127.53.210.227:7443->127.0.0.1:41000: read: connection reset by peer I0408 05:27:22.865286 1 log.go:172] http: TLS handshake error from 127.0.0.1:41092: read tcp 127.53.210.227:7443->127.0.0.1:41092: read: connection reset by peer I0408 05:27:32.865536 1 log.go:172] http: TLS handshake error from 127.0.0.1:41184: read tcp 127.53.210.227:7443->127.0.0.1:41184: read: connection reset by peer I0408 05:27:42.865347 1 log.go:172] http: TLS handshake error from 127.0.0.1:41276: read tcp 127.53.210.227:7443->127.0.0.1:41276: read: connection reset by peer I0408 05:27:52.865470 1 log.go:172] http: TLS handshake error from 127.0.0.1:41368: read tcp 127.53.210.227:7443->127.0.0.1:41368: read: connection reset by peer I0408 05:28:02.865547 1 log.go:172] http: TLS handshake error from 127.0.0.1:41460: read tcp 127.53.210.227:7443->127.0.0.1:41460: read: connection reset by peer I0408 05:28:12.865322 1 log.go:172] http: TLS handshake error from 127.0.0.1:41552: read tcp 127.53.210.227:7443->127.0.0.1:41552: read: connection reset by peer I0408 05:28:22.865550 1 log.go:172] http: TLS handshake error from 127.0.0.1:41644: read tcp 127.53.210.227:7443->127.0.0.1:41644: read: connection reset by peer I0408 05:28:32.865378 1 log.go:172] http: TLS handshake error from 127.0.0.1:41736: read tcp 127.53.210.227:7443->127.0.0.1:41736: read: connection reset by peer I0408 05:28:42.865469 1 log.go:172] http: TLS handshake error from 127.0.0.1:41828: read tcp 127.53.210.227:7443->127.0.0.1:41828: read: connection reset by peer I0408 05:28:52.865391 1 log.go:172] http: TLS handshake error from 127.0.0.1:41924: read tcp 127.53.210.227:7443->127.0.0.1:41924: read: connection reset by peer I0408 05:29:02.865241 1 log.go:172] http: TLS handshake error from 127.0.0.1:42016: read tcp 127.53.210.227:7443->127.0.0.1:42016: read: connection reset by peer I0408 05:29:12.865404 1 log.go:172] http: TLS handshake error from 127.0.0.1:42108: read tcp 127.53.210.227:7443->127.0.0.1:42108: read: connection reset by peer I0408 05:29:22.865364 1 log.go:172] http: TLS handshake error from 127.0.0.1:42200: read tcp 127.53.210.227:7443->127.0.0.1:42200: read: connection reset by peer I0408 05:29:32.865384 1 log.go:172] http: TLS handshake error from 127.0.0.1:42292: read tcp 127.53.210.227:7443->127.0.0.1:42292: read: connection reset by peer I0408 05:29:42.865458 1 log.go:172] http: TLS handshake error from 127.0.0.1:42384: read tcp 127.53.210.227:7443->127.0.0.1:42384: read: connection reset by peer I0408 05:29:52.865495 1 log.go:172] http: TLS handshake error from 127.0.0.1:42476: read tcp 127.53.210.227:7443->127.0.0.1:42476: read: connection reset by peer I0408 05:30:02.865463 1 log.go:172] http: TLS handshake error from 127.0.0.1:42568: read tcp 127.53.210.227:7443->127.0.0.1:42568: read: connection reset by peer I0408 05:30:12.865372 1 log.go:172] http: TLS handshake error from 127.0.0.1:42660: read tcp 127.53.210.227:7443->127.0.0.1:42660: read: connection reset by peer I0408 05:30:22.865536 1 log.go:172] http: TLS handshake error from 127.0.0.1:42752: read tcp 127.53.210.227:7443->127.0.0.1:42752: read: connection reset by peer I0408 05:30:32.865256 1 log.go:172] http: TLS handshake error from 127.0.0.1:42844: read tcp 127.53.210.227:7443->127.0.0.1:42844: read: connection reset by peer I0408 05:30:42.865218 1 log.go:172] http: TLS handshake error from 127.0.0.1:42936: read tcp 127.53.210.227:7443->127.0.0.1:42936: read: connection reset by peer I0408 05:30:52.865353 1 log.go:172] http: TLS handshake error from 127.0.0.1:43028: read tcp 127.53.210.227:7443->127.0.0.1:43028: read: connection reset by peer I0408 05:31:02.865566 1 log.go:172] http: TLS handshake error from 127.0.0.1:43120: read tcp 127.53.210.227:7443->127.0.0.1:43120: read: connection reset by peer I0408 05:31:12.865277 1 log.go:172] http: TLS handshake error from 127.0.0.1:43212: read tcp 127.53.210.227:7443->127.0.0.1:43212: read: connection reset by peer I0408 05:31:22.865466 1 log.go:172] http: TLS handshake error from 127.0.0.1:43304: read tcp 127.53.210.227:7443->127.0.0.1:43304: read: connection reset by peer I0408 05:31:32.865453 1 log.go:172] http: TLS handshake error from 127.0.0.1:43396: read tcp 127.53.210.227:7443->127.0.0.1:43396: read: connection reset by peer I0408 05:31:42.865396 1 log.go:172] http: TLS handshake error from 127.0.0.1:43488: read tcp 127.53.210.227:7443->127.0.0.1:43488: read: connection reset by peer I0408 05:31:52.865568 1 log.go:172] http: TLS handshake error from 127.0.0.1:43580: read tcp 127.53.210.227:7443->127.0.0.1:43580: read: connection reset by peer I0408 05:32:02.865331 1 log.go:172] http: TLS handshake error from 127.0.0.1:43672: read tcp 127.53.210.227:7443->127.0.0.1:43672: read: connection reset by peer I0408 05:32:12.865538 1 log.go:172] http: TLS handshake error from 127.0.0.1:43764: read tcp 127.53.210.227:7443->127.0.0.1:43764: read: connection reset by peer I0408 05:32:22.865634 1 log.go:172] http: TLS handshake error from 127.0.0.1:43856: read tcp 127.53.210.227:7443->127.0.0.1:43856: read: connection reset by peer I0408 05:32:32.865440 1 log.go:172] http: TLS handshake error from 127.0.0.1:43948: read tcp 127.53.210.227:7443->127.0.0.1:43948: read: connection reset by peer I0408 05:32:42.865374 1 log.go:172] http: TLS handshake error from 127.0.0.1:44040: read tcp 127.53.210.227:7443->127.0.0.1:44040: read: connection reset by peer I0408 05:32:52.865481 1 log.go:172] http: TLS handshake error from 127.0.0.1:44132: read tcp 127.53.210.227:7443->127.0.0.1:44132: read: connection reset by peer I0408 05:33:02.865537 1 log.go:172] http: TLS handshake error from 127.0.0.1:44224: read tcp 127.53.210.227:7443->127.0.0.1:44224: read: connection reset by peer I0408 05:33:12.865388 1 log.go:172] http: TLS handshake error from 127.0.0.1:44316: read tcp 127.53.210.227:7443->127.0.0.1:44316: read: connection reset by peer I0408 05:33:22.865412 1 log.go:172] http: TLS handshake error from 127.0.0.1:44408: read tcp 127.53.210.227:7443->127.0.0.1:44408: read: connection reset by peer I0408 05:33:32.865414 1 log.go:172] http: TLS handshake error from 127.0.0.1:44500: read tcp 127.53.210.227:7443->127.0.0.1:44500: read: connection reset by peer I0408 05:33:42.865310 1 log.go:172] http: TLS handshake error from 127.0.0.1:44592: read tcp 127.53.210.227:7443->127.0.0.1:44592: read: connection reset by peer I0408 05:33:52.865661 1 log.go:172] http: TLS handshake error from 127.0.0.1:44688: read tcp 127.53.210.227:7443->127.0.0.1:44688: read: connection reset by peer I0408 05:34:02.865376 1 log.go:172] http: TLS handshake error from 127.0.0.1:44780: read tcp 127.53.210.227:7443->127.0.0.1:44780: read: connection reset by peer I0408 05:34:12.865225 1 log.go:172] http: TLS handshake error from 127.0.0.1:44872: read tcp 127.53.210.227:7443->127.0.0.1:44872: read: connection reset by peer I0408 05:34:22.865455 1 log.go:172] http: TLS handshake error from 127.0.0.1:44964: read tcp 127.53.210.227:7443->127.0.0.1:44964: read: connection reset by peer I0408 05:34:32.865384 1 log.go:172] http: TLS handshake error from 127.0.0.1:45056: read tcp 127.53.210.227:7443->127.0.0.1:45056: read: connection reset by peer I0408 05:34:42.865546 1 log.go:172] http: TLS handshake error from 127.0.0.1:45148: read tcp 127.53.210.227:7443->127.0.0.1:45148: read: connection reset by peer I0408 05:34:52.865459 1 log.go:172] http: TLS handshake error from 127.0.0.1:45240: read tcp 127.53.210.227:7443->127.0.0.1:45240: read: connection reset by peer I0408 05:35:02.865305 1 log.go:172] http: TLS handshake error from 127.0.0.1:45332: read tcp 127.53.210.227:7443->127.0.0.1:45332: read: connection reset by peer I0408 05:35:12.865457 1 log.go:172] http: TLS handshake error from 127.0.0.1:45424: read tcp 127.53.210.227:7443->127.0.0.1:45424: read: connection reset by peer I0408 05:35:22.865279 1 log.go:172] http: TLS handshake error from 127.0.0.1:45516: read tcp 127.53.210.227:7443->127.0.0.1:45516: read: connection reset by peer I0408 05:35:32.865221 1 log.go:172] http: TLS handshake error from 127.0.0.1:45608: read tcp 127.53.210.227:7443->127.0.0.1:45608: read: connection reset by peer I0408 05:35:42.865400 1 log.go:172] http: TLS handshake error from 127.0.0.1:45700: read tcp 127.53.210.227:7443->127.0.0.1:45700: read: connection reset by peer I0408 05:35:52.865556 1 log.go:172] http: TLS handshake error from 127.0.0.1:45792: read tcp 127.53.210.227:7443->127.0.0.1:45792: read: connection reset by peer I0408 05:36:02.865322 1 log.go:172] http: TLS handshake error from 127.0.0.1:45884: read tcp 127.53.210.227:7443->127.0.0.1:45884: read: connection reset by peer I0408 05:36:12.865451 1 log.go:172] http: TLS handshake error from 127.0.0.1:45976: read tcp 127.53.210.227:7443->127.0.0.1:45976: read: connection reset by peer I0408 05:36:22.865459 1 log.go:172] http: TLS handshake error from 127.0.0.1:46068: read tcp 127.53.210.227:7443->127.0.0.1:46068: read: connection reset by peer I0408 05:36:32.865464 1 log.go:172] http: TLS handshake error from 127.0.0.1:46160: read tcp 127.53.210.227:7443->127.0.0.1:46160: read: connection reset by peer I0408 05:36:42.865437 1 log.go:172] http: TLS handshake error from 127.0.0.1:46252: read tcp 127.53.210.227:7443->127.0.0.1:46252: read: connection reset by peer I0408 05:36:52.865331 1 log.go:172] http: TLS handshake error from 127.0.0.1:46344: read tcp 127.53.210.227:7443->127.0.0.1:46344: read: connection reset by peer I0408 05:37:02.865328 1 log.go:172] http: TLS handshake error from 127.0.0.1:46432: read tcp 127.53.210.227:7443->127.0.0.1:46432: read: connection reset by peer I0408 05:37:12.865286 1 log.go:172] http: TLS handshake error from 127.0.0.1:46524: read tcp 127.53.210.227:7443->127.0.0.1:46524: read: connection reset by peer I0408 05:37:22.865442 1 log.go:172] http: TLS handshake error from 127.0.0.1:46616: read tcp 127.53.210.227:7443->127.0.0.1:46616: read: connection reset by peer I0408 05:37:32.865490 1 log.go:172] http: TLS handshake error from 127.0.0.1:46708: read tcp 127.53.210.227:7443->127.0.0.1:46708: read: connection reset by peer I0408 05:37:42.865182 1 log.go:172] http: TLS handshake error from 127.0.0.1:46800: read tcp 127.53.210.227:7443->127.0.0.1:46800: read: connection reset by peer I0408 05:37:52.865407 1 log.go:172] http: TLS handshake error from 127.0.0.1:46892: read tcp 127.53.210.227:7443->127.0.0.1:46892: read: connection reset by peer I0408 05:38:02.865497 1 log.go:172] http: TLS handshake error from 127.0.0.1:46984: read tcp 127.53.210.227:7443->127.0.0.1:46984: read: connection reset by peer I0408 05:38:12.865294 1 log.go:172] http: TLS handshake error from 127.0.0.1:47076: read tcp 127.53.210.227:7443->127.0.0.1:47076: read: connection reset by peer I0408 05:38:22.865423 1 log.go:172] http: TLS handshake error from 127.0.0.1:47168: read tcp 127.53.210.227:7443->127.0.0.1:47168: read: connection reset by peer I0408 05:38:32.865259 1 log.go:172] http: TLS handshake error from 127.0.0.1:47260: read tcp 127.53.210.227:7443->127.0.0.1:47260: read: connection reset by peer I0408 05:38:42.865257 1 log.go:172] http: TLS handshake error from 127.0.0.1:47352: read tcp 127.53.210.227:7443->127.0.0.1:47352: read: connection reset by peer I0408 05:38:52.865414 1 log.go:172] http: TLS handshake error from 127.0.0.1:47448: read tcp 127.53.210.227:7443->127.0.0.1:47448: read: connection reset by peer I0408 05:39:02.865317 1 log.go:172] http: TLS handshake error from 127.0.0.1:47540: read tcp 127.53.210.227:7443->127.0.0.1:47540: read: connection reset by peer I0408 05:39:12.865214 1 log.go:172] http: TLS handshake error from 127.0.0.1:47632: read tcp 127.53.210.227:7443->127.0.0.1:47632: read: connection reset by peer I0408 05:39:22.865372 1 log.go:172] http: TLS handshake error from 127.0.0.1:47724: read tcp 127.53.210.227:7443->127.0.0.1:47724: read: connection reset by peer I0408 05:39:32.865355 1 log.go:172] http: TLS handshake error from 127.0.0.1:47816: read tcp 127.53.210.227:7443->127.0.0.1:47816: read: connection reset by peer I0408 05:39:42.865518 1 log.go:172] http: TLS handshake error from 127.0.0.1:47908: read tcp 127.53.210.227:7443->127.0.0.1:47908: read: connection reset by peer I0408 05:39:52.865384 1 log.go:172] http: TLS handshake error from 127.0.0.1:48000: read tcp 127.53.210.227:7443->127.0.0.1:48000: read: connection reset by peer I0408 05:40:02.865402 1 log.go:172] http: TLS handshake error from 127.0.0.1:48092: read tcp 127.53.210.227:7443->127.0.0.1:48092: read: connection reset by peer I0408 05:40:12.865258 1 log.go:172] http: TLS handshake error from 127.0.0.1:48184: read tcp 127.53.210.227:7443->127.0.0.1:48184: read: connection reset by peer I0408 05:40:22.865564 1 log.go:172] http: TLS handshake error from 127.0.0.1:48276: read tcp 127.53.210.227:7443->127.0.0.1:48276: read: connection reset by peer I0408 05:40:32.865502 1 log.go:172] http: TLS handshake error from 127.0.0.1:48368: read tcp 127.53.210.227:7443->127.0.0.1:48368: read: connection reset by peer I0408 05:40:42.865565 1 log.go:172] http: TLS handshake error from 127.0.0.1:48460: read tcp 127.53.210.227:7443->127.0.0.1:48460: read: connection reset by peer I0408 05:40:52.865413 1 log.go:172] http: TLS handshake error from 127.0.0.1:48552: read tcp 127.53.210.227:7443->127.0.0.1:48552: read: connection reset by peer I0408 05:41:02.865171 1 log.go:172] http: TLS handshake error from 127.0.0.1:48644: read tcp 127.53.210.227:7443->127.0.0.1:48644: read: connection reset by peer I0408 05:41:12.865409 1 log.go:172] http: TLS handshake error from 127.0.0.1:48736: read tcp 127.53.210.227:7443->127.0.0.1:48736: read: connection reset by peer I0408 05:41:22.865574 1 log.go:172] http: TLS handshake error from 127.0.0.1:48828: read tcp 127.53.210.227:7443->127.0.0.1:48828: read: connection reset by peer I0408 05:41:32.865503 1 log.go:172] http: TLS handshake error from 127.0.0.1:48920: read tcp 127.53.210.227:7443->127.0.0.1:48920: read: connection reset by peer I0408 05:41:42.865405 1 log.go:172] http: TLS handshake error from 127.0.0.1:49012: read tcp 127.53.210.227:7443->127.0.0.1:49012: read: connection reset by peer I0408 05:41:52.865455 1 log.go:172] http: TLS handshake error from 127.0.0.1:49104: read tcp 127.53.210.227:7443->127.0.0.1:49104: read: connection reset by peer I0408 05:42:02.865494 1 log.go:172] http: TLS handshake error from 127.0.0.1:49196: read tcp 127.53.210.227:7443->127.0.0.1:49196: read: connection reset by peer I0408 05:42:12.865059 1 log.go:172] http: TLS handshake error from 127.0.0.1:49288: read tcp 127.53.210.227:7443->127.0.0.1:49288: read: connection reset by peer I0408 05:42:22.865470 1 log.go:172] http: TLS handshake error from 127.0.0.1:49380: read tcp 127.53.210.227:7443->127.0.0.1:49380: read: connection reset by peer I0408 05:42:32.865373 1 log.go:172] http: TLS handshake error from 127.0.0.1:49472: read tcp 127.53.210.227:7443->127.0.0.1:49472: read: connection reset by peer I0408 05:42:42.865285 1 log.go:172] http: TLS handshake error from 127.0.0.1:49564: read tcp 127.53.210.227:7443->127.0.0.1:49564: read: connection reset by peer I0408 05:42:52.865520 1 log.go:172] http: TLS handshake error from 127.0.0.1:49656: read tcp 127.53.210.227:7443->127.0.0.1:49656: read: connection reset by peer I0408 05:43:02.865297 1 log.go:172] http: TLS handshake error from 127.0.0.1:49748: read tcp 127.53.210.227:7443->127.0.0.1:49748: read: connection reset by peer I0408 05:43:12.865343 1 log.go:172] http: TLS handshake error from 127.0.0.1:49840: read tcp 127.53.210.227:7443->127.0.0.1:49840: read: connection reset by peer I0408 05:43:22.865307 1 log.go:172] http: TLS handshake error from 127.0.0.1:49932: read tcp 127.53.210.227:7443->127.0.0.1:49932: read: connection reset by peer I0408 05:43:32.865337 1 log.go:172] http: TLS handshake error from 127.0.0.1:50024: read tcp 127.53.210.227:7443->127.0.0.1:50024: read: connection reset by peer I0408 05:43:42.865318 1 log.go:172] http: TLS handshake error from 127.0.0.1:50116: read tcp 127.53.210.227:7443->127.0.0.1:50116: read: connection reset by peer I0408 05:43:52.865386 1 log.go:172] http: TLS handshake error from 127.0.0.1:50212: read tcp 127.53.210.227:7443->127.0.0.1:50212: read: connection reset by peer I0408 05:44:02.865411 1 log.go:172] http: TLS handshake error from 127.0.0.1:50304: read tcp 127.53.210.227:7443->127.0.0.1:50304: read: connection reset by peer I0408 05:44:12.865383 1 log.go:172] http: TLS handshake error from 127.0.0.1:50396: read tcp 127.53.210.227:7443->127.0.0.1:50396: read: connection reset by peer I0408 05:44:22.865233 1 log.go:172] http: TLS handshake error from 127.0.0.1:50488: read tcp 127.53.210.227:7443->127.0.0.1:50488: read: connection reset by peer I0408 05:44:32.865547 1 log.go:172] http: TLS handshake error from 127.0.0.1:50580: read tcp 127.53.210.227:7443->127.0.0.1:50580: read: connection reset by peer I0408 05:44:42.865490 1 log.go:172] http: TLS handshake error from 127.0.0.1:50672: read tcp 127.53.210.227:7443->127.0.0.1:50672: read: connection reset by peer I0408 05:44:52.865319 1 log.go:172] http: TLS handshake error from 127.0.0.1:50764: read tcp 127.53.210.227:7443->127.0.0.1:50764: read: connection reset by peer I0408 05:45:02.865512 1 log.go:172] http: TLS handshake error from 127.0.0.1:50856: read tcp 127.53.210.227:7443->127.0.0.1:50856: read: connection reset by peer I0408 05:45:12.865495 1 log.go:172] http: TLS handshake error from 127.0.0.1:50948: read tcp 127.53.210.227:7443->127.0.0.1:50948: read: connection reset by peer I0408 05:45:22.865516 1 log.go:172] http: TLS handshake error from 127.0.0.1:51040: read tcp 127.53.210.227:7443->127.0.0.1:51040: read: connection reset by peer I0408 05:45:32.865363 1 log.go:172] http: TLS handshake error from 127.0.0.1:51132: read tcp 127.53.210.227:7443->127.0.0.1:51132: read: connection reset by peer I0408 05:45:42.865457 1 log.go:172] http: TLS handshake error from 127.0.0.1:51224: read tcp 127.53.210.227:7443->127.0.0.1:51224: read: connection reset by peer I0408 05:45:52.865375 1 log.go:172] http: TLS handshake error from 127.0.0.1:51316: read tcp 127.53.210.227:7443->127.0.0.1:51316: read: connection reset by peer I0408 05:46:02.865296 1 log.go:172] http: TLS handshake error from 127.0.0.1:51408: read tcp 127.53.210.227:7443->127.0.0.1:51408: read: connection reset by peer I0408 05:46:12.865155 1 log.go:172] http: TLS handshake error from 127.0.0.1:51500: read tcp 127.53.210.227:7443->127.0.0.1:51500: read: connection reset by peer I0408 05:46:22.865458 1 log.go:172] http: TLS handshake error from 127.0.0.1:51592: read tcp 127.53.210.227:7443->127.0.0.1:51592: read: connection reset by peer I0408 05:46:32.865262 1 log.go:172] http: TLS handshake error from 127.0.0.1:51684: read tcp 127.53.210.227:7443->127.0.0.1:51684: read: connection reset by peer I0408 05:46:42.865272 1 log.go:172] http: TLS handshake error from 127.0.0.1:51776: read tcp 127.53.210.227:7443->127.0.0.1:51776: read: connection reset by peer I0408 05:46:52.865323 1 log.go:172] http: TLS handshake error from 127.0.0.1:51868: read tcp 127.53.210.227:7443->127.0.0.1:51868: read: connection reset by peer I0408 05:47:02.865295 1 log.go:172] http: TLS handshake error from 127.0.0.1:51960: read tcp 127.53.210.227:7443->127.0.0.1:51960: read: connection reset by peer I0408 05:47:12.865556 1 log.go:172] http: TLS handshake error from 127.0.0.1:52052: read tcp 127.53.210.227:7443->127.0.0.1:52052: read: connection reset by peer I0408 05:47:22.865549 1 log.go:172] http: TLS handshake error from 127.0.0.1:52144: read tcp 127.53.210.227:7443->127.0.0.1:52144: read: connection reset by peer I0408 05:47:32.865223 1 log.go:172] http: TLS handshake error from 127.0.0.1:52236: read tcp 127.53.210.227:7443->127.0.0.1:52236: read: connection reset by peer I0408 05:47:42.865411 1 log.go:172] http: TLS handshake error from 127.0.0.1:52328: read tcp 127.53.210.227:7443->127.0.0.1:52328: read: connection reset by peer I0408 05:47:52.865384 1 log.go:172] http: TLS handshake error from 127.0.0.1:52420: read tcp 127.53.210.227:7443->127.0.0.1:52420: read: connection reset by peer I0408 05:48:02.865321 1 log.go:172] http: TLS handshake error from 127.0.0.1:52512: read tcp 127.53.210.227:7443->127.0.0.1:52512: read: connection reset by peer I0408 05:48:12.865144 1 log.go:172] http: TLS handshake error from 127.0.0.1:52604: read tcp 127.53.210.227:7443->127.0.0.1:52604: read: connection reset by peer I0408 05:48:22.865404 1 log.go:172] http: TLS handshake error from 127.0.0.1:52696: read tcp 127.53.210.227:7443->127.0.0.1:52696: read: connection reset by peer I0408 05:48:32.865347 1 log.go:172] http: TLS handshake error from 127.0.0.1:52788: read tcp 127.53.210.227:7443->127.0.0.1:52788: read: connection reset by peer I0408 05:48:42.865366 1 log.go:172] http: TLS handshake error from 127.0.0.1:52880: read tcp 127.53.210.227:7443->127.0.0.1:52880: read: connection reset by peer I0408 05:48:52.865307 1 log.go:172] http: TLS handshake error from 127.0.0.1:52976: read tcp 127.53.210.227:7443->127.0.0.1:52976: read: connection reset by peer I0408 05:49:02.866662 1 log.go:172] http: TLS handshake error from 127.0.0.1:53068: read tcp 127.53.210.227:7443->127.0.0.1:53068: read: connection reset by peer I0408 05:49:12.865438 1 log.go:172] http: TLS handshake error from 127.0.0.1:53160: read tcp 127.53.210.227:7443->127.0.0.1:53160: read: connection reset by peer I0408 05:49:22.865304 1 log.go:172] http: TLS handshake error from 127.0.0.1:53252: read tcp 127.53.210.227:7443->127.0.0.1:53252: read: connection reset by peer I0408 05:49:32.865402 1 log.go:172] http: TLS handshake error from 127.0.0.1:53344: read tcp 127.53.210.227:7443->127.0.0.1:53344: read: connection reset by peer I0408 05:49:42.865641 1 log.go:172] http: TLS handshake error from 127.0.0.1:53436: read tcp 127.53.210.227:7443->127.0.0.1:53436: read: connection reset by peer I0408 05:49:52.865224 1 log.go:172] http: TLS handshake error from 127.0.0.1:53528: read tcp 127.53.210.227:7443->127.0.0.1:53528: read: connection reset by peer I0408 05:50:02.865155 1 log.go:172] http: TLS handshake error from 127.0.0.1:53620: read tcp 127.53.210.227:7443->127.0.0.1:53620: read: connection reset by peer I0408 05:50:12.865536 1 log.go:172] http: TLS handshake error from 127.0.0.1:53712: read tcp 127.53.210.227:7443->127.0.0.1:53712: read: connection reset by peer I0408 05:50:22.865156 1 log.go:172] http: TLS handshake error from 127.0.0.1:53806: read tcp 127.53.210.227:7443->127.0.0.1:53806: read: connection reset by peer I0408 05:50:32.865137 1 log.go:172] http: TLS handshake error from 127.0.0.1:53898: read tcp 127.53.210.227:7443->127.0.0.1:53898: read: connection reset by peer I0408 05:50:42.865485 1 log.go:172] http: TLS handshake error from 127.0.0.1:53990: read tcp 127.53.210.227:7443->127.0.0.1:53990: read: connection reset by peer I0408 05:50:52.865177 1 log.go:172] http: TLS handshake error from 127.0.0.1:54082: read tcp 127.53.210.227:7443->127.0.0.1:54082: read: connection reset by peer I0408 05:51:02.865430 1 log.go:172] http: TLS handshake error from 127.0.0.1:54174: read tcp 127.53.210.227:7443->127.0.0.1:54174: read: connection reset by peer I0408 05:51:12.865206 1 log.go:172] http: TLS handshake error from 127.0.0.1:54266: read tcp 127.53.210.227:7443->127.0.0.1:54266: read: connection reset by peer I0408 05:51:22.865296 1 log.go:172] http: TLS handshake error from 127.0.0.1:54358: read tcp 127.53.210.227:7443->127.0.0.1:54358: read: connection reset by peer I0408 05:51:32.865498 1 log.go:172] http: TLS handshake error from 127.0.0.1:54450: read tcp 127.53.210.227:7443->127.0.0.1:54450: read: connection reset by peer I0408 05:51:42.865374 1 log.go:172] http: TLS handshake error from 127.0.0.1:54542: read tcp 127.53.210.227:7443->127.0.0.1:54542: read: connection reset by peer I0408 05:51:52.865416 1 log.go:172] http: TLS handshake error from 127.0.0.1:54634: read tcp 127.53.210.227:7443->127.0.0.1:54634: read: connection reset by peer I0408 05:52:02.865443 1 log.go:172] http: TLS handshake error from 127.0.0.1:54726: read tcp 127.53.210.227:7443->127.0.0.1:54726: read: connection reset by peer I0408 05:52:12.865529 1 log.go:172] http: TLS handshake error from 127.0.0.1:54818: read tcp 127.53.210.227:7443->127.0.0.1:54818: read: connection reset by peer I0408 05:52:14.743943 1 log.go:172] http: TLS handshake error from 127.0.0.1:54838: local error: tls: unexpected message I0408 05:52:22.865542 1 log.go:172] http: TLS handshake error from 127.0.0.1:54912: read tcp 127.53.210.227:7443->127.0.0.1:54912: read: connection reset by peer I0408 05:52:32.865403 1 log.go:172] http: TLS handshake error from 127.0.0.1:55004: read tcp 127.53.210.227:7443->127.0.0.1:55004: read: connection reset by peer I0408 05:52:42.865443 1 log.go:172] http: TLS handshake error from 127.0.0.1:55096: read tcp 127.53.210.227:7443->127.0.0.1:55096: read: connection reset by peer I0408 05:52:52.865553 1 log.go:172] http: TLS handshake error from 127.0.0.1:55188: read tcp 127.53.210.227:7443->127.0.0.1:55188: read: connection reset by peer I0408 05:53:02.865251 1 log.go:172] http: TLS handshake error from 127.0.0.1:55280: read tcp 127.53.210.227:7443->127.0.0.1:55280: read: connection reset by peer I0408 05:53:12.865387 1 log.go:172] http: TLS handshake error from 127.0.0.1:55372: read tcp 127.53.210.227:7443->127.0.0.1:55372: read: connection reset by peer I0408 05:53:22.865250 1 log.go:172] http: TLS handshake error from 127.0.0.1:55464: read tcp 127.53.210.227:7443->127.0.0.1:55464: read: connection reset by peer I0408 05:53:32.865462 1 log.go:172] http: TLS handshake error from 127.0.0.1:55556: read tcp 127.53.210.227:7443->127.0.0.1:55556: read: connection reset by peer I0408 05:53:42.865295 1 log.go:172] http: TLS handshake error from 127.0.0.1:55648: read tcp 127.53.210.227:7443->127.0.0.1:55648: read: connection reset by peer I0408 05:53:52.865288 1 log.go:172] http: TLS handshake error from 127.0.0.1:55744: read tcp 127.53.210.227:7443->127.0.0.1:55744: read: connection reset by peer I0408 05:54:02.865441 1 log.go:172] http: TLS handshake error from 127.0.0.1:55836: read tcp 127.53.210.227:7443->127.0.0.1:55836: read: connection reset by peer I0408 05:54:12.865470 1 log.go:172] http: TLS handshake error from 127.0.0.1:55928: read tcp 127.53.210.227:7443->127.0.0.1:55928: read: connection reset by peer I0408 05:54:19.859764 1 log.go:172] http: TLS handshake error from 127.0.0.1:55984: local error: tls: unexpected message I0408 05:54:22.865213 1 log.go:172] http: TLS handshake error from 127.0.0.1:56022: read tcp 127.53.210.227:7443->127.0.0.1:56022: read: connection reset by peer I0408 05:54:32.865237 1 log.go:172] http: TLS handshake error from 127.0.0.1:56114: read tcp 127.53.210.227:7443->127.0.0.1:56114: read: connection reset by peer I0408 05:54:42.865463 1 log.go:172] http: TLS handshake error from 127.0.0.1:56206: read tcp 127.53.210.227:7443->127.0.0.1:56206: read: connection reset by peer I0408 05:54:52.865470 1 log.go:172] http: TLS handshake error from 127.0.0.1:56298: read tcp 127.53.210.227:7443->127.0.0.1:56298: read: connection reset by peer I0408 05:55:02.865285 1 log.go:172] http: TLS handshake error from 127.0.0.1:56390: read tcp 127.53.210.227:7443->127.0.0.1:56390: read: connection reset by peer I0408 05:55:12.865459 1 log.go:172] http: TLS handshake error from 127.0.0.1:56482: read tcp 127.53.210.227:7443->127.0.0.1:56482: read: connection reset by peer I0408 05:55:22.865471 1 log.go:172] http: TLS handshake error from 127.0.0.1:56574: read tcp 127.53.210.227:7443->127.0.0.1:56574: read: connection reset by peer I0408 05:55:32.865410 1 log.go:172] http: TLS handshake error from 127.0.0.1:56666: read tcp 127.53.210.227:7443->127.0.0.1:56666: read: connection reset by peer I0408 05:55:42.865578 1 log.go:172] http: TLS handshake error from 127.0.0.1:56758: read tcp 127.53.210.227:7443->127.0.0.1:56758: read: connection reset by peer I0408 05:55:52.865486 1 log.go:172] http: TLS handshake error from 127.0.0.1:56850: read tcp 127.53.210.227:7443->127.0.0.1:56850: read: connection reset by peer I0408 05:56:02.865320 1 log.go:172] http: TLS handshake error from 127.0.0.1:56942: read tcp 127.53.210.227:7443->127.0.0.1:56942: read: connection reset by peer I0408 05:56:12.865486 1 log.go:172] http: TLS handshake error from 127.0.0.1:57034: read tcp 127.53.210.227:7443->127.0.0.1:57034: read: connection reset by peer I0408 05:56:22.865361 1 log.go:172] http: TLS handshake error from 127.0.0.1:57126: read tcp 127.53.210.227:7443->127.0.0.1:57126: read: connection reset by peer I0408 05:56:32.865527 1 log.go:172] http: TLS handshake error from 127.0.0.1:57218: read tcp 127.53.210.227:7443->127.0.0.1:57218: read: connection reset by peer I0408 05:56:42.865524 1 log.go:172] http: TLS handshake error from 127.0.0.1:57310: read tcp 127.53.210.227:7443->127.0.0.1:57310: read: connection reset by peer I0408 05:56:52.865455 1 log.go:172] http: TLS handshake error from 127.0.0.1:57402: read tcp 127.53.210.227:7443->127.0.0.1:57402: read: connection reset by peer I0408 05:57:02.865471 1 log.go:172] http: TLS handshake error from 127.0.0.1:57494: read tcp 127.53.210.227:7443->127.0.0.1:57494: read: connection reset by peer I0408 05:57:12.865307 1 log.go:172] http: TLS handshake error from 127.0.0.1:57586: read tcp 127.53.210.227:7443->127.0.0.1:57586: read: connection reset by peer I0408 05:57:22.865363 1 log.go:172] http: TLS handshake error from 127.0.0.1:57678: read tcp 127.53.210.227:7443->127.0.0.1:57678: read: connection reset by peer I0408 05:57:32.865238 1 log.go:172] http: TLS handshake error from 127.0.0.1:57770: read tcp 127.53.210.227:7443->127.0.0.1:57770: read: connection reset by peer I0408 05:57:42.865349 1 log.go:172] http: TLS handshake error from 127.0.0.1:57862: read tcp 127.53.210.227:7443->127.0.0.1:57862: read: connection reset by peer I0408 05:57:52.865286 1 log.go:172] http: TLS handshake error from 127.0.0.1:57954: read tcp 127.53.210.227:7443->127.0.0.1:57954: read: connection reset by peer I0408 05:58:02.865504 1 log.go:172] http: TLS handshake error from 127.0.0.1:58046: read tcp 127.53.210.227:7443->127.0.0.1:58046: read: connection reset by peer I0408 05:58:12.865331 1 log.go:172] http: TLS handshake error from 127.0.0.1:58138: read tcp 127.53.210.227:7443->127.0.0.1:58138: read: connection reset by peer I0408 05:58:22.865428 1 log.go:172] http: TLS handshake error from 127.0.0.1:58230: read tcp 127.53.210.227:7443->127.0.0.1:58230: read: connection reset by peer I0408 05:58:32.865194 1 log.go:172] http: TLS handshake error from 127.0.0.1:58322: read tcp 127.53.210.227:7443->127.0.0.1:58322: read: connection reset by peer I0408 05:58:42.865434 1 log.go:172] http: TLS handshake error from 127.0.0.1:58414: read tcp 127.53.210.227:7443->127.0.0.1:58414: read: connection reset by peer I0408 05:58:52.865277 1 log.go:172] http: TLS handshake error from 127.0.0.1:58510: read tcp 127.53.210.227:7443->127.0.0.1:58510: read: connection reset by peer I0408 05:59:02.865571 1 log.go:172] http: TLS handshake error from 127.0.0.1:58602: read tcp 127.53.210.227:7443->127.0.0.1:58602: read: connection reset by peer I0408 05:59:12.865227 1 log.go:172] http: TLS handshake error from 127.0.0.1:58694: read tcp 127.53.210.227:7443->127.0.0.1:58694: read: connection reset by peer I0408 05:59:22.865478 1 log.go:172] http: TLS handshake error from 127.0.0.1:58786: read tcp 127.53.210.227:7443->127.0.0.1:58786: read: connection reset by peer I0408 05:59:32.865445 1 log.go:172] http: TLS handshake error from 127.0.0.1:58878: read tcp 127.53.210.227:7443->127.0.0.1:58878: read: connection reset by peer I0408 05:59:42.865338 1 log.go:172] http: TLS handshake error from 127.0.0.1:58970: read tcp 127.53.210.227:7443->127.0.0.1:58970: read: connection reset by peer I0408 05:59:52.865537 1 log.go:172] http: TLS handshake error from 127.0.0.1:59062: read tcp 127.53.210.227:7443->127.0.0.1:59062: read: connection reset by peer I0408 06:00:02.865498 1 log.go:172] http: TLS handshake error from 127.0.0.1:59154: read tcp 127.53.210.227:7443->127.0.0.1:59154: read: connection reset by peer I0408 06:00:12.865290 1 log.go:172] http: TLS handshake error from 127.0.0.1:59246: read tcp 127.53.210.227:7443->127.0.0.1:59246: read: connection reset by peer I0408 06:00:22.865429 1 log.go:172] http: TLS handshake error from 127.0.0.1:59338: read tcp 127.53.210.227:7443->127.0.0.1:59338: read: connection reset by peer I0408 06:00:32.865367 1 log.go:172] http: TLS handshake error from 127.0.0.1:59430: read tcp 127.53.210.227:7443->127.0.0.1:59430: read: connection reset by peer I0408 06:00:42.865347 1 log.go:172] http: TLS handshake error from 127.0.0.1:59522: read tcp 127.53.210.227:7443->127.0.0.1:59522: read: connection reset by peer I0408 06:00:52.865537 1 log.go:172] http: TLS handshake error from 127.0.0.1:59614: read tcp 127.53.210.227:7443->127.0.0.1:59614: read: connection reset by peer I0408 06:01:02.865561 1 log.go:172] http: TLS handshake error from 127.0.0.1:59706: read tcp 127.53.210.227:7443->127.0.0.1:59706: read: connection reset by peer I0408 06:01:12.865289 1 log.go:172] http: TLS handshake error from 127.0.0.1:59798: read tcp 127.53.210.227:7443->127.0.0.1:59798: read: connection reset by peer I0408 06:01:22.865252 1 log.go:172] http: TLS handshake error from 127.0.0.1:59890: read tcp 127.53.210.227:7443->127.0.0.1:59890: read: connection reset by peer I0408 06:01:32.865478 1 log.go:172] http: TLS handshake error from 127.0.0.1:59982: read tcp 127.53.210.227:7443->127.0.0.1:59982: read: connection reset by peer I0408 06:01:42.865586 1 log.go:172] http: TLS handshake error from 127.0.0.1:60074: read tcp 127.53.210.227:7443->127.0.0.1:60074: read: connection reset by peer I0408 06:01:52.865424 1 log.go:172] http: TLS handshake error from 127.0.0.1:60166: read tcp 127.53.210.227:7443->127.0.0.1:60166: read: connection reset by peer I0408 06:02:02.865383 1 log.go:172] http: TLS handshake error from 127.0.0.1:60258: read tcp 127.53.210.227:7443->127.0.0.1:60258: read: connection reset by peer I0408 06:02:12.865381 1 log.go:172] http: TLS handshake error from 127.0.0.1:60350: read tcp 127.53.210.227:7443->127.0.0.1:60350: read: connection reset by peer I0408 06:02:22.865452 1 log.go:172] http: TLS handshake error from 127.0.0.1:60442: read tcp 127.53.210.227:7443->127.0.0.1:60442: read: connection reset by peer I0408 06:02:32.865363 1 log.go:172] http: TLS handshake error from 127.0.0.1:60534: read tcp 127.53.210.227:7443->127.0.0.1:60534: read: connection reset by peer I0408 06:02:42.866723 1 log.go:172] http: TLS handshake error from 127.0.0.1:60626: read tcp 127.53.210.227:7443->127.0.0.1:60626: read: connection reset by peer I0408 06:02:52.865380 1 log.go:172] http: TLS handshake error from 127.0.0.1:60718: read tcp 127.53.210.227:7443->127.0.0.1:60718: read: connection reset by peer I0408 06:03:02.866856 1 log.go:172] http: TLS handshake error from 127.0.0.1:60810: EOF I0408 06:03:12.865524 1 log.go:172] http: TLS handshake error from 127.0.0.1:60902: read tcp 127.53.210.227:7443->127.0.0.1:60902: read: connection reset by peer I0408 06:03:22.865306 1 log.go:172] http: TLS handshake error from 127.0.0.1:60994: read tcp 127.53.210.227:7443->127.0.0.1:60994: read: connection reset by peer I0408 06:03:32.865398 1 log.go:172] http: TLS handshake error from 127.0.0.1:32854: read tcp 127.53.210.227:7443->127.0.0.1:32854: read: connection reset by peer I0408 06:03:42.865598 1 log.go:172] http: TLS handshake error from 127.0.0.1:32946: read tcp 127.53.210.227:7443->127.0.0.1:32946: read: connection reset by peer I0408 06:03:52.865433 1 log.go:172] http: TLS handshake error from 127.0.0.1:33042: read tcp 127.53.210.227:7443->127.0.0.1:33042: read: connection reset by peer I0408 06:04:02.865440 1 log.go:172] http: TLS handshake error from 127.0.0.1:33134: read tcp 127.53.210.227:7443->127.0.0.1:33134: read: connection reset by peer I0408 06:04:12.865423 1 log.go:172] http: TLS handshake error from 127.0.0.1:33226: read tcp 127.53.210.227:7443->127.0.0.1:33226: read: connection reset by peer I0408 06:04:22.865508 1 log.go:172] http: TLS handshake error from 127.0.0.1:33320: read tcp 127.53.210.227:7443->127.0.0.1:33320: read: connection reset by peer I0408 06:04:32.865203 1 log.go:172] http: TLS handshake error from 127.0.0.1:33412: read tcp 127.53.210.227:7443->127.0.0.1:33412: read: connection reset by peer I0408 06:04:42.865285 1 log.go:172] http: TLS handshake error from 127.0.0.1:33504: read tcp 127.53.210.227:7443->127.0.0.1:33504: read: connection reset by peer I0408 06:04:52.865290 1 log.go:172] http: TLS handshake error from 127.0.0.1:33596: read tcp 127.53.210.227:7443->127.0.0.1:33596: read: connection reset by peer I0408 06:05:02.865398 1 log.go:172] http: TLS handshake error from 127.0.0.1:33688: read tcp 127.53.210.227:7443->127.0.0.1:33688: read: connection reset by peer I0408 06:05:12.865549 1 log.go:172] http: TLS handshake error from 127.0.0.1:33780: read tcp 127.53.210.227:7443->127.0.0.1:33780: read: connection reset by peer I0408 06:05:22.865171 1 log.go:172] http: TLS handshake error from 127.0.0.1:33874: read tcp 127.53.210.227:7443->127.0.0.1:33874: read: connection reset by peer I0408 06:05:32.865366 1 log.go:172] http: TLS handshake error from 127.0.0.1:33968: read tcp 127.53.210.227:7443->127.0.0.1:33968: read: connection reset by peer I0408 06:05:42.865595 1 log.go:172] http: TLS handshake error from 127.0.0.1:34060: read tcp 127.53.210.227:7443->127.0.0.1:34060: read: connection reset by peer I0408 06:05:52.865234 1 log.go:172] http: TLS handshake error from 127.0.0.1:34152: read tcp 127.53.210.227:7443->127.0.0.1:34152: read: connection reset by peer I0408 06:06:02.865296 1 log.go:172] http: TLS handshake error from 127.0.0.1:34246: read tcp 127.53.210.227:7443->127.0.0.1:34246: read: connection reset by peer I0408 06:06:12.865477 1 log.go:172] http: TLS handshake error from 127.0.0.1:34338: read tcp 127.53.210.227:7443->127.0.0.1:34338: read: connection reset by peer I0408 06:06:22.865361 1 log.go:172] http: TLS handshake error from 127.0.0.1:34432: read tcp 127.53.210.227:7443->127.0.0.1:34432: read: connection reset by peer I0408 06:06:32.865286 1 log.go:172] http: TLS handshake error from 127.0.0.1:34524: read tcp 127.53.210.227:7443->127.0.0.1:34524: read: connection reset by peer I0408 06:06:42.865528 1 log.go:172] http: TLS handshake error from 127.0.0.1:34616: read tcp 127.53.210.227:7443->127.0.0.1:34616: read: connection reset by peer I0408 06:06:52.865208 1 log.go:172] http: TLS handshake error from 127.0.0.1:34708: read tcp 127.53.210.227:7443->127.0.0.1:34708: read: connection reset by peer I0408 06:07:02.865373 1 log.go:172] http: TLS handshake error from 127.0.0.1:34800: read tcp 127.53.210.227:7443->127.0.0.1:34800: read: connection reset by peer I0408 06:07:12.865449 1 log.go:172] http: TLS handshake error from 127.0.0.1:34892: read tcp 127.53.210.227:7443->127.0.0.1:34892: read: connection reset by peer I0408 06:07:22.865453 1 log.go:172] http: TLS handshake error from 127.0.0.1:34984: read tcp 127.53.210.227:7443->127.0.0.1:34984: read: connection reset by peer I0408 06:07:32.865236 1 log.go:172] http: TLS handshake error from 127.0.0.1:35076: read tcp 127.53.210.227:7443->127.0.0.1:35076: read: connection reset by peer I0408 06:07:42.865401 1 log.go:172] http: TLS handshake error from 127.0.0.1:35168: read tcp 127.53.210.227:7443->127.0.0.1:35168: read: connection reset by peer I0408 06:07:52.865301 1 log.go:172] http: TLS handshake error from 127.0.0.1:35262: read tcp 127.53.210.227:7443->127.0.0.1:35262: read: connection reset by peer I0408 06:08:02.865473 1 log.go:172] http: TLS handshake error from 127.0.0.1:35354: read tcp 127.53.210.227:7443->127.0.0.1:35354: read: connection reset by peer I0408 06:08:12.865352 1 log.go:172] http: TLS handshake error from 127.0.0.1:35446: read tcp 127.53.210.227:7443->127.0.0.1:35446: read: connection reset by peer I0408 06:08:22.865163 1 log.go:172] http: TLS handshake error from 127.0.0.1:35538: read tcp 127.53.210.227:7443->127.0.0.1:35538: read: connection reset by peer I0408 06:08:32.865322 1 log.go:172] http: TLS handshake error from 127.0.0.1:35626: read tcp 127.53.210.227:7443->127.0.0.1:35626: read: connection reset by peer I0408 06:08:42.865419 1 log.go:172] http: TLS handshake error from 127.0.0.1:35718: read tcp 127.53.210.227:7443->127.0.0.1:35718: read: connection reset by peer I0408 06:08:52.865449 1 log.go:172] http: TLS handshake error from 127.0.0.1:35814: read tcp 127.53.210.227:7443->127.0.0.1:35814: read: connection reset by peer I0408 06:09:02.865388 1 log.go:172] http: TLS handshake error from 127.0.0.1:35906: read tcp 127.53.210.227:7443->127.0.0.1:35906: read: connection reset by peer I0408 06:09:12.865479 1 log.go:172] http: TLS handshake error from 127.0.0.1:35998: read tcp 127.53.210.227:7443->127.0.0.1:35998: read: connection reset by peer I0408 06:09:22.865360 1 log.go:172] http: TLS handshake error from 127.0.0.1:36090: read tcp 127.53.210.227:7443->127.0.0.1:36090: read: connection reset by peer I0408 06:09:32.865327 1 log.go:172] http: TLS handshake error from 127.0.0.1:36182: read tcp 127.53.210.227:7443->127.0.0.1:36182: read: connection reset by peer I0408 06:09:42.865429 1 log.go:172] http: TLS handshake error from 127.0.0.1:36274: read tcp 127.53.210.227:7443->127.0.0.1:36274: read: connection reset by peer I0408 06:09:52.865401 1 log.go:172] http: TLS handshake error from 127.0.0.1:36366: read tcp 127.53.210.227:7443->127.0.0.1:36366: read: connection reset by peer I0408 06:10:02.865316 1 log.go:172] http: TLS handshake error from 127.0.0.1:36458: read tcp 127.53.210.227:7443->127.0.0.1:36458: read: connection reset by peer I0408 06:10:12.865383 1 log.go:172] http: TLS handshake error from 127.0.0.1:36550: read tcp 127.53.210.227:7443->127.0.0.1:36550: read: connection reset by peer I0408 06:10:22.865301 1 log.go:172] http: TLS handshake error from 127.0.0.1:36642: read tcp 127.53.210.227:7443->127.0.0.1:36642: read: connection reset by peer I0408 06:10:32.865382 1 log.go:172] http: TLS handshake error from 127.0.0.1:36734: read tcp 127.53.210.227:7443->127.0.0.1:36734: read: connection reset by peer I0408 06:10:42.865187 1 log.go:172] http: TLS handshake error from 127.0.0.1:36826: read tcp 127.53.210.227:7443->127.0.0.1:36826: read: connection reset by peer I0408 06:10:52.865353 1 log.go:172] http: TLS handshake error from 127.0.0.1:36918: read tcp 127.53.210.227:7443->127.0.0.1:36918: read: connection reset by peer I0408 06:11:02.865408 1 log.go:172] http: TLS handshake error from 127.0.0.1:37010: read tcp 127.53.210.227:7443->127.0.0.1:37010: read: connection reset by peer I0408 06:11:12.865381 1 log.go:172] http: TLS handshake error from 127.0.0.1:37102: read tcp 127.53.210.227:7443->127.0.0.1:37102: read: connection reset by peer I0408 06:11:22.865598 1 log.go:172] http: TLS handshake error from 127.0.0.1:37194: read tcp 127.53.210.227:7443->127.0.0.1:37194: read: connection reset by peer I0408 06:11:32.865314 1 log.go:172] http: TLS handshake error from 127.0.0.1:37286: read tcp 127.53.210.227:7443->127.0.0.1:37286: read: connection reset by peer I0408 06:11:42.865377 1 log.go:172] http: TLS handshake error from 127.0.0.1:37378: read tcp 127.53.210.227:7443->127.0.0.1:37378: read: connection reset by peer I0408 06:11:52.865389 1 log.go:172] http: TLS handshake error from 127.0.0.1:37470: read tcp 127.53.210.227:7443->127.0.0.1:37470: read: connection reset by peer I0408 06:12:02.865335 1 log.go:172] http: TLS handshake error from 127.0.0.1:37562: read tcp 127.53.210.227:7443->127.0.0.1:37562: read: connection reset by peer I0408 06:12:12.865316 1 log.go:172] http: TLS handshake error from 127.0.0.1:37654: read tcp 127.53.210.227:7443->127.0.0.1:37654: read: connection reset by peer I0408 06:12:22.865438 1 log.go:172] http: TLS handshake error from 127.0.0.1:37746: read tcp 127.53.210.227:7443->127.0.0.1:37746: read: connection reset by peer I0408 06:12:32.865171 1 log.go:172] http: TLS handshake error from 127.0.0.1:37838: read tcp 127.53.210.227:7443->127.0.0.1:37838: read: connection reset by peer I0408 06:12:42.865350 1 log.go:172] http: TLS handshake error from 127.0.0.1:37930: read tcp 127.53.210.227:7443->127.0.0.1:37930: read: connection reset by peer I0408 06:12:52.865444 1 log.go:172] http: TLS handshake error from 127.0.0.1:38022: read tcp 127.53.210.227:7443->127.0.0.1:38022: read: connection reset by peer I0408 06:13:02.865308 1 log.go:172] http: TLS handshake error from 127.0.0.1:38114: read tcp 127.53.210.227:7443->127.0.0.1:38114: read: connection reset by peer I0408 06:13:12.865454 1 log.go:172] http: TLS handshake error from 127.0.0.1:38206: read tcp 127.53.210.227:7443->127.0.0.1:38206: read: connection reset by peer I0408 06:13:22.865630 1 log.go:172] http: TLS handshake error from 127.0.0.1:38298: read tcp 127.53.210.227:7443->127.0.0.1:38298: read: connection reset by peer I0408 06:13:32.865409 1 log.go:172] http: TLS handshake error from 127.0.0.1:38390: read tcp 127.53.210.227:7443->127.0.0.1:38390: read: connection reset by peer I0408 06:13:42.865335 1 log.go:172] http: TLS handshake error from 127.0.0.1:38482: read tcp 127.53.210.227:7443->127.0.0.1:38482: read: connection reset by peer I0408 06:13:52.865430 1 log.go:172] http: TLS handshake error from 127.0.0.1:38578: read tcp 127.53.210.227:7443->127.0.0.1:38578: read: connection reset by peer I0408 06:14:02.865206 1 log.go:172] http: TLS handshake error from 127.0.0.1:38670: read tcp 127.53.210.227:7443->127.0.0.1:38670: read: connection reset by peer I0408 06:14:12.865130 1 log.go:172] http: TLS handshake error from 127.0.0.1:38762: read tcp 127.53.210.227:7443->127.0.0.1:38762: read: connection reset by peer I0408 06:14:22.865281 1 log.go:172] http: TLS handshake error from 127.0.0.1:38854: read tcp 127.53.210.227:7443->127.0.0.1:38854: read: connection reset by peer I0408 06:14:32.865253 1 log.go:172] http: TLS handshake error from 127.0.0.1:38946: read tcp 127.53.210.227:7443->127.0.0.1:38946: read: connection reset by peer I0408 06:14:42.865395 1 log.go:172] http: TLS handshake error from 127.0.0.1:39038: read tcp 127.53.210.227:7443->127.0.0.1:39038: read: connection reset by peer I0408 06:14:52.865437 1 log.go:172] http: TLS handshake error from 127.0.0.1:39130: read tcp 127.53.210.227:7443->127.0.0.1:39130: read: connection reset by peer I0408 06:15:02.865453 1 log.go:172] http: TLS handshake error from 127.0.0.1:39222: read tcp 127.53.210.227:7443->127.0.0.1:39222: read: connection reset by peer I0408 06:15:12.865547 1 log.go:172] http: TLS handshake error from 127.0.0.1:39314: read tcp 127.53.210.227:7443->127.0.0.1:39314: read: connection reset by peer I0408 06:15:22.865502 1 log.go:172] http: TLS handshake error from 127.0.0.1:39406: read tcp 127.53.210.227:7443->127.0.0.1:39406: read: connection reset by peer I0408 06:15:32.865460 1 log.go:172] http: TLS handshake error from 127.0.0.1:39498: read tcp 127.53.210.227:7443->127.0.0.1:39498: read: connection reset by peer I0408 06:15:42.865306 1 log.go:172] http: TLS handshake error from 127.0.0.1:39590: read tcp 127.53.210.227:7443->127.0.0.1:39590: read: connection reset by peer I0408 06:15:52.865289 1 log.go:172] http: TLS handshake error from 127.0.0.1:39682: read tcp 127.53.210.227:7443->127.0.0.1:39682: read: connection reset by peer I0408 06:16:02.865457 1 log.go:172] http: TLS handshake error from 127.0.0.1:39774: read tcp 127.53.210.227:7443->127.0.0.1:39774: read: connection reset by peer I0408 06:16:12.865339 1 log.go:172] http: TLS handshake error from 127.0.0.1:39866: read tcp 127.53.210.227:7443->127.0.0.1:39866: read: connection reset by peer I0408 06:16:22.865206 1 log.go:172] http: TLS handshake error from 127.0.0.1:39958: read tcp 127.53.210.227:7443->127.0.0.1:39958: read: connection reset by peer I0408 06:16:32.865490 1 log.go:172] http: TLS handshake error from 127.0.0.1:40050: read tcp 127.53.210.227:7443->127.0.0.1:40050: read: connection reset by peer I0408 06:16:42.865349 1 log.go:172] http: TLS handshake error from 127.0.0.1:40142: read tcp 127.53.210.227:7443->127.0.0.1:40142: read: connection reset by peer I0408 06:16:52.865376 1 log.go:172] http: TLS handshake error from 127.0.0.1:40234: read tcp 127.53.210.227:7443->127.0.0.1:40234: read: connection reset by peer I0408 06:17:02.865474 1 log.go:172] http: TLS handshake error from 127.0.0.1:40326: read tcp 127.53.210.227:7443->127.0.0.1:40326: read: connection reset by peer I0408 06:17:12.865195 1 log.go:172] http: TLS handshake error from 127.0.0.1:40418: read tcp 127.53.210.227:7443->127.0.0.1:40418: read: connection reset by peer I0408 06:17:22.865535 1 log.go:172] http: TLS handshake error from 127.0.0.1:40512: read tcp 127.53.210.227:7443->127.0.0.1:40512: read: connection reset by peer I0408 06:17:32.865299 1 log.go:172] http: TLS handshake error from 127.0.0.1:40608: read tcp 127.53.210.227:7443->127.0.0.1:40608: read: connection reset by peer I0408 06:17:33.272493 1 trace.go:116] Trace[797765573]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-764dc4d7f9-vrqkr/log,user-agent:kubectl/v1.18.0 (linux/amd64) kubernetes/9e99141,client:127.0.0.1 (started: 2020-04-08 06:17:28.364107649 +0000 UTC m=+3749.202602178) (total time: 4.908358993s): Trace[797765573]: [4.908357872s] [4.907056869s] Transformed response object I0408 06:17:42.865565 1 log.go:172] http: TLS handshake error from 127.0.0.1:40702: read tcp 127.53.210.227:7443->127.0.0.1:40702: read: connection reset by peer I0408 06:17:50.640977 1 trace.go:116] Trace[627371068]: "Get" url:/api/v1/namespaces/kube-system/pods/kube-apiserver-764dc4d7f9-vrqkr/log,user-agent:kubectl/v1.18.0 (linux/amd64) kubernetes/9e99141,client:127.0.0.1 (started: 2020-04-08 06:17:43.428327004 +0000 UTC m=+3764.266821533) (total time: 7.212624493s): Trace[627371068]: [7.212622802s] [7.211463662s] Transformed response object I0408 06:17:52.865576 1 log.go:172] http: TLS handshake error from 127.0.0.1:40796: read tcp 127.53.210.227:7443->127.0.0.1:40796: read: connection reset by peer ```

invidian commented 4 years ago

Is this on AWS? It might be HAProxy just doing the TCP health checks. I wonder if we can configure it to perform HTTP health checks.

Also, could we move this logs to pastebin or put it into collapse section, as they are very big.

surajssd commented 4 years ago

This is on packet.

invidian commented 4 years ago

This happens because kube-apiserver pod has TCP readiness probe defined. We should be using HTTPS probe instead:

        readinessProbe:
          httpGet:
            path: /healthz
            port: 8443
            scheme: HTTPS

We also need to change HAProxy to use HTTP probes as well:

          backend kube-apiserver
            option httpchk GET /healthz HTTP/1.1\r\nHost:\ kube-apiserver
            server 1 $ADDRESS:{{ .Values.securePort }} verify none check check-ssl
invidian commented 4 years ago

Okay, this is slightly more complicated, as we have --anonymous-auth=false, so out readiness probe would fail. We could use –token-auth-file and generate the probe token using Helm and then use httpHeaders to include Authentication header so the health probe would pass.

Please note, that this issue affects both single replica and multiple replicas of kube-apiserver.

iaguis commented 4 years ago

So AFAIU the readinessProbe is working fine (that is, the TCP connection is established) but the apiserver is logging an error because the connection fails the TLS handshake. So this is not really a problem, it's just annoying that it gets logged as an error.

If we want to get rid of it we could

  1. Enable anonymous auth which is enabled by default in k8s and distros like kubeadm, but I don't think we should do that because it decreases security (and it seems some past k8s CVEs used it...)
  2. Do as you said and use --token-auth-file. In this case we should create a user that only has access to a harmless API.
  3. Ignore this and live with errors in the logs.

Option 2 makes sense to me if we configure it right and document it properly.

invidian commented 4 years ago

Thanks for taking a look @iaguis and the summary. I'll look into option 2 then to see if I can get it to work easily.

invidian commented 4 years ago

Created a PR with fix for Packet: #297.

For AWS, the TLS handshake errors cannot be resolved, as they are caused by AWS Load Balancer, which is configured to only TCP checks and in HTTP mode, it does not support authentication header, so we would have to remove --anonymous-auth=false kube-apiserver option, which we shouldn't do.

invidian commented 4 years ago

See also upstream issues:

iaguis commented 4 years ago

Should we close this now?

invidian commented 4 years ago

Should we close this now?

Perhaps we should document the issue for AWS before closing.

invidian commented 4 years ago

Hmm, do you have some suggestions where we should put this information? It seems to me, that it should be in some kind of FAQ section, but we don't have this at the moment. Maybe I create docs/faq/aws.md? Or just docs/faq.md?

surajssd commented 4 years ago

Not sure what file to put it into, but rather than FAQ it should be under "known issues / gotchas" ?