http2: server: error reading preface from client, read: connection reset by peer

[khteh]

What happened:

[metrics-server-856457dd94-bq4x7] http2: server: error reading preface from client <IP>:35766: read tcp <IP>:4443-><IP>:35766: read: connection reset by peer

What you expected to happen: No error.

Anything else we need to know?:

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS
• Container Network Setup (flannel, calico, etc.): ?
• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-25T14:27:37Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.8-eks-ffeb93d", GitCommit:"abb98ec0631dfe573ec5eae40dc48fd8f2017424", GitTreeState:"clean", BuildDate:"2022-11-29T18:45:03Z", GoVersion:"go1.18.8", Compiler:"gc", Platform:"linux/amd64"}

• Metrics Server manifest https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml

spoiler for Metrics Server manifest:

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

• Status of Metrics API:

  spolier for Status of Metrics API:

  ``` kubectl describe apiservice v1beta1.metrics.k8s.io ``` ``` Name: v1beta1.metrics.k8s.io Namespace: Labels: k8s-app=metrics-server Annotations: API Version: apiregistration.k8s.io/v1 Kind: APIService Metadata: Creation Timestamp: 2023-03-22T07:59:57Z Managed Fields: API Version: apiregistration.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: .: k:{"type":"Available"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: Manager: kube-apiserver Operation: Update Subresource: status Time: 2023-03-22T07:59:57Z API Version: apiregistration.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:labels: .: f:k8s-app: f:spec: f:group: f:groupPriorityMinimum: f:insecureSkipTLSVerify: f:service: .: f:name: f:namespace: f:port: f:version: f:versionPriority: Manager: kubectl-client-side-apply Operation: Update Time: 2023-03-22T07:59:57Z Resource Version: 48512726 UID: 48556c78-9612-478f-bb71-09f805e05978 Spec: Group: metrics.k8s.io Group Priority Minimum: 100 Insecure Skip TLS Verify: true Service: Name: metrics-server Namespace: kube-system Port: 443 Version: v1beta1 Version Priority: 100 Status: Conditions: Last Transition Time: 2023-03-22T08:00:27Z Message: all checks passed Reason: Passed Status: True Type: Available Events: ```

/kind bug

[logicalhan]

/triage accepted /assign @dgrisonnet

[dgrisonnet]

This seems like a problem from your client resetting the connection before metrics-server is able to read any data that the client is supposed to be sending.

This should be pretty harmless but it could be worth checking if there is any misconfiguration on your client if that error happens frequently.

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[sondrelg]

Would it be possible to log this using the "regular" log formatting?

Here's a sample of our metrics-server pod stdout output:

E0722 10:29:00.913915       1 scraper.go:140] "Failed to scrape node" err="Get \"https://redacted:10250/metrics/resource\": context deadline exceeded" node="ip-redacted.eu-north-1.compute.internal"
E0722 10:29:15.908388       1 scraper.go:140] "Failed to scrape node" err="Get \"https://redacted:10250/metrics/resource\": context deadline exceeded" node="ip-redacted.eu-north-1.compute.internal"
http2: server: error reading preface from client redacted:56766: read tcp redacted:4443->redacted:56766: read: connection reset by peer
http2: server: error reading preface from client redacted:56766: read tcp redacted:4443->redacted:56766: read: connection reset by peer

We're parsing the first two lines using a log scraper, and the second two break our parsing logic.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/metrics-server/issues/1217#issuecomment-2304481769): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.