search

cybozu-go / accurate

Kubernetes controller for multi-tenancy. It propagates resources between namespaces accurately and allows tenant users to create/delete sub-namespaces.
https://cybozu-go.github.io/accurate/
Apache License 2.0
38 stars 5 forks source link

Annoying "http: TLS handshake error" in logs #104

Closed erikgb closed 3 days ago

erikgb commented 11 months ago

What

We are òbserving some mysterious errors in the Accurate pods:

2023/10/26 13:30:27 http: TLS handshake error from 10.101.10.1:53540: EOF
2023/10/26 13:30:27 http: TLS handshake error from 10.101.10.1:53546: EOF
2023/10/26 13:40:41 http: TLS handshake error from 10.101.10.1:47184: EOF
2023/10/26 13:40:41 http: TLS handshake error from 10.101.10.1:47194: EOF
2023/10/26 14:20:31 http: TLS handshake error from 10.101.10.1:41206: EOF
2023/10/26 15:20:41 http: TLS handshake error from 10.101.10.1:58124: EOF
2023/10/26 15:39:52 http: TLS handshake error from 10.101.10.1:58912: read tcp 10.101.13.130:9443->10.101.10.1:58912: read: connection reset by peer
2023/10/26 15:50:17 http: TLS handshake error from 10.101.10.1:53224: EOF
2023/10/26 16:20:22 http: TLS handshake error from 10.101.10.1:47576: EOF

Are anyone else observing something similar? The operator seems to work as it should, so this is just annoying messages in the logs. I suspect this might be related to https://github.com/kubernetes/kubernetes/issues/109022, and some suspect the root issue to be https://github.com/golang/go/issues/50984.

I'll suggest addressing https://github.com/cybozu-go/accurate/issues/103 to see if that can fix this issue.

How

Describe how to address the issue.

Checklist

erikgb commented 7 months ago

It would be nice to get to the bottom of this annoying bug. I still see this spamming the logs after upgrading to the latest version of Accurate (1.3.0):

2024/02/21 17:40:46 http: TLS handshake error from 10.101.4.1:60740: EOF
2024/02/21 18:19:35 http: TLS handshake error from 10.101.4.1:35884: EOF
2024/02/21 18:46:31 http: TLS handshake error from 10.101.4.1:46870: EOF
2024/02/21 20:06:17 http: TLS handshake error from 10.101.4.1:48064: EOF
2024/02/21 21:16:51 http: TLS handshake error from 10.101.4.1:60182: EOF
2024/02/21 21:17:25 http: TLS handshake error from 10.101.4.1:37474: EOF
2024/02/21 21:51:53 http: TLS handshake error from 10.101.4.1:43914: EOF
2024/02/21 22:07:02 http: TLS handshake error from 10.101.4.1:41566: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32964: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32976: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32978: EOF
2024/02/21 23:06:46 http: TLS handshake error from 10.101.4.1:42334: EOF
2024/02/21 23:19:32 http: TLS handshake error from 10.101.4.1:50816: EOF
2024/02/22 00:41:02 http: TLS handshake error from 10.101.4.1:55288: EOF
2024/02/22 00:50:44 http: TLS handshake error from 10.101.4.1:33014: EOF
2024/02/22 01:17:59 http: TLS handshake error from 10.101.4.1:39812: EOF
2024/02/22 01:21:29 http: TLS handshake error from 10.101.4.1:43696: EOF
2024/02/22 01:57:11 http: TLS handshake error from 10.101.4.1:36734: EOF
2024/02/22 02:01:17 http: TLS handshake error from 10.101.4.1:56558: EOF
2024/02/22 02:20:47 http: TLS handshake error from 10.101.4.1:53726: EOF
2024/02/22 02:42:26 http: TLS handshake error from 10.101.4.1:49920: EOF
2024/02/22 03:21:56 http: TLS handshake error from 10.101.4.1:58706: EOF
2024/02/22 03:42:03 http: TLS handshake error from 10.101.4.1:34064: EOF
2024/02/22 03:51:55 http: TLS handshake error from 10.101.4.1:35442: EOF
2024/02/22 03:52:26 http: TLS handshake error from 10.101.4.1:37308: EOF
2024/02/22 04:12:17 http: TLS handshake error from 10.101.4.1:52784: EOF
2024/02/22 04:21:43 http: TLS handshake error from 10.101.4.1:52986: EOF
2024/02/22 05:33:08 http: TLS handshake error from 10.101.4.1:48738: EOF
2024/02/22 05:42:24 http: TLS handshake error from 10.101.4.1:34250: EOF
2024/02/22 06:03:35 http: TLS handshake error from 10.101.4.1:33272: EOF
2024/02/22 06:22:36 http: TLS handshake error from 10.101.4.1:40154: EOF
2024/02/22 06:34:04 http: TLS handshake error from 10.101.4.1:47704: EOF
2024/02/22 06:39:35 http: TLS handshake error from 10.101.4.1:58154: EOF
2024/02/22 06:43:36 http: TLS handshake error from 10.101.4.1:60730: EOF
2024/02/22 07:13:22 http: TLS handshake error from 10.101.4.1:49454: EOF
2024/02/22 07:53:13 http: TLS handshake error from 10.101.4.1:35338: EOF
2024/02/22 08:20:07 http: TLS handshake error from 10.101.4.1:43340: EOF
2024/02/22 08:20:07 http: TLS handshake error from 10.101.4.1:43356: EOF
2024/02/22 08:59:44 http: TLS handshake error from 10.101.4.1:40694: EOF
2024/02/22 09:22:06 http: TLS handshake error from 10.101.4.1:54174: EOF

Am I the only one observing this? 🤔 @yamatcha, what about your installations of Accurate?

ymmt2005 commented 7 months ago

@erikgb can you tell what is running on 10.101.4.1?

erikgb commented 7 months ago

can you tell what is running on 10.101.4.1?

No, not at present. Waiting for our network expert to become available. But we don't see this problem in any other operator installed in our cluster. And we have quite a few....

yamatcha commented 7 months ago

@erikgb I found the same logs as the one you showed me in my environment. I'll look into it myself.

allenhaozi commented 4 months ago

same error but the webhook works fine

2024/05/17 12:23:03 http: TLS handshake error from 10.244.0.1:55642: EOF
2024/05/17 12:23:13 http: TLS handshake error from 10.244.0.1:49250: EOF
2024/05/17 12:23:13 http: TLS handshake error from 10.244.0.1:49252: EOF
2024/05/17 12:23:23 http: TLS handshake error from 10.244.0.1:41248: EOF
2024/05/17 12:23:23 http: TLS handshake error from 10.244.0.1:41244: EOF
2024/05/17 12:23:33 http: TLS handshake error from 10.244.0.1:60528: EOF
2024/05/17 12:23:33 http: TLS handshake error from 10.244.0.1:60530: EOF
2024/05/17 12:23:43 http: TLS handshake error from 10.244.0.1:46594: EOF
2024/05/17 12:23:43 http: TLS handshake error from 10.244.0.1:46592: EOF
2024/05/17 12:23:53 http: TLS handshake error from 10.244.0.1:33732: EOF
2024/05/17 12:23:53 http: TLS handshake error from 10.244.0.1:33730: EOF
2024/05/17 12:24:03 http: TLS handshake error from 10.244.0.1:55890: EOF
2024/05/17 12:24:03 http: TLS handshake error from 10.244.0.1:55882: EOF
2024/05/17 12:24:13 http: TLS handshake error from 10.244.0.1:46278: EOF
2024/05/17 12:24:13 http: TLS handshake error from 10.244.0.1:46290: EOF
2024/05/17 12:24:23 http: TLS handshake error from 10.244.0.1:55612: EOF
2024/05/17 12:24:23 http: TLS handshake error from 10.244.0.1:55614: EOF
2024/05/17 12:24:33 http: TLS handshake error from 10.244.0.1:34004: EOF