apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.26k stars 3.59k forks source link

[Bug] proxy DNS UDP timed out after 5000 milliseconds error in kubernetes #23588

Closed graysonzeng closed 5 days ago

graysonzeng commented 1 week ago

Search before asking

Read release policy

Version

pulsar 3.3.0 kubernetes 1.18.4

Minimal reproduce step

online error, reproduction is not easy

What did you expect to see?

consume success

What did you see instead?

java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'pulsar-uqhf-178-pulsar-cluster-broker-1.pulsar-uqhf-178-pulsar-cluster-broker-headless.pulsar.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]
Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'pulsar-uqhf-178-pulsar-cluster-broker-1.pulsar-uqhf-178-pulsar-cluster-broker-headless.pulsar.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]
Caused by: io.netty.resolver.dns.DnsNameResolverTimeoutException: [37005: /9.165.255.254:53] DefaultDnsQuestion(pulsar-uqhf-178-pulsar-cluster-broker-1.pulsar-uqhf-178-pulsar-cluster-broker-headless.pulsar.svc.cluster.local. IN A) query '37005' via UDP timed out after 5000 milliseconds (no stack trace available)
2024-10-20T15:18:34,102+0800 [pulsar-proxy-io-2-5] INFO  org.apache.pulsar.proxy.server.ProxyConnection - [/21.10.52.200:38756] Connection closed

this error happen in proxy and then some consumers will disconnect and close the client

Anything else?

With the increase in traffic, we have recently discovered that this problem will occasionally occur in the cluster. And there are many consumers using reader mode in our cluster.

Are you willing to submit a PR?