search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.61k stars 2.45k forks source link

Agent's gRPC client should disable TXT DNS record lookup for nonexistent gRPC service configuration #2029

Closed mindw closed 4 months ago

mindw commented 4 years ago

Requirement - what kind of business use case are you trying to solve?

jaeger (1.16.0) deployed in k8s 1.16 (observed at least since jaeger 1.14 and k8s 1.14).

Problem - what in Jaeger blocks you from solving the requirement?

When jaeger-agent uses its gRPC client, it issues TXT DNS queries against the in-cluster DNS pods (CoreDNS in my case), repeatedly looking up these records to find service configuration. CoreDNS does not serve these records, and the lookups prompt log messages complaining about the client's mistaken assumptions (fixed recently in coredns/coredns#2291, addressing coredns/coredns#2290).

similar issue: uswitch/kiam#204

coredns-74cddc8fd4-vddrk coredns [INFO] 10.1.10.0:40363 - 47579 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000974516s
coredns-74cddc8fd4-vddrk coredns [INFO] 10.1.8.0:44150 - 52701 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001126075s
coredns-74cddc8fd4-vddrk coredns [INFO] 10.0.3.151:51981 - 3000 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.00138541s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.6.0:52901 - 26218 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000851205s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.7.0:47305 - 21513 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000896296s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.12.0:32893 - 16834 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001108178s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.21.0:57339 - 30016 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001045959s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.9.0:48467 - 33837 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001193555s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.3.0:60879 - 31903 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001211577s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.0.4.155:42152 - 63691 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000987131s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.8.0:57373 - 11960 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001033072s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.5.0:47958 - 38588 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000940131s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.6.0:32778 - 46664 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001051965s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.7.0:42275 - 13959 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.001029183s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.12.0:35936 - 36668 "SRV IN _grpclb._tcp.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.00197792s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.12.0:42617 - 16882 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000957578s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.9.0:39829 - 45116 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000995002s
coredns-74cddc8fd4-2rc7x coredns [INFO] 10.1.3.0:33149 - 11875 "TXT IN _grpc_config.jaeger-collector.eu-west-1.compute.internal. udp 85 false 1232" NXDOMAIN qr,rd,ra 74 0.000979289s

Proposal - what do you suggest to solve the problem or improve the existing situation?

The Go gRPC client library used by kiam gained a new option in grpc/grpc-go#2010, addressing grpc/grpc-go#1998: grpc.WithDisableServiceConfig.

potential solution: uswitch/kiam#205

Any open questions to address

jpkrohling commented 4 years ago

Would you like to give it a try? The tricky part is that we want to disable this when Jaeger is running in Kubernetes, but we probably still want this enabled for other cases.

My personal preference would be to keep this as it is, and just suppress the message. Not sure this alone deserves a new flag...

mindw commented 4 years ago

@jpkrohling Thanks for looking at this! Alas my golang skills are non existent - but maybe common sense and some trial and error will be enough 🤔

sanjeev98kumar commented 2 years ago

@jpkrohling ,i want to work on this issue. could you assign me.

albertteoh commented 2 years ago

@jpkrohling ,i want to work on this issue. could you assign me.

Sure, I've assigned this issue to you.

vishal-chdhry commented 2 years ago

If this is inactive, may I pick this up?

mindw commented 2 years ago

Go ahead I won't get it anytime soon.

yurishkuro commented 4 months ago

nobody stepped up to fix it, and jaeger-agent is deprecated anyway, so closing