search

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.92k stars 510 forks source link

TIME_WAIT Issue inside pod for tempo-distributed #3883

Open raghav20241 opened 2 months ago

raghav20241 commented 2 months ago

Describe the bug

I have deployed tempo and tempo-distributed using helm in same cluster with demo opentel service, but I am able to get traces for tempo without any issue for data source but for tempo-distributed getting TIME_WAIT error inside the pod and not getting any traces (0 series ) or connect to demo service. **To Reproduce** Steps to reproduce the behavior: 1. Start Tempo (SHA or version) 2.5.0. 2. Perform Operations (Read/Write/Others) **Expected behavior** Please suggest conf for TIME_WAIT Issue and need to established connection between opentel demo service with tempo-distributed. **Environment:** - Infrastructure: [e.g., Kubernetes, bare-metal, laptop] ---kubernetes - Deployment tool: [e.g., helm, jsonnet]---helm **Additional Context** plz find pod with netstat image
joe-elliott commented 2 months ago

you seem to be struggling to get the most basic elements of tempo to work. this is quite difficult to debug with limited information and requires an operator to debug simple networking and other config elements.

in the past i've used these steps to startup the distributed helm chart with a load generator:

https://github.com/grafana/tempo/tree/main/example/helm

raghav20241 commented 2 months ago

@joe-elliott tempo is working fine with opentel demo services able to get traces but same for tempo-distributed i am facing issue with traces not able to get traces ,can you share any end-to-end document for grafana tempo-distributed with demo service for testing.

github-actions[bot] commented 3 days ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.