Research: Why does xk6-distributed-tracing cause excess load on Tempo?

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.

https://grafana.com/oss/tempo/

GNU Affero General Public License v3.0

3.76k stars 488 forks source link

Research: Why does xk6-distributed-tracing cause excess load on Tempo? #2891

Open joe-elliott opened 10 months ago

joe-elliott commented 10 months ago

A recent discussion found that, for unknown reasons, xk6-client-tracing generates extra load on Tempo. This caused two operators to see OOMing ingesters at fairly low load.

We should research why this is occuring. xk6-client-tracing may employ a particular write pattern that exposes a memory issue not seen with normal trafffic. If so, it would be nice to clean this up.

Thanks to @itheodoro and @dhanvsagar for the reports and research.

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.