jaegertracing / jaeger-clickhouse

Jaeger ClickHouse storage plugin implementation
Apache License 2.0
236 stars 50 forks source link

Too many jaeger-query WriteSpan traces written. #84

Closed EinKrebs closed 2 years ago

EinKrebs commented 2 years ago

Describe the bug Plugin writes very much traces(service='jaeger-query', operation='/jaeger.storage.v1.SpanWriterPlugin/WriteSpan'), even where there's no else spans to write. Count of traces per tick on default settings is at least 20. After writing spans that are not jaeger-query, circa several thousands of spans every tick can last for very long, even without writing any spans to jaeger.

To Reproduce Steps to reproduce the behavior:

  1. Start docker image of clickhouse-server
  2. Start Jaeger with jaeger-clickhouse on default settings
  3. Generate a little spans(e.g. with tracegen or HotR.O.D.)
  4. See huge amount of spans being written every timer tick.
  5. If to check what are these spans, there's almost only jaeger-query/WriteSpan.
  6. Query done after a long time after tracegen finished work:
    
    SELECT count()
    FROM jaeger_index_local 
    WHERE (service = 'jaeger-query') AND (operation = '/jaeger.storage.v1.SpanWriterPlugin/WriteSpan') AND (timestamp >= (now() - toIntervalMinute(1)))

┌─count()─┐ │ 15329 │ └─────────┘



**Expected behavior**
No/very little of such spans.

**Version (please complete the following information):**
 - OS:  Linux
 - Jaeger version: Jaeger v1.24, jaeger-clickhouse v0.7.0
 - Deployment: bare metal

**What troubleshooting steps did you try?**
Didn't find any of the info about such problem.
EinKrebs commented 2 years ago

The solution is not using jaeger-all-in-one.

the-elder commented 10 months ago

The solution is not using jaeger-all-in-one.

Is there a way for HA deployments?i cannot find any way but all-in-one.