search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.6k stars 2.45k forks source link

[Feature]: Support elasticsearch data stream #4708

Open pengweiqhca opened 1 year ago

pengweiqhca commented 1 year ago

Requirement

https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html

Problem

I has created a data stream index template, and I tried --es.use-aliases=true but not work.

Proposal

No response

Open questions

No response

jack78901 commented 9 months ago

Is this because the client library used for connecting to Elasticsearch is outdated? Namely, I see in the logs of version 1.54 of the collector that the client is olivere/elastic@v6.2.37. This client is from July 27th, 2021. The client is also not being updated anymore, it would appear. Namely, it was last updated to version 7.0.32 on Mar 19th, 2022. This is almost two years ago now. Elastic has its own go library now: https://github.com/elastic/go-elasticsearch.

tronda commented 9 months ago

I see that the OpenSearch project has the same feature-request.

JaredTan95 commented 3 months ago

I am currently working on this feature, but I prefer to implement it in jaeger v2

yurishkuro commented 3 months ago

@JaredTan95 +2 for v2, but what specifically is the difference? Our storage implementations are currently identical in v1 and v2 (with the exception of configuration)

JaredTan95 commented 3 months ago

@JaredTan95 +2 for v2, but what specifically is the difference? Our storage implementations are currently identical in v1 and v2 (with the exception of configuration)

Currently, if we want to use esILM policy, we need to create aliases and we need to create the xxx-00001 initial index before jaeger-colelctor writes data. This increases the difficulty of operation and maintenance. ES data stream shields users from the above problems. We no longer need to do some bootstrap works, and can directly write data to es through data stream. Scrolling of indexes and switching of aliases are managed within the data stream its-self.

yurishkuro commented 3 months ago
  1. that doesn't really address my question - all ES-specific details are hidden behind es.Factory, which is used by both Jaeger v1 and v2, so I don't see a difference
  2. Data streams also need to be set up once, just like index mappings. We want to do that on collector start-up, with some coordination between multiple collector instances (if creation operation are idempotent then it's less of an issue)