search

SigNoz / signoz-otel-collector

SigNoz distro for OpenTelemetry Collector
44 stars 39 forks source link

Add processing timings for each step in traces pipeline #318

Open srikanthccv opened 4 months ago

ankitnayan commented 3 months ago

Could we figure out anything from this?

srikanthccv commented 3 months ago

We now have clarity on the time it takes in each step https://signoz-team.slack.com/archives/C02SGR46RRR/p1714625576474649. We looked into whether async inserts help, the wait_for_async_insert setting decides whether we wait till ack. If we do not wait for ack we risk losing data because we won't be able to retry. If we wait for ACK, it doesn't make any difference from the write duration.

We need to parallelize these 4 calls https://github.com/SigNoz/signoz-otel-collector/blob/275a1e6873f7e1e06145c16014b35694e75c8fd1/exporter/clickhousetracesexporter/writer.go#L410-L436 and writeIndexBatch can further be broken to send in multiple batches.

ankitnayan commented 3 months ago

If we wait for ACK, it doesn't make any difference from the write duration.

Still it would lead to better batching when multiple otel-collectors are writing to clickhouse?

https://github.com/SigNoz/signoz/issues/5013

srikanthccv commented 3 months ago

Yes, and you already mentioned the main drawback of the async in the linked issue.