Validate new OTLP log setup - Githubissues

kyma-project / telemetry-manager

Manager for the Kyma telemetry module

https://kyma-project.io/#/telemetry-manager/user/README

Apache License 2.0

5 stars 23 forks source link

Validate new OTLP log setup #1631

Open a-thaler opened 4 days ago

a-thaler commented 4 days ago

Description Review and collect the result of the PoCs done for https://github.com/kyma-project/telemetry-manager/issues/556 and come up with a final config for the OTLP based log agent. The agent should:

tail logs from the container runtime
map the content to OTLP
if applicable parse the JSON payload
send the data synchronuous to the log gateway
will pause tailing in case of gateway refusals

With that setup, perform a load test to see that the maximum ingestion rate of a Cloud Logging instance can be achieved.

Document the final configmap and document the load test results

Considerations:

An observed maximum output per overall fluentbit DaemonSet observed was 60 MB/s or 44K records/s (where the backend could not stand the load)
An observed maximum per fluentbit instance was 4,4MB/s or 15K records/s
Cloud Logging can handle in fluentbit format 2,5K logs/s in standard or avg. 15K logs/s (max 25K logs/s) in large
Cloud Logging can handle in OTLP format 10K logs/s in standard or 30K logs/s in large