OTEL sidecar collector documentation

[falgofrancis]

We have a use case to collect logs from a cluster and send logs to different log aggregators based on namespace prefix. We did not find any option to do it via daemonset deployment of otel collector. We are currently using vector for it.

Hence we are trying to use sidcar pattern to collect logs, metrics and traces. The documentation have decent amount of information about sending traces via sidecar. However documentation is not good around sending logs and metrics using sidecar. Is there any roadmap/plans to improve documentation?

We found 2 options to collect logs via sidecar.

One option is to route all application logs to a director which is mounted in the application and sidecar log collection containers. We still want to use kubectl logs option to view logs. However this approach needs changes to the all applications in the cluster.

Other option is to allow sidecar to be able to read log files from the host using hostpath mount. There are security concerns around it. We know that the log collector deamonsets currently do this, however if we do not want to allow all pods to do it.

Kubernetes Attributes Processor cannot be used when running as sidecar. Documentations do not suggest any other ways to add attributes to it as well.

[TylerHelmuth]

I am not totally clear on what the ask is. Are you wondering which components work int he collector when run as a side car or best practices for how to use the collector as a side car?

[falgofrancis]

Best practices on how to use the collector as a side car for logs and metrics?

[TylerHelmuth]

I wonder if this would be a good doc to add to https://opentelemetry.io/docs/collector/deployment/ as a third deployment pattern.

/cc @open-telemetry/docs-approvers

[cartermp]

@TylerHelmuth or others, can we initiate an issue transfer?

@falgofrancis this would be good docs to have, since this is a good topic to cover:

Best practices on how to use the collector as a side car for logs and metrics?