Add support for egress logs in the Otelcollector client

[jriguera]

Right now ,there is only support for forwarding envelope metrics to an Otel collector.: https://github.com/cloudfoundry/loggregator-agent-release/blob/07c246750dd7fc79ed90f771652d5e06d5895895/src/pkg/otelcolclient/otelcolclient.go#L116-L123

What do you think about adding support for logs?

We are happy to contribute if a PR is welcome.

[ctlong]

🙌 We're very happy for support for logs to be added, if you'd like to contribute via PR!

[ctlong]

I forgot to mention before, but my preferred solution for getting logs into the OTel Collector would actually be to build a "loggregatorreceiver". See https://github.com/cloudfoundry/otel-collector-release/issues/25.

[jriguera]

Oh wow!! Tomorrow I will have a look at your implementation, but it sounds similar to what we are trying to achieve. We already have submitted a PR (accepted) to the OtelCollector codebase cloudfoundryreceiver to read the logs. We are going to do another PR to generate traces from RTR logs.

Because of the amount of logs we have, we would prefer trying an approach similar to what we have in K8S with K8Sobserver/receivercreator running on the local node and gathering (and filtering) app metrics and logs on the same cell (as close as possible where they are generated, in the same node), from there they would be sent to other endpoints. You can also have a look to this other proposal we have got accepted: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33618. Eventually, the idea is telling developers to use annotations in their apps, with those annotations we can:

1. If an app has an specific annotation, scrape all (local) containers to gather app metrics (/metrics endpoint)
2. If there is another annotation, accept also the application logs.
3. We can use other annotations to decide tenants or other features .

Apart of that there would be another set of OTEL collectors which would deal with:

4. RTR logs would be all transformed to traces (and then sampled)
5. Other logs and platform metrics would be collected with the current cloudfoundryreceiver

For the points 2 and 3 we would need another OTEL processor similar to K8Sattributes processor which would enrich telemetry taking the appid and querying the CF API for extra metadata (we will implement something basic in a second PR for the cfgardenobserver, but eventually all functionality should be in a "CFAttributes processor").

Right now our highest priority is dealing with app metrics and logs in a way which would be really simple for end users (adding annotations to apps) and check if this whole idea performs and works as expected, after that we would think about the "CFAttributes processor" which needs to be properly implemented because of the amount of CF API calls it can generate.