[exporter/awscloudwatchlogs] placeholders support for log_group_name and log_stream_name

[prashant-shahi]

Component(s)

exporter/awscloudwatchlogs

Is your feature request related to a problem? Please describe.

Yes.

Currently, all incoming log data are written to a single log group/stream by the exporter. Making it difficult to filter them in AWS Cloudwatch.

Describe the solution you'd like

Support for placeholders like:

• {ClusterName}
• {TaskId}
• {NodeName}
• {ContainerInstanceId}
• {TaskDefinitionFamily}

Similar to the format supported by awsemfexporter. With the values supported from OpenTelemetry ECS-like attributes.

Describe alternatives you've considered

No response

Additional context

No response

/cc @boostchicken, @bryan-aguilar, @rapphil

[github-actions[bot]]

Pinging code owners:

• exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bryan-aguilar]

I've thought about this in the past but don't have anything written down quite yet. I think there are a few different options to go with and I would seek something a bit more permissive than the original suggestion. I think one of the thoughts that I have had in the past is that the exporter could look for a specific resource attribute, if present use that. It gives more options for where that resource attributes is populated.

[sbarhouche]

@bryan-aguilar it could be similar to the approach followed by fluentbit for aws

[koenvandesande]

Being able to reference attributes from the log message itself for the logStreamName would be nice (to split it out again over containers/components). Or referencing something unique like the date or time that the collector started.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sbarhouche]

@bryan-aguilar any workaround for this ?

[bpalermo]

For simple use cases, at the very least we could start with support for templating based on attributes for the log stream name, since it's possible to use routingprocessor to route to different log groups. Although this is not ideal since would mean multiple exporters.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[using-system]

Same here ! We needs for this features because we want to have on collector for mulitple app and each app has his own log group name !

any update ?

[eOkadas]

It would seem this use case of sharing a collector and having it dispatch to different log groups is a widespread need. We have the same issue.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sbarhouche]

Pinging code owners:

exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil