Dynamic selection of log_group_name and log_stream_name in aws cloudwatch logs exporter

sorobon commented 8 months ago

Component(s)

exporter/awscloudwatchlogs

Is your feature request related to a problem? Please describe.

Duplicate with https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14554 We are running the opentelemetry collector to send logs from our edge systems to a central opentelemetry gateway service. From the gateway we'd like to write logs to cloudwatch logs in different log groups and streams based on log resources.

Describe the solution you'd like

We'd like to be able to set the log_group_name and log_stream_name dynamically based on fields from the log event:

awscloudwatchlogs:
    log_group_name: "{log_file_name}"
    log_stream_name: "{resource.hostname}"

Describe alternatives you've considered

No response

Additional context

No response

github-actions[bot] commented 8 months ago

Pinging code owners:

exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 6 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

sorobon commented 6 months ago

Any update?

abhishekmahajan0709222 commented 5 months ago

Any update, we are also looking for same feature

bpalermo commented 5 months ago

Duplicate https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/30773?

github-actions[bot] commented 3 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/awscloudwatchlogs: @boostchicken @bryan-aguilar @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

using-system commented 2 months ago

Same here ! We needs for this features because we want to have on collector for mulitple app and each app has his own log group name !

using-system commented 2 months ago

Any update ? @boostchicken @bryan-aguilar @rapphil

warsiafaz commented 2 months ago

Any update we are also looking for same solution . At least log streams should be dynamic . Fluent bit provides that solution , but for awscloudwatchlogexporter log_stream is a requirement and not dynamic .

atoulme commented 1 month ago

@boostchicken @bryan-aguilar @rapphil please weigh in.

lasith-kg commented 3 weeks ago

Same here! Right now the cloudwatchlogsexporter forces all logs to go to a stream of a fixed name. There is no inherent limit to a size of a stream. However, I imagine there are performance benefits to introducing segmentation between streams.

By minimum, if a log_stream is not provided, then it should default to a dynamic value like a UUID

open-telemetry / opentelemetry-collector-contrib