search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.43k stars 211 forks source link

AWS Firehose CloudWatchLogs not using internal timestamp from the event #1460

Open geronime opened 3 months ago

geronime commented 3 months ago

What's wrong?

The loki.source.awsfirehose has use_incoming_timestamp boolean attribute documented as:

Whether or not to use the timestamp received from the request.

I was trying to have the original CloudWatch Log event timestamp used - which is what I would expect by setting the use_incoming_timestamp attribute to true but that's simply not the case based on my understanding of the source code.

In aws_filrehose handler I see the timestamp is always passed through to the handleCloudwatchLogsRecord function and it is either the current timestamp or, in case of use_incoming_timestamp:true, the timestamp in the Firehose request.

Is there any reason why the actual event timestamp is never used, even though it is declared in the CloudwatchLogEvent structure?

Or maybe it would also make sense to have another configuration attribute which would imply using the source event timestamp?

Steps to reproduce

In any configuration of Grafana Alloy I don't think it's possible to get the original CloudWatch Log event timestamp when using the AWS Firehose as source because it's ignored - see linked code in the description.

System information

No response

Software version

No response

Configuration

No response

Logs

No response

github-actions[bot] commented 2 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!