Open nikhilo opened 1 year ago
@nikhilo - hopefully you found your way through. Like you, I was looking to implement additional log streams by using templates. I found out that the cloudwatch_logs
documentation mentions log_stream_template. However, using $(ec2_instance_id)
or other documented syntax for the previous plugin cloudwatch
did not match the usage in cloudwatch_logs
, which uses record-accessor.
Long story short, I got it to work with an OUTPUT like so:
[OUTPUT]
Name cloudwatch_logs
Match ${APPLICATION_NAME}-firelens*
region ${AWS_REGION}
log_group_name ${LOG_GROUP_BASE_NAME}/var/jenkins_home/stdout
log_stream_name other
log_stream_template $ec2_instance_id
log_key log # Only ship the log message, without log metadata
auto_create_group false
retry_limit 2
With ${XYZ}
being environment variables.
Also found out that the variables I could use are (in the case of an ECS task running on EC2):
{
"source": "stdout",
"container_id": "---",
"container_name": "---",
"ec2_instance_id": "---",
"ecs_cluster": "---",
"ecs_task_arn": "---",
"ecs_task_definition": "---"
}
Hope this helps.
@nikhilo there are two templating features in two different plugins.
cloudwatch_logs
templating is documented here: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch#log-stream-and-group-name-templating-using-record_accessor-syntax
cloudwatch
templating is documented here: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names
Each feature is different.
@ojacques Your example works because you are using the built-in metadata keys provided by FireLens:
We have this new image build with an init script: https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md#how-init-process-works
It can set env vars with ECS Metadata. You can reference env vars in the config. This is an alternative to templating that is more efficient since templating has a CPU overhead.
For example:
[OUTPUT]
Name cloudwatch_logs
Match ${APPLICATION_NAME}-firelens*
region ${AWS_REGION}
log_group_name ${ECS_CLUSTER}
log_stream_name ${ECS_FAMILY}.${ECS_TASK_ID}
log_key log # Only ship the log message, without log metadata
auto_create_group true
retry_limit 2
@PettitWesley The init image worked as expected. Thank you! 🎉
Keeping this open as guidance/common question.
Possible I have the same issue (and I am also confused on where to report it). I did create an issue at https://github.com/aws/eks-charts/issues/927 but possible here is the better place.
Small summary:
set {
name = "cloudWatchLogs.logGroupTemplate"
value = "/aws/eks/${local.cluster_name}/$kubernetes['labels']['app']"
}
does not work (accessor error in the log)
but the following does work:
set {
name = "cloudWatchLogs.logGroupTemplate"
value = "/aws/eks/${local.cluster_name}/$kubernetes['namespace_name']"
}
If anyone can point me in the right direction that would be great.
I'm not sure what's going wrong @Mattie112, see my answer here: https://github.com/aws/eks-charts/issues/927#issuecomment-1478332478
This is regarding the cloudwatch_logs plugin. We have been facing an issue where several ECS tasks are writing to the same log stream, and might be getting throttled by CloudWatch. I tried to set the
log_stream_template
to$(ecs_task_id)
and$(uuid)
. But none of them seemed to work. I got following errors,log_stream_template
config ?