ECS Metadata - Githubissues

maggiedeuitch commented 4 years ago

Hi there, From what I understand, Firelens automatically adds some ECS metadata to the logs. I'm not using Firelens (we use Shippable and unfortunately it's incompatible), but have Fluent Bit set up and I have ECS Container Metadata enabled. My question is, how can we manually add ECS metadata? Any guidance would be much appreciated. Thank you.

EDIT: I know there's a record_modifier filter in your example configs, but the values would be static. Is there a way to have variables using the ECS metadata?


    Name record_modifier
    Match *
    Record ec2_instance_id i-01dce3798d7c17a58
    Record ecs_cluster furrlens
    Record ecs_task_arn arn:aws:ecs:ap-south-1:144718711470:task/737d73bf-8c6e-44f1-aa86-7b3ae3922011
    Record ecs_task_definition firelens-example-twitch-session:5```

sonofachamp commented 4 years ago

These values get populated via the ecs-agent when using Firelens. This metadata is available through the ECS Task Metadata Endpoint, but the Fluent Bit AWS Metadata filter does not yet support the ECS Task Metadata endpoint.

PettitWesley commented 4 years ago

This is on our long term roadmap; it is very possible that we will not get to it till next year though.

maggiedeuitch commented 4 years ago

Thank you for the update!

On Mon., Aug. 17, 2020, 03:52 Wesley Pettit, notifications@github.com wrote:

This is on our long term roadmap; it is very possible that we will not get to it till next year though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws/aws-for-fluent-bit/issues/62#issuecomment-674722738, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJQ7BFMUHBSST43NTVNQN6TSBDOURANCNFSM4PL3GBPQ .

PettitWesley commented 3 years ago

@maggiedeuitch This has been added specifically to the CW Logs plugin for naming log streams and groups: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit/pull/115

lcc3108 commented 3 years ago

I use below entrypoint.sh

# Work only ecs metadata enabled in firelens
AWS_CONFIG_FILE_PATH=/fluent-bit/etc/fluent-bit.conf
CUSTOM_CONFIG_FILE_PATH=/fluent-bit/alt/fluent-bit.conf

echo "aws config file path: $AWS_CONFIG_FILE_PATH"
echo "custom config file path: $CUSTOM_CONFIG_FILE_PATH"
cat $AWS_CONFIG_FILE_PATH
CONFIG_FILE=$(cat $AWS_CONFIG_FILE_PATH)

export ECS_CLUSTER=$(cat $AWS_CONFIG_FILE_PATH | grep ecs_cluster | awk '{print $3}')
export ECS_TASK_ARN=$(cat $AWS_CONFIG_FILE_PATH | grep ecs_task_arn | awk '{print $3}')
export ECS_TASK_DEFINITION=$(cat $AWS_CONFIG_FILE_PATH | grep ecs_task_definition | awk '{print $3}')
export ECS_CLUSTER_LOWER_CASE=$(echo $ECS_CLUSTER | tr '[:upper:]' '[:lower:]')
echo "--DEBUG---"
echo $ECS_CLUSTER $ECS_TASK_ARN $ECS_TASK_DEFINITION $ECS_CLUSTER_LOWER_CASE
echo "----------"

echo "---REPLACE_RESULT---"
cat $CUSTOM_CONFIG_FILE_PATH
echo "--------------------"

exec /fluent-bit/bin/fluent-bit -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c $CUSTOM_CONFIG_FILE_PATH

and fluent-bit config can use envrionment variable

[FILTER]
    Name record_modifier
    Match *
    Record ecs_cluster ${ECS_CLUSTER}
    Record ecs_task_arn ${ECS_TASK_ARN}
    Record ecs_task_definition ${ECS_TASK_DEFINITION}

If environmental variables are also available in Shippable, use the script above to parse.

PettitWesley commented 2 years ago

So we now support ECS metadata as env vars through this, which is the easiest way to get metadata when you are deploying Fluent Bit as a sidecar in ECS (aka FireLens): https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md

Regarding the built-in metadata for FireLens, that works like this: https://github.com/aws-samples/amazon-ecs-firelens-under-the-hood/blob/mainline/generated-configs/fluent-bit/generated_by_firelens.conf#L21

And unless what I said here is wrong this means with enable-ecs-log-metadata: true in your task def that any log source you add in a custom config file still should the metadata added: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

The diagram above shows the internal log processing pipeline for Fluent Bit. This ordering is always enforced, regardless of the order of sections in the configuration file. This is very convenient for FireLens users. It means that you can add log sources, filters, and outputs to your optional extra config file.

If the user provided Fluent Bit config file includes a log source, ECS Metadata will be added to the logs it ingests even though the source comes after the ECS Metadata transformer

aws / aws-for-fluent-bit

ECS Metadata #62