Route the same log-event to multiple outputs (Cloudwatch and other)

[Oegma2]

I am trying to route my AWS Fargate container app's logs to two outputs - that being in our case Cloudwatch and Datadog. So the same log event in json must be routed at the same time to 2 outputs

The current setup is using ECS fargate containers with firelens enabled for streaming the logs to datadog. The single [output] in this case Datadog is easy. Add the config in your task-definition and routes the logs. The trick comes in when you want to route that same log-event to cloudwatch and datadog

Would we be able to route the same log-file to two different outputs?

[PettitWesley]

@Oegma2 That is possible! We have FireLens examples in this repo, including a multi-destination example: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/master/examples/fluent-bit/send-to-multiple-destinations

[PettitWesley]

I should also note that right now, on Fargate, you can't use the S3 config injection option, you can only use the file type (config file is baked into your Fluent Bit image).

[Oegma2]

@PettitWesley - Thanks for the head's up on S3 - I tried that actually last week and thought I did something wrong - the fargate containers with the fluentbit sidecar kept on crashing using the S3 custom config even after setting the role to full S3 access as test - What I will do is to change the task-definition to file and use my custom file. Got the aws docker file running with the new mutli-output config using cloudwatch and datadog in the config-file.

So if my config look likes this and the app-container firelens config is "empty", meaning not specific any "output", the logs would be catch by cloudwatch output & datadog output?

[PettitWesley]

@Oegma2 As shown here, your app container must use the awsfirelens log driver without any options: https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/master/examples/fluent-bit/send-to-multiple-destinations/task-definition.json#L33

Also, looking at your config, I am not sure about the line: log_format json. That config option adds a header to the request, AFAIK EMF is the only use case: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#plugin-options

We should update that description- the string the user provides goes in the x-amzn-logs-format HTTP header in requests.

[Oegma2]

@PettitWesley - I had to remove the log_format json and the [input] from the conf file above. I also had to update my Dockerfile to place the config not in the normal /etc/fluent-bit/ but in the plain root folder /custom.conf and update the task definition to use my /custom.conf as File

Once I've done those changes, the fluentbit started routing the logs to cloudwatch and datadog

Not sure why the [input] section cause problems... and also odd that I can't update the /etc/fluentbit/...

Will continue testing today, but the good news is - it works! You can use different multi-output options like cloudwatch and datadog for single log-event :)

[PettitWesley]

@Oegma2 Short form answers:

• FireLens already adds that input
• /etc/fluentbit/ is used by FireLens for the generated config

Long answer, this blog explains how FireLens works: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

[hencrice]

@Oegma2 Looks like the problem was resolved. Closing this now. Feel free to re-open if you have further questions.

[kmajic]

@Oegma2 and maybe @PettitWesley I'm also attempting to send logs to multiple different locations, but since this case isn't very well documented - could you please provide all the steps you took, along with the configuration files that you got to work in the end (Dockerfile, custom-fluent-bit.conf, etc.)? Any help will be appreciated!

I'm currently building a custom image based off AWS-fluent-bit, appending custom.conf, updating Fargate task defitnition to use it, but I still can't get the logs to route to neither Cloudwatch nor Datadog. Here's my files:

=== Dockerfile ===
FROM amazon/aws-for-fluent-bit:latest
ADD custom.conf /custom.conf

=== custom.conf ===
[OUTPUT]
    Name cloudwatch
    Match *
    region eu-central-1
    log_group_name test
    log_stream_prefix from-fluent-bit-
    auto_create_group true
    log_key log

[OUTPUT]
    Name datadog
    Match *
    Host http-intake.logs.datadoghq.eu
    TLS on
    compress gzip
    dd_service firelens-test
    dd_source firelens-test
    dd_message_key log
    provider ecs
    apikey <API_KEY>

=== task def (I'm using Terraform) ===
resource "aws_ecs_task_definition" "service-definition" {
  family                   = var.service-name
  cpu                      = var.fargate-cpu
  memory                   = var.fargate-memory
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_full_role.arn
  container_definitions = <<DEFINITION
[
  {
    "image": "${var.image-name}",
    "name": "${var.service-name}",
    "environment": [
      {
        "name": "ASPNETCORE_ENVIRONMENT",
        "value": ${var.env}
      }
    ],
    "logConfiguration": {
      "logDriver": "awsfirelens"
    },
    "logConfiguration": {
      "logDriver": "awsfirelens"
    },
    "networkMode": "awsvpc",
    "portMappings": [
      {
        "containerPort": 80,
        "hostPort": 80,
        "protocol": "tcp"
      }
    ]
  },
  {
    "image": "${var.fluent-image}",
    "name": "fluent",
    "firelensConfiguration": {
      "type": "fluentbit",
      "options": {
        "enable-ecs-log-metadata": "true",
        "config-file-type": "file",
        "config-file-value": "/custom.conf"
      }
    },
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
          "awslogs-group": "test",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "fluent"
      }
    }
  }
]
DEFINITION
}

[PettitWesley]

@kmajic Check out both of these examples:

https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/send-to-multiple-destinations

I think the first one is what you're looking for, your config looks fine at a glance.

Let me know if you need more help.

[kmajic]

@PettitWesley I think I may have found the problem. I'm using two outputs - Cloudwatch and Datadog. None was working and I got the Cloudwatch to work by adding both roles: "task_role_arn" and "execution_role_arn". However, there's still no output to Datadog. I've connected to the AWS FluentBit container and I'm seeing three output plugins in /fluent-bit: cloudwatch.so, firehose.so and kineses.so.

Apparently the Datadog plugin is missing. Is there a simpler way of adding it than building from source?

[PettitWesley]

@kmajic The datadog plugin is there. The 3 .so files you see are external plugins- the aws plugins are external. The datadog plugin is in the core fluent bit binary.

[PettitWesley]

What do the logs show for the fluent bit container?

[kmajic]

When I revert back to the AWS Fluent bit container, every time I generate a log entry on my app container, the Fluent bit picks up on it and generates this:

[2020/08/20 06:54:28] [ info] [output:datadog:datadog.2] https://http-intake.logs.datadoghq.eu, port=443, HTTP status=200 payload={}

So it looks like it's not sending anything over. I tried outputting JSON: {"name":"Log info","time":"2020-08-20T07:07:43.9927115Z"}

and a regular string: 637335052884174726

but there's no difference, the payload stays empty.

[PettitWesley]

@kmajic That log messages comes from here, the empty json is the response from datadog, not the request: https://github.com/fluent/fluent-bit/blob/master/plugins/out_datadog/datadog.c#L379

[PettitWesley]

Whatever is wrong here- it must be something with your config/setup/image. As a sanity check, I just ran fluent bit locally and made it send to both CW and DataDog- it worked. @kmajic

[PettitWesley]

I'm currently building a custom image based off AWS-fluent-bit

Can you add your dockerfile?

Also- your terraform task def looks a bit weird- you have two logConfigurations.

[kmajic]

I've pasted it in the code dump above, and the TF code is a pasting error that I didn't catch. :D Can you please paste your configuration? Here's my current iteration:

### Dockerfile ###
FROM amazon/aws-for-fluent-bit:latest
ADD custom.conf /custom.conf

### custom.conf ###
[OUTPUT]
    Name cloudwatch
    Match **
    region eu-central-1
    log_group_name test
    log_stream_prefix -
    auto_create_group true
    log_key log

[OUTPUT]
    Name datadog
    Match **
    Host http-intake.logs.datadoghq.eu
    TLS on
    apikey ASDASD

### TF plan ###
resource "aws_ecs_task_definition" "service-definition" {
  family                   = var.service-name
  cpu                      = var.fargate-cpu
  memory                   = var.fargate-memory
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = aws_iam_role.ecs_full_role.arn
  task_role_arn            = aws_iam_role.ecs_full_role.arn
  container_definitions = <<DEFINITION
[
  {
    "image": "${var.image-name}",
    "name": "${var.service-name}",
    "environment": [
      {
        "name": "ASPNETCORE_ENVIRONMENT",
        "value": ${var.env}
      }
    ],
    "logConfiguration": {
      "logDriver": "awsfirelens"
    },
    "networkMode": "awsvpc",
    "portMappings": [
      {
        "containerPort": 80,
        "hostPort": 80,
        "protocol": "tcp"
      }
    ]
  },
  {
    "image": "${var.fluent-image}",
    "name": "fluent",
    "firelensConfiguration": {
      "type": "fluentbit",
      "options": {
        "enable-ecs-log-metadata": "false",
        "config-file-type": "file",
        "config-file-value": "/custom.conf"
      }
    },
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
          "awslogs-group": "test",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "fluent"
      }
    }
  }
]
DEFINITION
}

[kmajic]

@PettitWesley I solved it! Turns out the setup was correct, like you said. However, without specifying the (correct) DataDog tags, the log entries wouldn't show up on any searches in Datadog. I finally found them by disabling all filters and manually looking for sources. Once I setup the correct DD tags, the entries were easily identified. Thank you for all the help, and I'll leave the code samples up, to maybe make someone else's life easier. Cheers!

[hossain-rayhan]

@kmajic Thanks for the update. We are closing this.

[BartekKloza]

I'm running a python django app on ECS Fargate and recently I had to change my setup to send logs from containers to both DataDog and CloudWatch. After using a similiar config to the one posted by @kmajic, my logs were not being parsed correctly and DataDog failed match them with traces, etc. so I wanted to share a custom.conf that worked for me:

[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Flush 1
    Grace 30

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json
    Reserve_Data True

[OUTPUT]
    Name datadog
    region eu-central-1
    apikey DD_API_KEY_PLACEHOLDER
    dd_service DD_SERVICE_PLACEHOLDER
    dd_source python
    dd_tags project:MY_PROJECT,env:DD_ENV_PLACEHOLDER
    provider ecs
    Host http-intake.logs.datadoghq.eu
    Match *

[OUTPUT]
    Name cloudwatch
    region eu-central-1
    log_group_name LogGroupName
    log_stream_prefix LogGroupPrefix
    auto_create_group false
    Match *

[PettitWesley]

@BartekKloza Please note that this plugin is semi deprecated and no longer recommended for most use cases: https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#new-higher-performance-core-fluent-bit-plugin

[nithin-kumar]

https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/master/examples/fluent-bit/send-to-multiple-destinations

@PettitWesley Does the s3 output plugin support the same behavior? Send the same logs to multiple s3 accounts?

[PettitWesley]

@nithin-kumar Fluent Bit supports multiple output definitions, for every output

[hmaddisett]

thanks for the links in above comments. One approach is to use file type and pass an custom config with multiple outputs i.e one for CW and another for Datadog or any other logging tool.

In my case, I need to use the custom docker image with custom config for different regions and services. So there will be a change in region, log group name, stream prefix etc. If I add the variables under options in LogConfiguration will it work? or do I need to pass them as environment variables to fluent bit container

[hmaddisett]

@PettitWesley is there an example to use Grafana Loki has another output along with Cloudwatch? If yes then which image should I use for that.

[PettitWesley]

@hmaddisett You can use env vars in the config, but they need to set as real env vars in the container definition.

The logConfiguration.options is just pasted directly into the generated Fluent bit config by FireLens, so you can put anything in them that works in Fluent Bit config include ${VAR} syntax: https://aws.amazon.com/blogs/containers/under-the-hood-firelens-for-amazon-ecs-tasks/

We now have an init tag which makes it easy to configure FLB with multiple config files pulled from S3: https://github.com/aws-samples/amazon-ecs-firelens-examples#aws-for-fluent-bit-init-tag-examples

is there an example to use Grafana Loki has another output along with Cloudwatch?

https://docs.fluentbit.io/manual/pipeline/outputs/loki

Do you mean the built-in loki output? If yes, then all built-in outputs are included in our distro. If its some custom third party plugin, then you'd need to build your own image with it using our Makefile/Dockerfile as a reference: https://github.com/aws/aws-for-fluent-bit/blob/mainline/Makefile

[hmaddisett]

@PettitWesley Thanks for the response. yes I was asking about built-in loki output plugin. If its already included in aws-for-fluent-bit image then I will try to implement and see. Thanks.

I hope the dockerfile with below will run the custom config with outputs for CW and Loki CMD ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/alt/fluent-bit.conf"]

[PettitWesley]

@hmaddisett why do you need a custom entrypoint?

[hmaddisett]

@PettitWesley for the fluent bit to pick the custom config file instead of the default one. How fluent bit will know that the custom outputs mentioned in /fluent-bit/alt/fluent-bit.conf needs to be processed instead of the default one in "/fluent-bit/etc/fluent-bit.conf"

[PettitWesley]

@hmaddisett your use case is not clear to me, if you want help I need full details. Why can't you customize the content in /fluent-bit/etc/fluent-bit.conf?

Are you following: https://aws.amazon.com/blogs/opensource/splitting-application-logs-multiple-streams-fluent/

[hmaddisett]

@PettitWesley My case is that I am trying to use fluent bit container as a sidecar to the application container to send application logs to LOKI and cloudwatch log via fluent bit. I am using ECS Fargate here

I have to send logs to grafana loki and cloudwatch in my case so the fluent bit config should have multiple outputs configured ( one for loki and another one for cloudwatch).

will I be able to customize the default fluent-bit.conf with multiple outputs? the output variables will be dynamic and the values are passed via container env vars

[PettitWesley]

@hmaddisett On ECS Fargate with FireLens this would be the recommended solution: https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/mainline/examples/fluent-bit/config-file-type-file

Use a custom config file. Or use this project to download config from S3: https://github.com/aws/aws-for-fluent-bit/blob/mainline/use_cases/init-process-for-fluent-bit/README.md

You can have both outputs match the same tag: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#firelens-tag-and-match-pattern-and-generated-config

[jared-christensen]

Do you have a CDK example?