aws / aws-for-fluent-bit

The source of the amazon/aws-for-fluent-bit container image
Apache License 2.0
458 stars 135 forks source link

Unknown configuration property 'log_group_name' after upgrade to 2.30.0 #770

Closed MatthewDiPietroTR closed 11 months ago

MatthewDiPietroTR commented 11 months ago

Describe the question/issue

We recently upgraded from 2.19 to 2.30 and have started receiving this error when deploying our services:

[2023/11/13 17:33:10] [error] [config] datadog: unknown configuration property 'log_group_name'. The following properties are allowed: compress, apikey, dd_service, dd_source, dd_tags, proxy, include_tag_key, tag_key, dd_message_key, provider, and json_date_key.

Nothing about our configuration has changed, though we are also switching from amd64 to arm64. Prior to moving to 2.30.0, we were experiencing issues during service startup with SIGTERM errors that were reported in other issues to have been resolved in 2.30.0. Since making that change, the SIGTERMs have been resolved leaving us with this newest problem.

Configuration

I've provided what I can regarding the datadog/fluent bit configuration from the CF template:

"LogConfiguration": {
       "LogDriver": "awsfirelens",
       "Options": {
        "Name": "datadog",
        "Host": "http-intake.logs.datadoghq.com",
        "dd_service": "refapp",
        "dd_source": "arn:aws:ecr:us-east-1:{account}:repository/{source}",
        "dd_message_key": "log",
        "dd_tags": "env:dev",
        "log_group_name": "firelens-fluent-bit",
        "log_stream_prefix": "from-fluent-bit",
        "apikey": "__SECRET__",
        "tls": "on",
        "provider": "ecs"
       }
      }
...
 {
  "Environment": [
   {
    "Name": "ECS_FARGATE",
    "Value": "true"
   },
   {
    "Name": "DD_API_KEY",
    "Value": "__SECRET__"
   },
   {
    "Name": "DD_APM_ENABLED",
    "Value": "true"
   },
   {
    "Name": "DD_APM_NON_LOCAL_TRAFFIC",
    "Value": "true"
   },
   {
    "Name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
    "Value": "true"
   },
   {
    "Name": "DD_PROCESS_AGENT_ENABLED",
    "Value": "true"
   },
   {
    "Name": "DD_SYSTEM_PROBE_ENABLED",
    "Value": "true"
   },
   {
    "Name": "DD_TRACE_ANALYTICS_ENABLED",
    "Value": "true"
   },
   {
    "Name": "DD_EXPVAR_PORT",
    "Value": "6000"
   }
  ],
  "Essential": true,
  "Image": "datadog/agent",
  "Name": "datadog_agent",
  "PortMappings": [
   {
    "ContainerPort": 8126,
    "HostPort": 8126,
    "Protocol": "tcp"
   },
   {
    "ContainerPort": 8125,
    "HostPort": 8125,
    "Protocol": "udp"
   }
  ]
 },
 {
  "Essential": true,
  "FirelensConfiguration": {
   "Options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "file",
    "config-file-value": "/fluent-bit/configs/parse-json.conf"
   },
   "Type": "fluentbit"
  },
  "Image": {
   "Ref": "SsmParameterValueawsserviceawsforfluentbit2300C96584B6F00A464EAD1953AFF4B05118Parameter"
  },
  "LogConfiguration": {
   "LogDriver": "awslogs",
   "Options": {
    "awslogs-group": {
     "Ref": "ecspatternrefappwebapiv48955refappwebapiv48955taskdeflogrouterLogGroupBF9700B1"
    },
    "awslogs-stream-prefix": "firelens",
    "awslogs-region": "us-east-1"
   }
  },
  "Name": "log_router"
 }

Fluent Bit Version Info

2.30.0

Cluster Details

ECS on Fargate

MatthewDiPietroTR commented 11 months ago

I think I found the problem here. It seems there is more stringent configuration validation occurring, and the log_group_name and log_stream_prefix properties included are not part of the datadog fluent bit configuration. https://docs.datadoghq.com/integrations/fluentbit/#configuration-parameters

Unfortunately, removing those hasn't changed the end result for us--we're still having problems getting our services deployed, but I'm not sure if fluent bit is the failure point any longer, so I will close this issue.