Issue with FARGATE 1.4.0

[vrusmanov]

Trying to use awsfirelens and send to multiple destinations. Created fluentbit config and uploaded to S3:

[SERVICE]
    Flush 1
    Grace 30

[OUTPUT]
    Name cloudwatch_logs
    Match *
    log_stream_name fluent-bit-cloudwatch
    log_group_name fluent-bit-cloudwatch
    region eu-central-1
    log_format json/emf
    auto_create_group true

[OUTPUT]
    Name        kafka
    Match       *
    Brokers     borker1:9092,broker2:9092,broker3:9092
    Topics      myTopic

And TaskDefintion using yml:

 TaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Properties:
      ExecutionRoleArn: !GetAtt 
        - ECSTaskExecutionRole
        - Arn
      TaskRoleArn: !GetAtt 
        - ECSTaskExecutionRole
        - Arn
      ContainerDefinitions:
        - Name: 'log_router'
          Image: '906394416424.dkr.ecr.eu-central-1.amazonaws.com/aws-for-fluent-bit:stable'
          Essential: true
          FirelensConfiguration:
            Type: "fluentbit"
            Options:
              config-file-type: "s3"
              config-file-value: "arn:aws:s3:::XXXXXX/fluentbit-service.conf"
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: 'log_router'
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: 'firelens'
        - Name: 'logger'
          Image: !Sub '${AWS::AccountId}.dkr.ecr.eu-central-1.amazonaws.com/random-logger:latest'
          Essential: true
          PortMappings:
            - HostPort: 80
              Protocol: tcp
              ContainerPort: 80
          LogConfiguration:
            LogDriver: awsfirelens
      RequiresCompatibilities:
        - FARGATE
      NetworkMode: awsvpc
      Cpu: '256'
      Memory: '512'
      Family: 'task-family'

In Cloudformaion during creation I see all the time message:

Resource handler returned message: "One or more of the requested capabilities are not supported. 
(Service: AmazonECS; Status Code: 400; Error Code: PlatformTaskDefinitionIncompatibilityException; 
Request ID: 3926503c-e2d8-4d02-b385-7619f4a7a5c3; Proxy: null)" 
(RequestToken: 32f71748-d116-65e1-185e-467067390ded, HandlerErrorCode: GeneralServiceException)

If I set PlatformVersion: 1.3.0 for Service - all works just fine. Seems like I can't use LogConfiguartion with just LogDriver: awsfirelens and no options. Tried to add also options for 'cloudwatch' was thinking that it will be added to 2 existing [OUTPUTS] in config file. But see exactly same issue. awsfirelens also works with "file" config-file-type. But clearly there is issue when config-file-type==s3 and LogConfigration has just LogDriver: awsfirelens but no options.

[rajpawar02]

Same issue I am getting while adding a custom key for stdout logs example link

It is working fine if we run the one-off task with platform version 1.3.0. but whenever we run a task with ECS service then by default service takes the 1.4.0 version and we are getting following error

Resource handler returned message: "One or more of the requested capabilities are not supported. 
(Service: AmazonECS; Status Code: 400; Error Code: PlatformTaskDefinitionIncompatibilityException; 
Request ID: 3926503c-e2d8-4d02-b385-7619f4a7a5c3; Proxy: null)" 
(RequestToken: 32f71748-d116-65e1-185e-467067390ded, HandlerErrorCode: GeneralServiceException)

[JohnPreston]

S3 config file for FireLens when using Fargate is not supported. You have to build a custom docker image with your config files built into it. From documentation anyway.

I am building a whole integration / wrapper around this which will use a sidecar that will first pull the file from "anywhere", put it in a path, and pass it to the fluent[bit] container from a shared volume.

[Bhavikpatel576]

This post has an example of how to set up a reference to using a config file