search

SumoLogic / sumologic-collector-docker

A Sumo Logic collector for Docker.
Apache License 2.0
69 stars 55 forks source link

Issues when deploying collector as a sidecar in AWS ECS Fargate #118

Open frossi85 opened 11 months ago

frossi85 commented 11 months ago

I am trying to run the collector as a sidecar of a main service using AWS ECS Fargate, whch works correctly when deploying it without the collectot, but I am not able to do it due to some errors that are not clear to me.

I created a custom Dockerfile:

FROM sumologic/collector
ADD file-sumo-sources.json /etc/sumo-sources.json

file-sumo-sources.json:

{
  "api.version": "v1",
  "sources": [
    {
      "sourceType" : "LocalFile",
      "name": "my-service-local-log",
      "pathExpression": "/var/lib/docker/containers/**/*.log",
      "multilineProcessingEnabled": false,
      "automaticDateParsing": true,
      "forceTimeZone": false,
      "category": "file-docker"
    },
    {
        "name": "my-service-local-stats",
        "category": "docker",
        "automaticDateParsing": true,
        "forceTimeZone": false,
        "cutoffTimestamp": 0,
        "uri": "unix:///var/run/docker.sock",
        "specifiedContainers": ["my-service", "logging-service"],
        "allContainers": false,
        "multilineProcessingEnabled": false,
        "certPath": "",
        "sourceType": "DockerStats",
        "pollInterval": 5000
    }
    ]
}

It works locally using docker-compose and from ECS it creates the collector in sumologic dashboard but when I deploy it to ECS it fails with messages like this:

Full log:

2023-12-20T17:15:18.376-03:00 Running SumoLogic Collector...
2023-12-20T17:15:36.734-03:00 wrapper | JVM exited unexpectedly.
2023-12-20T17:15:43.249-03:00 wrapper | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
2023-12-20T17:15:43.249-03:00 wrapper | --> Wrapper Started as Console
2023-12-20T17:15:43.250-03:00 �]0;Sumo Logic Collector�wrapper | Java Service Wrapper Standard Edition 64-bit 3.5.49
2023-12-20T17:15:43.250-03:00 wrapper | Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
2023-12-20T17:15:43.250-03:00 wrapper | http://wrapper.tanukisoftware.com
2023-12-20T17:15:43.250-03:00 wrapper | Licensed to Sumo Logic Inc. for Collector
2023-12-20T17:15:43.250-03:00 wrapper \
2023-12-20T17:15:43.250-03:00 wrapper | Launching a JVM...
2023-12-20T17:15:43.250-03:00 jvm 1 | WrapperManager: Initializing...
2023-12-20T17:15:43.250-03:00 jvm 1 | . . . . . . . . .
2023-12-20T17:15:43.250-03:00 jvm 1 | .+'|=|+. .+'\| \|+. .+'|=|+.=\|+. .+'|=|`+.
2023-12-20T17:15:43.250-03:00 jvm 1 | | | +.\| \| \| \| \| \| \|+ | `+ | | | | \
2023-12-20T17:15:43.250-03:00 jvm 1 | | | . | | | | | | | | | | | | | \
2023-12-20T17:15:43.250-03:00 jvm 1 | +.\|=\|+. | | | | | | | | | | | | | \
2023-12-20T17:15:43.250-03:00 jvm 1 | . | | | | | | | | | | | | | | | \
2023-12-20T17:15:43.250-03:00 jvm 1 | |`+. | | | | | | | | | | | | | | | \
2023-12-20T17:15:43.250-03:00 jvm 1 | +.\|=\|.+'+.|=|.+' +.\| \|.\| \|+'+.|=|.+'
2023-12-20T17:15:43.250-03:00 jvm 1 | Sumo Logic Collector Version 19.467-2
2023-12-20T17:15:43.250-03:00 jvm 1 | Sumo Logic Build Hash 54d48e135565
2023-12-20T17:15:43.251-03:00 jvm 1 | current folder:/opt/SumoCollector
2023-12-20T17:15:43.251-03:00 jvm 1 | * See /opt/SumoCollector/./logs for more details.
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.251-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.251-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.252-03:00 jvm 1 | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
2023-12-20T17:15:43.252-03:00 jvm 1 | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
2023-12-20T17:15:43.252-03:00 jvm 1 | (oslib.linux.CommandContainsType) cmd=lsb_release -sd > null
2023-12-20T17:15:43.252-03:00 jvm 1 | * Connecting to https://collectors.sumologic.com.
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.252-03:00 jvm 1 | WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2023-12-20T17:15:43.253-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.253-03:00 jvm 1 | ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
2023-12-20T17:15:43.253-03:00 wrapper | JVM received a signal SIGKILL (9).
2023-12-20T17:15:43.253-03:00 wrapper | JVM process is gone.
2023-12-20T17:15:43.253-03:00 wrapper | JVM process exited with a code of 1, setting the Wrapper exit code to 1.
2023-12-20T17:15:43.253-03:00 wrapper | Reloading Wrapper configuration...
2023-12-20T17:15:43.253-03:00 wrapper | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
2023-12-20T17:15:43.253-03:00 �]0;Sumo Logic Collector�wrapper | JVM process is gone.

My task definition definition as a sidecar in terraform is:

resource "aws_ecs_task_definition" "service_task_definition" {
  family                   = "${var.env}-${var.name}-task-definition"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  execution_role_arn       = var.task_execution_role

  //https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html
  cpu                      = 2048 //var.cpu +  256
  memory                   = 4096 //var.memory + 256

  runtime_platform {
    cpu_architecture        = "X86_64"
    operating_system_family = "LINUX"
  }

  volume {
    name = "service-logs-storage"  
  }

  volume {
    name = "service-sock-storage"  
  }

  container_definitions = jsonencode([
    {
      name      = var.container_name
      image     = var.docker_image
      repositoryCredentials = {
        credentialsParameter = var.docker_repository_credentials
      },
      cpu       = var.cpu
      memory    = var.memory
      essential = true
      networkMode = "awsvpc",

      secrets = [
        for key in var.secrets_names :
        {
          name      = key
          valueFrom = "${var.secrets_arn}:${key}::"
        }
      ]

      environment = [
          {
              "name": "ECS_AVAILABLE_LOGGING_DRIVERS",
              "value": "[\"json-file\",\"awslogs\"]"
          }
      ]

      healthCheck = {
        command = [
            "CMD-SHELL",
            "curl -f http://localhost/healthcheck || exit 1"
        ],
        interval = 300,
        timeout = 5,
        retries = 3,
        startPeriod = 120
      }

      logConfiguration = {
        logDriver = "awslogs",
        options   = {
          awslogs-create-group  = "true",
          awslogs-group         = "/ecs/${var.env}/${var.name}-task",
          awslogs-region        = "us-west-1",
          awslogs-stream-prefix = "ecs"
        }
      }

      portMappings = [
        {
          containerPort = 80
          hostPort      = 80
          protocol      = "tcp"
          appProtocol   = "http"
        }
      ]

      mountPoints = [
        {
            sourceVolume: "service-logs-storage",
            containerPath: "/var/lib/docker/containers/logs",
            readOnly: false
        },
        {
            sourceVolume: "service-sock-storage",
            containerPath: "/var/run/docker.sock",
            readOnly: false
        }      
      ]
    },
    {
      name      = "${var.container_name}-logs-collector"
      image     = var.logs_collector_docker_image
      repositoryCredentials = {
        credentialsParameter = var.docker_repository_credentials
      },
      cpu       = 256
      memory    = 512
      essential = true
      networkMode = "awsvpc",

      secrets = [
        for key in var.secrets_names :
        {
          name      = key
          valueFrom = "${var.secrets_arn}:${key}::"
        }
      ]

      environment = [
          {
              "name": "SUMO_COLLECTOR_NAME",
              "value": "${var.env}"
          },
          {
              "name": "SUMO_COLLECTOR_NAME_PREFIX",
              "value": ""
          },
          {
              "name": "SUMO_COLLECTOR_FIELDS",
              "value": "env=local,another=none"
          },
          {
              "name": "SUMO_ACCESS_ID",
              "value": "...."
          },
          {
              "name": "SUMO_ACCESS_KEY",
              "value": "..."
          },
      ]

      logConfiguration = {
        logDriver = "awslogs",
        options   = {
          awslogs-create-group  = "true",
          awslogs-group         = "/ecs/${var.env}/${var.name}-task",
          awslogs-region        = "us-west-1",
          awslogs-stream-prefix = "ecs"
        }
      }

      mountPoints = [
        {
            sourceVolume: "service-logs-storage",
            containerPath: "/var/lib/docker/containers/logs",
            readOnly: false
        },
        {
            sourceVolume: "service-sock-storage",
            containerPath: "/var/run/docker.sock",
            readOnly: false
        }
      ]
    }
  ])
}

I could not find any useful information on what could be the issue or how to deploy it on ECS Fargate without using an HTTP source.

frossi85 commented 11 months ago

I could solve it by using my own Dockerfile installing what my error said was missing in the original image:

FROM sumologic/collector
ADD file-sumo-sources.json /etc/sumo-sources.json

RUN apt-get update --quiet && \
  apt-get install -y lsb-core && \
  apt-get reinstall --quiet -y lsb-release