Container hangs - Githubissues

ragnarkurmwunder commented 2 years ago

Hi

We run Sumologic in the container on ECS Fargate on AWS. It would run for a month and then the container hangs. We see that memory has climbed to 99%, and the CPU had a very mild activity. The container has 512MB of memory. It is not possible to exec into the container, because there is no memory left in the container to execute a shell. SumoLogic conf and logs are kept on EFS (a variant of NFS), and that works great. I checked the logs, but nothing alarming.

build.log:

sumologic/collector:release-v19.308-12

collector.out.log:

INFO   | jvm 1    | 2021/11/15 10:51:51 | Sumo Logic Collector Version 19.308-12
INFO   | jvm 1    | 2021/11/15 10:51:51 | Sumo Logic Build Hash 3ef308a0f515

collector-usage.log collector.log

Questions:

How to figure out the root cause?
How to fix the problem?

maimaisie commented 2 years ago

Hi @ragnarkurmwunder, sorry for the late reply. We will need the full collector log file to start the investigation. If you have support access, it's best to open a support ticket and that way you will get more timely response. Thank you!

frossi85 commented 6 months ago

@ragnarkurmwunder kinda off topic but I am trying to run the collector using ECS but I am not able to do it, it works locally using docker-compose and from ECS it creates the collector in sumologic dashboard but when I deploy it to ECS it fails with messages like this:

Full log:


2023-12-20T17:15:18.376-03:00 | Running SumoLogic Collector...
-- | --
  | 2023-12-20T17:15:36.734-03:00 | wrapper \| JVM exited unexpectedly.
  | 2023-12-20T17:15:43.249-03:00 | wrapper \| The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
  | 2023-12-20T17:15:43.249-03:00 | wrapper \| --> Wrapper Started as Console
  | 2023-12-20T17:15:43.250-03:00 | ]0;Sumo Logic Collectorwrapper \| Java Service Wrapper Standard Edition 64-bit 3.5.49
  | 2023-12-20T17:15:43.250-03:00 | wrapper \| Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
  | 2023-12-20T17:15:43.250-03:00 | wrapper \| http://wrapper.tanukisoftware.com
  | 2023-12-20T17:15:43.250-03:00 | wrapper \| Licensed to Sumo Logic Inc. for Collector
  | 2023-12-20T17:15:43.250-03:00 | wrapper \|
  | 2023-12-20T17:15:43.250-03:00 | wrapper \| Launching a JVM...
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| WrapperManager: Initializing...
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| . . . . . . . . .
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| .+'\|=\|`+. .+'\| \|`+. .+'\|=\|`+.=\|`+. .+'\|=\|`+.
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| \| \| `+.\| \| \| \| \| \| \| `+ \| `+ \| \| \| \| \|
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| \| \| . \| \| \| \| \| \| \| \| \| \| \| \| \| \|
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| `+.\|=\|`+. \| \| \| \| \| \| \| \| \| \| \| \| \| \|
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| . \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| \|`+. \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| `+.\|=\|.+' `+.\|=\|.+' `+.\| \|.\| \|+' `+.\|=\|.+'
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| Sumo Logic Collector Version 19.467-2
  | 2023-12-20T17:15:43.250-03:00 | jvm 1 \| Sumo Logic Build Hash 54d48e135565
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| current folder:/opt/SumoCollector
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| * See /opt/SumoCollector/./logs for more details.
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
  | 2023-12-20T17:15:43.251-03:00 | jvm 1 \| ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| (oslib.linux.CommandContainsType) cmd=lsb_release -sd > null
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| * Connecting to https://collectors.sumologic.com.
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.252-03:00 | jvm 1 \| WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
  | 2023-12-20T17:15:43.253-03:00 | jvm 1 \| ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
  | 2023-12-20T17:15:43.253-03:00 | jvm 1 \| ERROR StatusConsoleListener Appender Console cannot be located. Route ignored
  | 2023-12-20T17:15:43.253-03:00 | wrapper \| JVM received a signal SIGKILL (9).
  | 2023-12-20T17:15:43.253-03:00 | wrapper \| JVM process is gone.
  | 2023-12-20T17:15:43.253-03:00 | wrapper \| JVM process exited with a code of 1, setting the Wrapper exit code to 1.
  | 2023-12-20T17:15:43.253-03:00 | wrapper \| Reloading Wrapper configuration...
  | 2023-12-20T17:15:43.253-03:00 | wrapper \| The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
  | 2023-12-20T17:15:43.253-03:00 | ]0;Sumo Logic Collectorwrapper \| JVM process is gone.

My container definition as a sidecar in terraform is:

{
      name      = "${var.container_name}-logs-collector"
      image     = var.logs_collector_docker_image
      repositoryCredentials = {
        credentialsParameter = var.docker_repository_credentials
      },
      cpu       = 256
      memory    = 512
      essential = true
      networkMode = "awsvpc",

      secrets = [
        for key in var.secrets_names :
        {
          name      = key
          valueFrom = "${var.secrets_arn}:${key}::"
        }
      ]

      environment = [
          {
              "name": "SUMO_COLLECTOR_NAME",
              "value": "${var.env}"
          },
          {
              "name": "SUMO_COLLECTOR_NAME_PREFIX",
              "value": ""
          },
          {
              "name": "SUMO_COLLECTOR_FIELDS",
              "value": "env=local,another=none"
          },
          {
              "name": "SUMO_ACCESS_ID",
              "value": "..."
          },
          {
              "name": "SUMO_ACCESS_KEY",
              "value": "...."
          },
      ]

      logConfiguration = {
        logDriver = "awslogs",
        options   = {
          awslogs-create-group  = "true",
          awslogs-group         = "/ecs/${var.env}/${var.name}-task",
          awslogs-region        = "us-west-1",
          awslogs-stream-prefix = "ecs"
        }
      }

      mountPoints = [
        {
            sourceVolume: "service-logs-storage",
            containerPath: "/var/lib/docker/containers/logs",
            readOnly: false
        }
      ]
    }

frossi85 commented 6 months ago

I could solve it by using my own Dockerfile installing what my error said was missing in the original image:

FROM sumologic/collector
ADD file-sumo-sources.json /etc/sumo-sources.json

RUN apt-get update --quiet && \
  apt-get install -y lsb-core && \
  apt-get reinstall --quiet -y lsb-release

SumoLogic / sumologic-collector-docker

Container hangs #97