Netbox Restarts in Nomad

[k1rb]

Current Behavior

In nomad, using the docker driver, the "netbox" application exits (0) and is started again, constantly. I think the process is being killed due to resource (over) utilization but I can't seem to give it enough resources to keep it from restarting. The task or allocation associated with the netbox application isn't being killed or HUP'd by nomad, the process within the allocation/container is exiting and starting again.

The result is a momentary pause when working with netbox or worse, a 3 second http 503 from the reverse proxy, if it happens to coincide with a healthcheck.

The netbox-worker and netbox-housekeeping containers from the docker-compose haven't been implemented in my jobspec yet so this might be part of the problem.

Expected Behavior

The process starts once and doesn't exit.

Docker Compose Version

NA

Docker Version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:18 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:20 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The git Revision

NA

The git Status

NA

Startup Command

nomad job run

NetBox Logs

2024/03/27 21:45:22 [info] 672#672 "netbox" application started
# normal http access logs...
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
2024/03/27 21:45:23 [notice] 19#19 app process 669 exited with code 0
# normal http access logs...

Content of docker-compose.override.yml

job "netbox" {

  datacenters = ["dc1"]
  type = "service"

  group "app" {

    count = 1

    network {
      port "http"        { to = 8080 }
      port "psql"        { to = 5432 }
      port "redis"       { to = 6379 }
      port "redis-cache" { to = 6379 }

    }

    # service provider consul with health checks

    task "netbox-app" {
      driver = "docker"
      config {
        image = "docker.io/netboxcommunity/netbox:v3.6-2.7.0"
        ports = ["http"]
        volumes = [
          "/srv/containers/netbox/configuration:/etc/netbox/config",
          "/srv/containers/netbox/data/media:/opt/netbox/netbox/media",
          "/srv/containers/netbox/data/reports:/opt/netbox/netbox/reports",
          "/srv/containers/netbox/data/scripts:/opt/netbox/netbox/scripts"
        ]
      }
      # template for env values from consul kv & vault
      resources {
        cpu        = 6000 # 6Ghz
        memory     = 6144 # 6G
      }
    }

#   task "netbox-worker" {} from docker-compose, not implemented 

#   task "netbox-housekeeping" {} from docker-compose, not implemented 

    task "netbox-postgres" {
      driver = "docker"
      config {
        image = "docker.io/postgres:14-alpine"
        ports = ["psql"]
        volumes = [
          "/srv/containers/netbox/data/postgres/data:/var/lib/postgresql/data"
        ]
      }
      # template for env values from consul kv & vault
      resources {
        cpu        = 1000
        memory     = 1024
      }
    }

    task "netbox-redis" {
      driver = "docker"
      config {
        image = "docker.io/redis:6-alpine"
        command = "redis-server"
        args = ["--appendonly","yes","--requirepass","${REDIS_PASSWORD}",]
        ports = ["redis"]
        mount {
          type = "bind"
          source = "/srv/containers/netbox/data/redis/data"
          target = "/data"
          readonly = false
        }
      }
      # template for env values from consul kv & vault
    }

    task "netbox-redis-cache" {
      driver = "docker"
      config {
        image = "docker.io/redis:6-alpine"
        ports = ["redis-cache"]
        command = "redis-server"
        args = ["--requirepass","${REDIS_PASSWORD}"]
        volumes = [
          "/srv/containers/netbox/data/redis-cache/data:/data"
        ]
      }
      # template for env values from consul kv & vault
    }
  }
}