search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.89k stars 1.95k forks source link

Zabbix and Fping: /usr/sbin/fping: Operation not permitted #11588

Closed replay111 closed 2 years ago

replay111 commented 2 years ago

Nomad version

nomad -v : Nomad v1.2.0 (0e11f8d5173e78dce23433903d086e3f9dfea7ea)

Operating system and Environment details

cat /etc/redhat-release
CentOS Stream release 8
docker --version
Docker version 20.10.11, build dea9396

Issue

When I am running task with zabbix-server using docker driver I am not able to use fping - bash: /usr/sbin/fping: Operation not permitted

When I am running the same container but directly on docker - everything is working fine

Reproduction steps

cat << 'EOF' > ./zabbix.nomad
job "zabbix" {
  datacenters = [ "CENTOS" ]
  type = "service"

  group "zabbix-server" {
    count = 1
    network {
      port "zabbix-server-port" {
        static = 10051
      }
    }

    task "prepare_dirs" {
      driver = "raw_exec"
      lifecycle {
        hook = "prestart"
        sidecar = false
      }
      template {
        data = <<TPLEOF
#!/bin/bash
set -x
docker volume prune -f
docker network prune -f
docker system prune -f -a
mkdir -p /nfs_shared/zabbix/snmptraps
mkdir -p /nfs_shared/zabbix/export
chmod -Rv 777 /nfs_shared/zabbix
TPLEOF
        destination = "/local/prepare-dirs.bash"
      }
      config {
        command = "/bin/bash"
        args    = ["-x","local/prepare-dirs.bash"]
      }
    }

    task "zabbix-server" {
      driver = "docker"
      env {
        TZ = "Europe/Warsaw"
        DB_SERVER_HOST="192.168.1.1"
        DB_SERVER_PORT="5432"
        POSTGRES_USER="zabbix"
        POSTGRES_PASSWORD="zabbix"
        POSTGRES_DB="zabbix"
        DB_SERVER_SCHEMA="zabbix"
      }
      config {
        image  = "zabbix/zabbix-server-pgsql:ol-5.4-latest"
        mount {
          type = "bind"
          target = "/var/lib/zabbix/export/"
          source = "/nfs_shared/zabbix/export/"
          readonly = false
        }
        mount {
          type = "bind"
          target = "/var/lib/zabbix/snmptraps/"
          source = "/nfs_shared/zabbix/snmptraps/"
          readonly = false
        }
        ports = ["zabbix-server-port"]
      }
      resources {
        memory = 256
        cpu    = 1024
      }
    }
  }

}
EOF

nomad  stop --yes --purge zabbix ;  nomad system gc ; nomad run zabbix.nomad
docker ps -a | grep -i zabbix-server
docker exec -it <container id> bash
fping 8.8.8.8

Expected Result

fping 8.8.8.8
8.8.8.8 is alive

Actual Result

bash-4.4$ fping
bash: /usr/sbin/fping: Operation not permitted

Nomad Server logs (if appropriate)

2021-11-30T12:38:42.011+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=aff412f8-bd25-1e03-a560-28aacbc2b472 task=prepare_dirs path=/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/alloc/logs/.prepare_dirs.stdout.fifo @module=logmon timestamp="2021-11-30T12:38:42.011+0100"
2021-11-30T12:38:42.011+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=aff412f8-bd25-1e03-a560-28aacbc2b472 task=prepare_dirs @module=logmon path=/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/alloc/logs/.prepare_dirs.stderr.fifo timestamp="2021-11-30T12:38:42.011+0100"
2021-11-30T12:38:42.016+0100 [INFO]  agent: 2021/11/30 12:38:42.016960 [INFO] (runner) creating new runner (dry: false, once: false)
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017080 [DEBUG] (runner) final config: {"Consul":{"Address":"127.0.0.1:8500","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":""},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"#!/bin/bash\nset -x\ndocker volume prune -f\ndocker network prune -f\ndocker system prune -f -a\nmkdir -p /nfs_shared/zabbix/snmptraps\nmkdir -p /nfs_shared/zabbix/export\nchmod -Rv 777 /nfs_shared/zabbix\n","CreateDestDirs":true,"Destination":"/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/prepare_dirs/local/prepare-dirs.bash","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":["plugin"],"SandboxPath":"/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/prepare_dirs"}],"Vault":{"Address":"","Enabled":false,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":5,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017160 [INFO] (runner) creating watcher
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017295 [INFO] (runner) starting
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017348 [DEBUG] (runner) running initial templates
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017395 [DEBUG] (runner) initiating run
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017430 [DEBUG] (runner) checking template 5241c4206b0374110d3e411d49578f65
2021-11-30T12:38:42.017+0100 [INFO]  agent: 2021/11/30 12:38:42.017628 [DEBUG] (runner) rendering "(dynamic)" => "/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/prepare_dirs/local/prepare-dirs.bash"
2021-11-30T12:38:42.018+0100 [INFO]  agent: 2021/11/30 12:38:42.018641 [INFO] (runner) rendered "(dynamic)" => "/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/prepare_dirs/local/prepare-dirs.bash"
2021-11-30T12:38:42.018+0100 [INFO]  agent: 2021/11/30 12:38:42.018689 [DEBUG] (runner) diffing and updating dependencies
2021-11-30T12:38:42.018+0100 [INFO]  agent: 2021/11/30 12:38:42.018719 [DEBUG] (runner) watching 0 dependencies
2021-11-30T12:38:42.018+0100 [INFO]  agent: 2021/11/30 12:38:42.018742 [DEBUG] (runner) all templates rendered
2021-11-30T12:38:42.021+0100 [INFO]  client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-x local/prepare-dirs.bash]}"
2021-11-30T12:38:42.493+0100 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=aff412f8-bd25-1e03-a560-28aacbc2b472 task=prepare_dirs reason="Restart unnecessary as task terminated successfully"
2021-11-30T12:38:42.497+0100 [INFO]  agent: 2021/11/30 12:38:42.497530 [INFO] (runner) stopping
2021-11-30T12:38:42.497+0100 [INFO]  agent: 2021/11/30 12:38:42.497605 [DEBUG] (runner) stopping watcher
2021-11-30T12:38:42.497+0100 [INFO]  agent: 2021/11/30 12:38:42.497647 [DEBUG] (watcher) stopping all views
2021-11-30T12:38:42.497+0100 [INFO]  agent: 2021/11/30 12:38:42.497708 [INFO] (runner) received finish
2021-11-30T12:38:42.511+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=aff412f8-bd25-1e03-a560-28aacbc2b472 task=zabbix-server @module=logmon path=/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/alloc/logs/.zabbix-server.stdout.fifo timestamp="2021-11-30T12:38:42.511+0100"
2021-11-30T12:38:42.511+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=aff412f8-bd25-1e03-a560-28aacbc2b472 task=zabbix-server @module=logmon path=/app/nomad/storage/alloc/aff412f8-bd25-1e03-a560-28aacbc2b472/alloc/logs/.zabbix-server.stderr.fifo timestamp="2021-11-30T12:38:42.511+0100"
2021-11-30T12:38:53.113+0100 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=3e629915d30f805ee0cb0d2d2a1913fa871619823084b0e95765bdb54e9dd051
2021-11-30T12:38:53.758+0100 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=3e629915d30f805ee0cb0d2d2a1913fa871619823084b0e95765bdb54e9dd051

Nomad Client logs (if appropriate)

   219:20211130:123232.496 server #41 started [history poller #4]
   203:20211130:123232.497 server #25 started [poller #1]
   218:20211130:123232.499 server #40 started [history poller #3]
   217:20211130:123232.502 server #39 started [history poller #2]
   207:20211130:123232.503 server #29 started [poller #5]
   208:20211130:123232.505 server #30 started [unreachable poller #1]
   220:20211130:123232.506 server #42 started [history poller #5]
   210:20211130:123232.509 server #32 started [trapper #2]
   213:20211130:123232.512 server #35 started [trapper #5]
   221:20211130:123232.512 server #43 started [availability manager #1]
   215:20211130:123232.512 server #37 started [alert syncer #1]
   214:20211130:123303.509 sh: /usr/sbin/fping: Operation not permitted
   214:20211130:123403.531 sh: /usr/sbin/fping: Operation not permitted
   214:20211130:123503.546 sh: /usr/sbin/fping: Operation not permitted
   214:20211130:123603.562 sh: /usr/sbin/fping: Operation not permitted
   214:20211130:123703.581 sh: /usr/sbin/fping: Operation not permitted

=========================== I've raised issue for this with zabbix guys - but after a couple of tests it turns out that docker-compose/docker is working without any problems - for me this problems only occurs when using nomad

https://github.com/zabbix/zabbix-docker/issues/900

apollo13 commented 2 years ago

@replay111 Disable selinux and check if it works then

apollo13 commented 2 years ago

Actually, nomad drops the cap net_raw -- if you readd that it can work again https://www.nomadproject.io/docs/drivers/docker#allow_caps

You will get the same issue with docker if you execute:

docker run --rm -it --cap-drop net_raw ubuntu:latest bash -c "apt update && apt -y install fping && fping 8.8.8.8"
replay111 commented 2 years ago

@apollo13 selinux is disabled, and docker itself is working perfectly - only when running via nomad I have a problem - please see here: https://github.com/zabbix/zabbix-docker/issues/900 - I did some checks earlier with gus from zabbix.

replay111 commented 2 years ago

@apollo13 - thank you! adding allow_caps in server satnza solved this issue ;-)

DerekStrickland commented 2 years ago

Glad that worked out! I'm closing this issue, but feel free to re-open if you have further need.

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.