search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.02k stars 175 forks source link

Sensu-go ignores cluster configuration in the config file #4735

Open IvanovOleg opened 2 years ago

IvanovOleg commented 2 years ago

Expected Behavior

Sensu starts in the cluster mode

Current Behavior

/etc/sensu $ sensu-backend start --config-file /etc/sensu/config.yml
== running backend init...
{"component":"cmd","level":"info","msg":"attempting to connect to etcd server: http://root-statefulset-1:2379","time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://[::]:2380"],"time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://[::]:2379"],"time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.2","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.17.6","go-os":"linux","go-arch":"amd64","max-cpu-set":48,"max-cpu-available":48,"member-initialized":false,"name":"root-statefulset-1","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","wal-dir":"/var/lib/sensu/sensu-backend/etcd/wal","wal-dir-dedicated":"/var/lib/sensu/sensu-backend/etcd/wal","member-dir":"/var/lib/sensu/sensu-backend/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"300ms","election-timeout":"3s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://root-statefulset-1:2380"],"listen-peer-urls":["http://[::]:2380"],"advertise-client-urls":["http://root-statefulset-1:2379"],"listen-client-urls":["http://[::]:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://root-statefulset-1:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-size-bytes":4294967296,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"revision","auto-compaction-retention":"2ns","auto-compaction-interval":"2ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/sensu/sensu-backend/etcd/data/member/snap/db","took":"7.241318ms","time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"root-statefulset-1","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://root-statefulset-1:2380"],"advertise-client-urls":["http://root-statefulset-1:2379"],"time":"2022-05-05T16:40:27Z"}
{"component":"etcd","level":"info","caller":"embed/etcd.go:370","msg":"closed etcd server","name":"root-statefulset-1","data-dir":"/var/lib/sensu/sensu-backend/etcd/data","advertise-peer-urls":["http://root-statefulset-1:2380"],"advertise-client-urls":["http://root-statefulset-1:2379"],"time":"2022-05-05T16:40:27Z"}
{"component":"sensu-enterprise","error":"error starting etcd: couldn't find local name \"root-statefulset-1\" in the initial cluster configuration","level":"fatal","msg":"error executing sensu-backend","time":"2022-05-05T16:40:27Z"}

Possible Solution

Steps to Reproduce (for bugs)

  1. Deploy sensu in kubernetes
  2. Run it using sensu-backend start --config-file /etc/sensu/config.yml
  3. Use the following config file 
    ---
etcd-advertise-client-urls: "http://10.129.3.87:2379"
etcd-listen-client-urls: "http://10.129.3.87:2379"
etcd-listen-peer-urls: "http://0.0.0.0:2380"
etcd-initial-cluster: "root-statefulset-1=http://10.129.3.87:2380"
etcd-initial-advertise-peer-urls: "http://10.129.3.87:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "unique_token_for_this_cluster"
etcd-name: "root-statefulset-1"
debug: true
log-level: debug

Context

It doesn't ignore that file completely. I tried to change debug, log level, etcd-name and it reacts. But when it comes to the cluster configuration, it ignores it. I also tried to use backend.yml as a config file, didn't help. Sensu reacts if I add that cluster configuration using runtime parameter, but this way is unacceptable for me since I need that configuration to be dynamic. Changing IPs to the hostname in that config doesn't help either.

Your Environment

IvanovOleg commented 2 years ago

I believe the reason of the issue is in the entrypoint.sh file:

elif [ $called = "sensu-backend" ]; then
    : ${SENSU_BACKEND_CLUSTER_ADMIN_USERNAME:=admin}
    : ${SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD:=P@ssw0rd!}
    : ${SENSU_BACKEND_API_URL:=http://${SENSU_HOSTNAME}:8080}
    : ${SENSU_BACKEND_ETCD_INITIAL_CLUSTER:=default=http://${SENSU_HOSTNAME}:2380}
    : ${SENSU_BACKEND_ETCD_ADVERTISE_CLIENT_URLS:=http://${SENSU_HOSTNAME}:2379}
    : ${SENSU_BACKEND_ETCD_INITIAL_ADVERTISE_PEER_URLS:=http://${SENSU_HOSTNAME}:2380}
    : ${SENSU_BACKEND_ETCD_LISTEN_CLIENT_URLS:=http://[::]:2379}
    : ${SENSU_BACKEND_ETCD_LISTEN_PEER_URLS:=http://[::]:2380}
    : ${WAIT_PORT:=2379}

    export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME
    export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD
    export SENSU_BACKEND_API_URL
    export SENSU_BACKEND_ETCD_INITIAL_CLUSTER
    export SENSU_BACKEND_ETCD_ADVERTISE_CLIENT_URLS
    export SENSU_BACKEND_ETCD_INITIAL_ADVERTISE_PEER_URLS
    export SENSU_BACKEND_ETCD_LISTEN_CLIENT_URLS
    export SENSU_BACKEND_ETCD_LISTEN_PEER_URLS

    backend_init &
fi

most likely env vars take precedence

IvanovOleg commented 2 years ago

Fixed by mounting modified entrypoint.sh

#!/usr/bin/dumb-init sh

    SENSU=/opt/sensu

    called=$(basename $0)
    called_path=${SENSU}/bin/${called}

    : ${HOSTNAME:=$(hostname)}
    : ${SENSU_HOSTNAME:=$HOSTNAME}

    backend_init() {
        echo "== running backend init..."
        set -e
        ${called_path} init --wait
        INIT_RC=$?
        set +e
        if [ "x$INIT_RC" != "x0" ] && [ "x$INIT_RC" != "x3" ]; then
        echo "== backend init failed - exiting..."
            exit 1
        fi
    }

    if [ $called = "sensu-agent" ]; then
        : ${SENSU_BACKEND_URL:=ws://${SENSU_HOSTNAME}:8081}

        export SENSU_BACKEND_URL
    elif [ $called = "sensu-backend" ] && [ -f /etc/sensu/backend.yml ]; then
        : ${SENSU_BACKEND_CLUSTER_ADMIN_USERNAME:=admin}
        : ${SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD:=P@ssw0rd!}
        : ${SENSU_BACKEND_API_URL:=http://${SENSU_HOSTNAME}:8080}
        : ${WAIT_PORT:=2379}

        export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME
        export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD
        export SENSU_BACKEND_API_URL

        backend_init &
    elif [ $called = "sensu-backend" ]; then
        : ${SENSU_BACKEND_CLUSTER_ADMIN_USERNAME:=admin}
        : ${SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD:=P@ssw0rd!}
        : ${SENSU_BACKEND_API_URL:=http://${SENSU_HOSTNAME}:8080}
        : ${SENSU_BACKEND_ETCD_INITIAL_CLUSTER:=default=http://${SENSU_HOSTNAME}:2380}
        : ${SENSU_BACKEND_ETCD_ADVERTISE_CLIENT_URLS:=http://${SENSU_HOSTNAME}:2379}
        : ${SENSU_BACKEND_ETCD_INITIAL_ADVERTISE_PEER_URLS:=http://${SENSU_HOSTNAME}:2380}
        : ${SENSU_BACKEND_ETCD_LISTEN_CLIENT_URLS:=http://[::]:2379}
        : ${SENSU_BACKEND_ETCD_LISTEN_PEER_URLS:=http://[::]:2380}
        : ${WAIT_PORT:=2379}

        export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME
        export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD
        export SENSU_BACKEND_API_URL
        export SENSU_BACKEND_ETCD_INITIAL_CLUSTER
        export SENSU_BACKEND_ETCD_ADVERTISE_CLIENT_URLS
        export SENSU_BACKEND_ETCD_INITIAL_ADVERTISE_PEER_URLS
        export SENSU_BACKEND_ETCD_LISTEN_CLIENT_URLS
        export SENSU_BACKEND_ETCD_LISTEN_PEER_URLS

        backend_init &
    fi

    ${called_path} $@