search

percona / mongodb_exporter

A Prometheus exporter for MongoDB including sharding, replication and storage engines
Apache License 2.0
1.16k stars 425 forks source link

Label cl_role not set #458

Closed McLeod095 closed 2 years ago

McLeod095 commented 2 years ago

Hello Testing on percona mongodb db.version(); 4.4.10-11 and community mongodb db.version() 4.4.13

Label cl_role not set correctly mongodb_term{cl_id="5e7395e86f46cb6d2d3a94de",cl_role="",rs_nm="Shard001",rs_state="1"} 27

Command db.adminCommand( { getCmdLineOpts: 1 } ) on both version return "replication" : { "replSetName" : "Shard002" }, "sharding" : { "clusterRole" : "shardsvr" }, and "replication" : { "replSetName" : "prdct-stats01" },

https://github.com/percona/mongodb_exporter/blob/main/exporter/topology_info.go#L186

kkrasnov1 commented 2 years ago

I have similar error. Messages from mongodb_exporter:

WARN[0004] cannot load topology labels: cannot check if the instance is an arbiter: timed out while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0
ERRO[0004] Cannot get node type to check if this is a mongos: timed out while checking out a connection from connection pool: context canceled; maxPoolSize: 100, connections in use by cursors: 0, connections in use by transactions: 0, connections in use by other operations: 0

func isArbiter call client.Database("admin").RunCommand(ctx, primitive.M{"isMaster": 1}).

db.getSiblingDB("admin").isMaster() not returned fileld "arbiterOnly" and returned

{
        "topologyVersion" : {
                "processId" : ObjectId("6234a958249aa818539f6f64"),
                "counter" : NumberLong(22)
        },
        "hosts" : [
                "xxxxx0201.yyyy:27018",
                "xxxxx0202.yyyy:27018",
                "xxxxx0203.yyyy:27018"
        ],
        "setName" : "MDLP-lt01-shard02",
        "setVersion" : 33,
        "ismaster" : false,
        "secondary" : true,
        "primary" : "xxxxx0201.yyyy:27018",
        "tags" : {
                "type_schedule" : "false"
        },
        "me" : "xxxxx0203.yyyy:27018",
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1651752135, 40794),
                        "t" : NumberLong(47)
                },
                "lastWriteDate" : ISODate("2022-05-05T12:02:15Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1651752135, 40794),
                        "t" : NumberLong(47)
                },
                "majorityWriteDate" : ISODate("2022-05-05T12:02:15Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2022-05-05T12:02:15.117Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 1575126,
        "minWireVersion" : 0,
        "maxWireVersion" : 9,
        "readOnly" : false,
        "ok" : 1,
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff000000000000002c")
        },
        "lastCommittedOpTime" : Timestamp(1651752135, 40794),
        "$configServerState" : {
                "opTime" : {
                        "ts" : Timestamp(1651752131, 4840),
                        "t" : NumberLong(30)
                }
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1651752135, 41843),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1651752135, 40794)
}

https://www.mongodb.com/docs/v4.4/reference/command/isMaster/#mongodb-data-isMaster.arbiterOnly

isMaster.arbiterOnly A boolean value that , when true, indicates that the current instance is an arbiter. The arbiterOnly field is only present, if the instance is an arbiter.

percona-csalguero commented 2 years ago

Hi. I wasn't able to reproduce the issue. Anyway, I can imagine a scenario where is could be happening. I made a test build for you to test on you environment: https://github.com/percona/mongodb_exporter/releases/tag/issue-458

Anyway, I would also like to share my tests:

Started docker sandbox

docker ps
CONTAINER ID   IMAGE       COMMAND                  CREATED         STATUS         PORTS                                           NAMES
6499627afb6e   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17000->27017/tcp, :::17000->27017/tcp   mongos
82b65a4cf9a5   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17001->27017/tcp, :::17001->27017/tcp   mongo-1-1
52be70b39b63   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17009->27017/tcp, :::17009->27017/tcp   mongo-cnf-1
c5ae07017bcc   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17006->27017/tcp, :::17006->27017/tcp   mongo-2-1
e67780ed25d7   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17012->27017/tcp, :::17012->27017/tcp   mongo-2-arbiter
1b5515c960bd   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17002->27017/tcp, :::17002->27017/tcp   mongo-1-2
8e824a047afb   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17007->27017/tcp, :::17007->27017/tcp   mongo-cnf-2
e38cda45ce9d   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17003->27017/tcp, :::17003->27017/tcp   mongo-1-3
bc821ece8ff1   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17005->27017/tcp, :::17005->27017/tcp   mongo-2-3
dbc58509964b   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17004->27017/tcp, :::17004->27017/tcp   mongo-2-2
7af34e4a9d1c   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17008->27017/tcp, :::17008->27017/tcp   mongo-cnf-3
ff2b9292f81f   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:17011->27017/tcp, :::17011->27017/tcp   mongo-1-arbiter
d4ab411e21e1   mongo:4.2   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   0.0.0.0:27017->27017/tcp, :::27017->27017/tcp   standalone

Ran this bash script

  #!/bin/bash

  PORTS="17000 17001 17002 17003 17004 17005 17006 17007 17008 17009 17011 17012 27017"
  HOST=127.0.0.1

  for PORT in $PORTS
  do
      echo "Starting MongoDB exporter for MongoDB running on port $PORT"
      ./mongodb_exporter --mongodb.uri=${HOST}:${PORT} --collect-all --compatible-mode --discovering-mode > /dev/null 2> /dev/null &
      sleep 2
      # First time we won't have all metrics. Run CURL twice
      curl --silent http://127.0.0.1:9216/metrics > /dev/null
      curl --silent http://127.0.0.1:9216/metrics | grep mongodb_members_self
      echo
      kill $(pidof mongodb_exporter)
  done

Results

Starting MongoDB exporter for MongoDB running on port 17000
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.2:27017",member_state="SECONDARY",rs_nm="rs2",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17001
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.13:27017",member_state="PRIMARY",rs_nm="rs1",rs_state="1"} 1

Starting MongoDB exporter for MongoDB running on port 17002
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.10:27017",member_state="SECONDARY",rs_nm="rs1",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17003
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.5:27017",member_state="SECONDARY",rs_nm="rs1",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17004
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.2:27017",member_state="SECONDARY",rs_nm="rs2",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17005
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.4:27017",member_state="SECONDARY",rs_nm="rs2",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17006
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="shardsvr",member_idx="172.18.0.11:27017",member_state="PRIMARY",rs_nm="rs2",rs_state="1"} 1

Starting MongoDB exporter for MongoDB running on port 17007
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="configsvr",member_idx="172.18.0.7:27017",member_state="SECONDARY",rs_nm="cnf-serv",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17008
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="configsvr",member_idx="172.18.0.8:27017",member_state="SECONDARY",rs_nm="cnf-serv",rs_state="2"} 1

Starting MongoDB exporter for MongoDB running on port 17009
# HELP mongodb_members_self members.
# TYPE mongodb_members_self untyped
mongodb_members_self{cl_id="6277ea2504e4573abdb34cdc",cl_role="configsvr",member_idx="172.18.0.12:27017",member_state="PRIMARY",rs_nm="cnf-serv",rs_state="1"} 1

Starting MongoDB exporter for MongoDB running on port 17011

Starting MongoDB exporter for MongoDB running on port 17012

Starting MongoDB exporter for MongoDB running on port 27017
kkrasnov1 commented 2 years ago

Hi @percona-csalguero, What version of mongodb do you have? My version of mongodb is 4.4.9.

percona-csalguero commented 2 years ago

db.version(); 4.2.18 I ran the test with the same version you are using and I got the same results. I'll appreciate if you found a reliable way to reproduce the issue so we can add specific tests.

Thanks

kkrasnov1 commented 2 years ago

db.version(); 4.2.18 I ran the test with the same version you are using and I got the same results. I'll appreciate if you found a reliable way to reproduce the issue so we can add specific tests.

Thanks

It works!

Thank you!