MQPROMETHEUS service fails to start

[JJinMaine]

MQ 9.0.4.0 mq-golang 4.1.1 go 1.12.13 MQ Client 9.1.2.0 Libraries

We're trying to get some basic prometheus stats out of a sandbox queue manager that has I think 3 queues defined and that's it. We get this from the debug:

time="2019-11-01T10:43:52-04:00" level=debug msg="Monitored topics are 'IBM/Software/Results'"
time="2019-11-01T10:43:52-04:00" level=debug msg="Connecting to queue manager IRTEST"
time="2019-11-01T10:43:52-04:00" level=info msg="Connected to queue manager  IRTEST"
time="2019-11-01T10:43:52-04:00" level=fatal msg="Error subscribing to $SYS/MQ/INFO/QMGR/IRTEST/Monitor/STATQ/%s/OPENCLOSE: Error subscribing to topic '$SYS/MQ/INFO/QMGR/IRTEST/Monitor/STATQ/SYSTEM.SELECTION.VALIDATION.QUEUE/OPENCLOSE': MQSUB: MQCC = MQCC_FAILED [2] MQRC = MQRC_HANDLE_NOT_AVAILABLE [2017]"

And the service never starts. We checked the max handles and it's the default 256 so we're really not sure why a handle isn't available. Even after a queue manager restart we get the same result.

I built the mq_prometheus binary on a different machine than the queue manager and moved it, the sh scripts and the mqsc file to the queue manager server.

This is the startup script and the options we use:

#!/bin/bash

# This is used to start the IBM MQ monitoring service for Prometheus

# The queue manager name comes in from the service definition as the
# only command line parameter
qMgr=$1

# Set the environment to ensure we pick up libmqm.so etc
# Try to run it for a local qmgr; if that fails fallback to a
# default
# If this is a client connection, then deal with no known qmgr of the given name.
. /opt/mqm/bin/setmqenv -m $qMgr -k >/dev/null 2>&1
if [ $? -ne 0 ]
then
  . /opt/mqm/bin/setmqenv -s -k
fi

# A list of queues to be monitored is given here.
# It is a set of names or patterns ('*' only at the end, to match how MQ works),
# separated by commas. When no queues match a pattern, it is reported but
# is not fatal.
# The set can also include negative patterns such as "!SYSTEM.*".
queues="IR360.*,SYSTEM.*,PULSE.*"

# An alternative is to have a file containing the patterns, and named
# via the ibmmq.monitoredQueuesFile option.

# Do similar for channels
channels="TO.*,SYSTEM.DEF.SVRCONN"

# See config.go for all recognised flags
ARGS="-ibmmq.queueManager=$qMgr"
ARGS="$ARGS -ibmmq.userid=user"
ARGS="$ARGS -ibmmq.password=password"
ARGS="$ARGS -ibmmq.monitoredQueues=$queues"
ARGS="$ARGS -ibmmq.monitoredChannels=$channels"
ARGS="$ARGS -ibmmq.monitoredTopics=IBM/Software/Results"
ARGS="$ARGS -ibmmq.monitoredSubscriptions=*"
ARGS="$ARGS -rediscoverInterval=1h"

ARGS="$ARGS -ibmmq.useStatus=true"
ARGS="$ARGS -log.level=debug"

# This may help with some issues if the program has a SEGV. It
# allows Go to do a better stack trace.
export MQS_NO_SYNC_SIGNAL_HANDLING=true

# Start via "exec" so the pid remains the same. The queue manager can
# then check the existence of the service and use the MQ_SERVER_PID value
# to kill it on shutdown.
exec /app/mqgo/mq_prometheus $ARGS

Here is the MQSC file:

* Cleanup any existing service
STOP SERVICE(MQPROMETHEUS)
DELETE SERVICE(MQPROMETHEUS)

* Reset the definition
DEFINE SERVICE(MQPROMETHEUS)         +
       CONTROL(QMGR)               +
       SERVTYPE(SERVER)            +
       STARTCMD('/app/mqgo/mq_prometheus.sh') +
       STARTARG(+QMNAME+)          +
       STOPCMD('/usr/bin/kill ' )  +
       STOPARG(+MQ_SERVER_PID+)    +
       STDOUT('/app/mqgo/mq_prometheus.out')  +
       STDERR('/app/mqgo/mq_prometheus.err')  +
       DESCR('MQ exporter for Prometheus')

* Start it manually now; will be automatically started on future qmgr startup
START SERVICE(MQPROMETHEUS)

Any guidance would be appreciated - thanks!

[ibmmqmet]

You're asking to monitor all the SYSTEM queues - there's about 50 of those on my system. So it's easy to eat up that 256 handles - each queue gets about 4 subscriptions for different topics.

[JJinMaine]

You're asking to monitor all the SYSTEM queues - there's about 50 of those on my system. So it's easy to eat up that 256 handles - each queue gets about 4 subscriptions for different topics.

I set the max handlers to 500 and that resolved the issue - thank you!