This tool allows the monitoring of BLS keys that can participate into the consensus, regardless of the shard they are currently operating in.
This is done by continuously polling the /validator/statistics
API endpoint route.
It can monitor one or more networks so one instance of this tool is enough for all keys used on mainnet/testnet or devnet networks.
The monitored BLS keys will be defined as lists. Also, there is the possibility to define just an owner address that staked the BLS keys
and the application will automatically fetch the registered BLS keys for that identity.
You can choose to run this tool either in a Docker on in a systemd service.
Although it's possible, it is not recommended to run the application as root
. For that reason, a new user is required to be created.
For example, the following script creates a new user called ubuntu
. This script can be run as root
.
# host update/upgrade
apt-get update
apt-get upgrade
apt autoremove
adduser ubuntu
# set a long password
usermod -aG sudo ubuntu
echo 'StrictHostKeyChecking=no' >> /etc/ssh/ssh_config
visudo
# add this line:
ubuntu ALL=(ALL) NOPASSWD:ALL
# save & exit
sudo su ubuntu
sudo visudo -f /etc/sudoers.d/myOverrides
# add this line:
ubuntu ALL=(ALL) NOPASSWD:ALL
# save & exit
You need to have Docker installed on your machine.
Copy the template configs from the example
directory:
cd
git clone https://github.com/multiversx/mx-chain-keys-monitor-go
cd mx-chain-keys-monitor-go
cp ./cmd/monitor/config/example/* ./cmd/monitor/config
Customize your config files from the ./cmd/monitor/config
directory:
Fetch the image & start it, using the docker-compose.yml file:
sudo docker compose -f ./docker-compose.yml up -d mx-chain-keys-monitor-go
You're ready 🚀
cd
git clone https://github.com/multiversx/mx-chain-keys-monitor-go
cd ~/mx-chain-keys-monitor-go/scripts
# the following init call will create ~/mx-chain-keys-monitor-go/scripts/config/local.cfg file
# and will copy the configs from ~/mx-chain-keys-monitor-go/cmd/monitor/config/example to ~/mx-chain-keys-monitor-go/cmd/monitor/config
# to avoid github pull problems
./script.sh init
cd config
# edit the local.cfg file for the scripts setup
nano local.cfg
The generated local.cfg file contains the following lines:
#!/bin/bash
set -e
CUSTOM_HOME=/home/ubuntu
CUSTOM_USER=ubuntu
GITHUBTOKEN=""
MONITOR_EXTRA_FLAGS=""
#Allow user to override the current version of the monitor
OVERRIDE_VER=""
The CUSTOM_HOME
and CUSTOM_USER
will need to be changed if the current user is not ubuntu
.
To easily figure out the current user, the bash command whoami
can be used.
It is strongly recommended to use a GitHub access token because the scripts consume the GitHub APIs and throttling might occur without the access token.
The MONITOR_EXTRA_FLAGS
can contain extra flags to be called whenever the application is started.
The complete list of the cli command can be found here
The OVERRIDE_VER
can be used during testing to manually specify an override tag/branch that will be used when building
the application. If left empty, the upgrade process will automatically fetch and use the latest release.
After the local.cfg
configuration step, the scripts can now install the application.
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh install
After the application has been installed, it is now time to configure it.
For this, you should edit the config.toml
and credentials.toml
files and add files containing lists of BLS keys or addresses in the
~/mx-chain-keys-monitor-go/cmd/monitor/config
directory.
The scripts init step already created some initial .toml and .list files to be ready to be used directly.
Configuring the config.toml file:
This file contains the general application configuration file.
[General]
ApplicationName = "Keys monitoring app"
# the application can send messages about the internal status at regular intervals
[General.SystemSelfCheck]
Enabled = true
DayOfWeek = "every day" # can also be "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday" and "Sunday"
Hour = 12 # valid interval 0-23
Minute = 0 # valid interval 0-59
PollingIntervalInSec = 30
[General.Logs]
LogFileLifeSpanInMB = 1024 # 1GB
LogFileLifeSpanInSec = 86400 # 1 day
[General.AlarmSnooze]
Enabled = true
NumNotificationsForEachFaultyKey = 3
SnoozeTimeInSec = 28800 # 8 hours
[OutputNotifiers]
NumRetries = 3
SecondsBetweenRetries = 10
# Uses Pushover service that can notify Desktop, Android or iOS devices. Requires a valid subscription.
# If you enable this notifier, remember to specify the credentials in credentials.toml file
[OutputNotifiers.Pushover]
Enabled = false
URL = "https://api.pushover.net/1/messages.json"
# SMTP (email) based notification
# If you enable this notifier, remember to specify the credentials in credentials.toml file
# If you are using gmail server, please make sure you activate the IMAP server and use App passwords instead of the account's password
[OutputNotifiers.Smtp]
Enabled = false
To = "to@email.com"
SmtpPort = 587
SmtpHost = "smtp.gmail.com"
# Uses Telegram service that can notify Desktop, Android or iOS devices. Requires a running bot and the chat ID for
# the user that will be notified.
# If you enable this notifier, remember to specify the credentials in credentials.toml file
[OutputNotifiers.Telegram]
Enabled = false
URL = "https://api.telegram.org"
[[BLSKeysMonitoring]]
AlarmDeltaRatingDrop = 1.0 # maximum Rating-TempRating value that will trigger an alarm, for the public testnet might use a higher value (2 or 3)
Name = "network 1"
ApiURL = "API URL 1"
ExplorerURL = ""
PollingIntervalInSeconds = 300 # 5 minutes
ListFile = "./config/network1.list"
The General
section
The ApplicationName
can be any string, it will be used in the notification messages.
The General.SystemSelfCheck
section can enable & configure the self check monitoring system
that will output a notification message once a day or week, depending on the configuration parameters.
The General.Logs
will configure the internal logging system (for debugging purposes).
The General.AlarmSnooze
section will configure the number of the notify events for each monitored
BLS key that is faulty and the time "snooze" duration time, in seconds. If disabled, all the notification
events will be pushed to the notifiers after each rating drop checks.
The OutputNotifiers
is the section containing the implemented notifiers.
There are 3 types of notifiers implemented: Pushover
, Smtp
and Telegram
, each with its configuration sections.
The credentials for the notifiers are defined separately in the credentials.toml
so that is the place where
passwords or access tokens will be specified.
The BLSKeysMonitoring
defines the section used on one network.
The config.toml
file accepts any number of this type of section.
The AlarmDeltaRatingDrop
option will specify how large the difference between the "Epoch start rating" - "Current rating" is allowed before
emitting an alert. Usually, a value of 1
will suffice in most cases. On the public testnet, this value might be increased to 2
or even 3
to
reduce the number of false positive alarms due to the nature of the other nodes operating on the network.
The Name
defines a string for the monitored network. Can be something like Mainnet
, Testnet
, Devnet
or any kind of identification string.
The ApiURL
defines the API url for that network. Examples here include https://api.multiversx.com
for the mainnet,
https://testnet-api.multiversx.com
for the testnet, and https://devnet-api.multiversx.com
for the devnet.
The ExplorerURL
is used whenever a BLS key alert message is emitted, to automatically include the link to that BLS key page.
If left empty, the message will still be emitted, but it will not contain any link.
Examples here include https://explorer.multiversx.com
for the mainnet,
https://testnet-explorer.multiversx.com
for the testnet, and https://devnet-explorer.multiversx.com
for the devnet.
The PollingIntervalInSeconds
represents the time in seconds between the calls on the API URL.
The ListFile
will contain the name of the file containing BLS or identity keys. Refer to the example file called
network1.list
to check how keys/identities can be defined.
Before the application start, it is a good practice to test the configured notifiers
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh test_notifiers
If all enabled notifiers are configured correctly, no error messages should appear on the screen.
Important: This command won't start the monitoring process!
After editing the required config files, the application can be started.
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh start
When the application starts, it automatically emits a notification message. This is valid also for the stop
operation
issued by the ./script.sh stop
command.
It is a good practice to save the .toml, .list and the local.cfg files somewhere else just in case the application is cleaned up accidentally. The upgrade call for the monitor app is done through this command:
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh upgrade
The application can be removed by executing the following script:
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh cleanup
If the application fails to start (maybe there is a bad config in the .toml files), the following command can be issued:
sudo journalctl -f -u mx-chain-keys-monitor.service
Also, if the application misbehaves, the logs can be retrieved by using this command:
cd ~/mx-chain-keys-monitor-go/scripts
./script.sh get_logs
If the application crashes, and you have followed the installation via Docker, the command to retrieve the logs is as follows:
sudo docker logs -f mx-chain-keys-monitor-go