DISCLAIMER: Don't allow public access to PANIC UI as it might contain sensitive information about your infrastructure. UI authentication is still to be developed.
PANIC is an open source monitoring and alerting solution for Cosmos-SDK, Substrate and Chainlink based nodes by Simply VC. The tool was built with user friendliness in mind and comes with numerous features, such as phone calls for critical alerts, a UI Dashboard, a Web based installation process and Telegram/Slack commands for increased control over your alerter.
We are sure that PANIC will be beneficial for node operators and we look forward for feedback. Feel free to read on if you are interested in the design of the alerter, if you wish to try it out, or if you would like to support and contribute to this open source project.
If you want to dive into the design and feature set of PANIC click here.
We will now guide you through the steps required to get PANIC up and running. We recommend that PANIC is installed on a Linux system and that everything needed in the Requirements section is done before the installation procedure is started.
As you will notice below, PANIC supports many alerting channels. It is recommended that at least one of the alerting channels mentioned in the requirements section is set-up.
TIP: If your terminal is telling you that you do not have permissions to run a command try adding sudo
to your command e.g, sudo docker --version
this will run your command as root. If you have any issues during the installation procedure check out the FAQ section.
Note: Skip this step if Git is already installed.
Firstly we will install and verify your Git installation.
# Install Git
sudo apt install git
# Verify that git is now installed
git --version
This should give you the current version of git that has been installed.
Note: Skip this step if Docker and Docker Compose is already installed.
First, install Docker and Docker Compose by running these commands on your terminal.
# Install docker and docker-compose
curl -sSL https://get.docker.com/ | sh
sudo apt install docker-compose -y
# Confirm that installation successful
docker --version
docker-compose --version
These should give you the current versions of the software that have been installed. At the time of writing the current working docker version is 20.10.10
while the docker-compose version is 1.25.0
. If you have a different version that doesn't allow you to run the docker-compose.yml
file then either upgrade your versions of docker
and docker-compose
or change the version inside of the docker-compose.yml
file which is currently at 3.7
.
# Clone the panic repository and navigate into it
git clone https://github.com/SimplyVC/panic
cd panic
Now that you're inside the PANIC directory, open up the .env file and change the UI_ACCESS_IP
field to the IP of where PANIC UI is going to be hosted (can be set to localhost
if running locally). This is to ensure that the API (PANIC UI Backend) is only accessible from the UI. Helper scripts which can be used to get the IP address (scripts/get_ip_linux.sh
and scripts/get_ip_mac.sh
) are available but please note that these are not guaranteed to work on all servers/machines.
# This will access the .env file on your terminal
nano .env
Once inside change UI_ACCESS_IP
accordingly. Here is an example:
UI_ACCESS_IP=1.1.1.1
Then to exit hit the following keys:
Running PANIC
Once you have everything setup, you can start PANIC by running the below command:
docker-compose up -d --build
NOTE If build fails run these commands to clean your docker
images and try again. Please be aware that these commands will also stop other docker
images that you might have running on your system.
docker-compose kill
docker system prune -a --volumes
docker-compose up -d --build
The next step is to configure PANIC to monitor your nodes and systems as well as give it the channels to alert you through. You can do this by navigating to the PANIC UI at https://{UI_ACCESS_IP}:3333
, or at https://localhost:3333
if you're running it locally. The PANIC UI will start the installation procedure if it does not detect any configurations. Make sure you type HTTPS if you're getting an error when accessing PANIC UI on your browser.
After you set-up PANIC to your liking the installation procedure will save these details in the Mongo database. For correct behavior the database should never be modified manually. If you would like to edit the configurations at some point you can do so by accessing the settings option on the PANIC UI header.
PANIC will automatically read these configuration files and begin monitoring the data sources. To confirm that PANIC is running as expected we suggest running the command docker-compose logs -f alerter
and docker-compose logs -f health-checker
. By this you can see the different components starting up. If you have set-up telegram/slack commands we suggest that you enter the command /status
(telegram) or /panicstatus
(slack) to check that all PANIC components are running. If you want to check that every PANIC component is up and running without any issue we suggest that you check that all the logs inside panic/alerter/logs
have no errors.
After PANIC is up and running you can visualise node metrics and alerts using PANIC UI at https://{UI_ACCESS_IP}:3333
, or at https://localhost:3333
if you're running it locally.
For more information regarding PANIC UI, click here.
Congratulations you should have PANIC up and running!
Note: This needs to be done on every host machine that you want the system metrics monitored and alerted on.
GitHub link to most recent version of Node Exporter we support.
sudo useradd --no-create-home --shell /bin/false node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
tar -xzvf node_exporter-0.18.1.linux-amd64.tar.gz
sudo cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo rm node_exporter-0.18.1.linux-amd64 -rf
sudo nano /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter
Check if the installation was successful by checking if {NODE_IP}:{PORT}/metrics is accessible from a web browser.
/newbot
command and fill in the requested details, including a bot name and username.111111:AAA-AAA111111-aaaaa11111
.t.me/<username>
to your new bot given by BotFather and press Start.api.telegram.org/bot<token>/getUpdates
, replacing <token>
with the bot's API token. This gives a list of the bot's activity, including messages sent to the bot./start
command to the bot should do the trick. Take a note of the "id"
number in the "chat"
section of this message.At the end, you should have:
Create New App
. The steps below are to be used to set-up the app, which includes gathering the app-level token, the bot token, and the channel ID:
From an app manifest
option in the pop-up window.workspace
which contains the target channel.Next
followed by Create
.App-Level Tokens
section and click Generate Token and Scopes
.Token Name
(this is just a reference to the token which can be set to anything), add the connections:write
scope, and click Generate
.Install to Workspace
, followed by Allow
.Bot User OAuth Token
.PANIC Notifications
app to the target channel by typing /add
within the channel and selecting Add apps to this channel
.Open app details
and from there + add app to channel
.Open channel details
, and take note of the Channel ID (found at the bottom).At the end, you should have:
At the end, you should have:
If you wish to explore more advanced features, PANIC also supports configurable TwiML; instructions which can re-program Twilio to do more than just call numbers. By default, the TwiML is set to reject calls as soon as the recipient picks up, without any charges. This can be re-configured from the twilio section of the .env
file to either a URL or raw TwiML instructions.
+ Add new services
button on the right side of the pageIntegration Settings
select Use our API directly
and choose Events API v2
,Add Service
Integrations
tab and take note of the (i)Integration Key
.At the end, you should have:
This will be used later on in the installation procedure.
Note You can also install an app for Android / iPhone as well as setup your phone number to receive alerts.
Integrate with Jira and your monitoring tools
on your home page.API
integration is selectedSave integrations
Now, go to the integrations page and explore
API Key
.At the end, you should have:
This will be used later on in the installation procedure.
Note You can also install the Opsgenie app for Android / iPhone as well as setup your phone number to receive calls.
Apply your own SSL certificate signed by a certificate authority. The SSL certificate (cert.pem) and the key (key.pem) should be placed in the panic/certificates
folder, and they should replace the existing dummy files. Note that these dummy files were given just for convenience as the API (PANIC UI Backend) server won't start without them, however, for maximum security these must be replaced.
We suggest reading this for more details on SSL certificates, and how to generate a self signed certificate in case you do not want to obtain a certificate signed by a certificate authority. However, for maximum security, the self signed certificate option is not recommended.
To run the tests for the alerter component within PANIC, do the following:
docker-compose kill # To stop any running containers (to avoid conflicts)
docker-compose -p panic-tests -f docker-compose-tests.yml up --build -d # To build the tests container
docker-compose -p panic-tests -f docker-compose-tests.yml logs test-suite # To see the result of the tests
docker-compose -p panic-tests -f docker-compose-tests.yml kill # To remove test environment
To run the tests for the API component within PANIC, navigate to the
/api
directory and do the following:
npm install # Install API project dependencies
npm test # Run API unit tests
To run the tests for the Substrate API component within PANIC, navigate to the
/substrate-api
directory and do the following:
npm install # Install API project dependencies
npm test # Run API unit tests
To run the tests for the UI component within PANIC, navigate to the
/ui
directory and do the following:
npm install # Install UI project dependencies
npm test # Run UI E2E and unit tests
On top of the additional work that we will put in ourselves to improve and maintain the tool, any support from the community through development will be greatly appreciated.
Simply VC runs highly reliable and secure infrastructure in our own datacentre in Malta, built with the aim of supporting the growth of the blockchain ecosystem. Read more about us on our website and Twitter: