Nodies Monitoring

Nodies Monitoring is a customizable and extensible monitoring solution for monitoring host machine metrics, container metrics, node status and node logs.

Our project is split into a server (monitoring) stack, and a client (exporter) stack.

Please see architecture documentation for a deeper analysis of the server/client stack.

Want to Contribute?

TBD instructions on how to contribute to this project.

Installation

NOTE: These dependencies are required on both the server and client stack.

Note: Tested and recommended installation on Ubuntu 22.04.1 LTS host OS

Dependencies

Python

Python 3.10.6

Docker

Uninstall existing docker ```bash sudo apt-get remove docker docker-engine docker.io containerd runc ``` Install required packages ```bash sudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release ``` Add docker official GPG key ```bash sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg ``` Setup docker repo ```bash echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ``` Update apt package index ```bash sudo apt-get update ``` Install latest docker ```bash sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin ```

Clone the repository

sudo git clone https://github.com/baaspoolsllc/nodies_monitoring.git

Change the current working directory to the cloned repository

cd nodies_monitoring

Install python3-pip

apt install python3-pip

Install requirements

pip3 install -r requirements.txt

Getting Started

Configuration Setup (REQUIRED)

NOTE: This step is required on both the server and the client stack. When setting up this stack, you should follow the principles of least privilege when allowing users to access your exposed ports by setting up network or host based firewalls.

Run sudo python3 setup.py to generate a template settings.yml.new file.
Update clients.promtail.loki_endpoint with the ip address of the host that will run the monitoring stack (loki, grafana, minio, prometheus, alertmanager)
Update server.prometheus.exporter_endpoint with the endpoints any services of the exporter_stack (blockchain_exporter, cadvisor, node_exporter, promtail)
Update server.alerts.contactpoints with the contact points of your alert recievers to send grafana-managed alerts to.
Update clients.promtail.log_root_path with the root path of your nginx logs.
Rename settings.yml.new to settings.yml

Server setup

On your monitoring server:

Run setup.py again to save your configurations and set folder permissions.
```
sudo python3 setup.py
```
Edit the settings.yml.new file
```
sudo vi settings.yml.new
```
Rename the file settings.yml.new to settings.yml
```
sudo mv settings.yml.new settings.yml
```

Create the chains.json file - Mandatory

echo "[]" > clients/bcexporter/config/chains.json

Execute the setup.py once again
```
sudo python3 setup.py
```
Change directory into the server subfolder, and boot up all server services
```
cd server && docker compose up -d
```
Launch a browser and navigate to http://<host>:<port> e.g http://192.168.1.2:3000

Default username and password for Grafana is admin:admin

NOTE: Sudo is required here to set some folder permissions

Client setup

On your exporter server:

Blockchain exporter setup

Include your blockchain accessible endpoints inside a chains.json (./templates/chains.json) file. An example file is provided for you in chains.example.json.

Log aggregation setup (Promtail)

Ensure that your access logs are formatted with the following format:

log_format json_combined escape=json
'{'
    '"region":"us-east-1",'
    '"time_local":"$time_local",'
    '"request":"$scheme://$host$request_uri",'
    '"method":"$request_method",'
    '"protocol":"$server_protocol",'
    '"status":"$status",'
    '"request_body":"$request_body",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent",'
    '"upstream_addr":"$upstream_addr",'
    '"upstream_status":"$upstream_status",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"upstream_response_time":"$upstream_response_time",'
    '"upstream_connect_time":"$upstream_connect_time",'
    '"upstream_header_time":"$upstream_header_time",'
    '"request_time":"$request_time"'
'}';

Note: Replace the region value with a unique region identifer, i.e us-east-1, eu-central-1, etc based off where your nginx server is located.

Run setup.py again to save your configurations and set folder permissions.
```
sudo python3 setup.py
```
Edit the settings.yml.new file
```
sudo vi settings.yml.new
```
Rename the file settings.yml.new to settings.yml
```
sudo mv settings.yml.new settings.yml
```

Create the chains.json file - Mandatory

echo "[]" > clients/bcexporter/config/chains.json

Execute the setup.py once again
```
sudo python3 setup.py
```
Modify clients.promtail.log_root_path in the settings.yml file to reflect your nginx log directory
Change directory into the clients subfolder, and boot up all client services
```
cd clients && docker compose up -d
```
NOTE: Sudo is required here to set some folder permissions

Run individual clients

setup.py has an optional CLI flag that allows control over which clients are ran on the exporter stack. For example, if you don't want to run log aggregation, then you can disable the log shipper promtail.

sudo python3 setup.py --clients blockchain_exporter cadvisor node_exporter

Customization

Datasources

Default datasources have been provisioned in ./templates/datasources

To add additional datasources, please refer to grafana datasource documentation

Dashboards

Default dashboards have been provisioned in ./server/grafana/dashboards

To add additional dashboards, please refer to grafana dashboard documentation

Alerts

Default alerting has been provisioned in ./server/grafana_provisioning/alerting

To add additional alerting, please refer to grafana alerting documentation

Links/Contact

For any inquiries, please reach out to PoktBlade(PoktBlade#5970) or poktdachi(dachi#0005) on the pokt discord

baaspoolsllc / nodies_monitoring

readme