Nodies Monitoring is a customizable and extensible monitoring solution for monitoring host machine metrics, container metrics, node status and node logs.
Our project is split into a server (monitoring) stack, and a client (exporter) stack.
Please see architecture documentation for a deeper analysis of the server/client stack.
TBD instructions on how to contribute to this project.
NOTE: These dependencies are required on both the server and client stack.
Note: Tested and recommended installation on Ubuntu 22.04.1 LTS host OS
Clone the repository
sudo git clone https://github.com/baaspoolsllc/nodies_monitoring.git
Change the current working directory to the cloned repository
cd nodies_monitoring
Install python3-pip
apt install python3-pip
Install requirements
pip3 install -r requirements.txt
NOTE: This step is required on both the server and the client stack. When setting up this stack, you should follow the principles of least privilege when allowing users to access your exposed ports by setting up network or host based firewalls.
sudo python3 setup.py
to generate a template settings.yml.new
file.clients.promtail.loki_endpoint
with the ip address of the host that will run the monitoring stack (loki, grafana, minio, prometheus, alertmanager)server.prometheus.exporter_endpoint
with the endpoints any services of the exporter_stack (blockchain_exporter, cadvisor, node_exporter, promtail)server.alerts.contactpoints
with the contact points of your alert recievers to send grafana-managed alerts to.clients.promtail.log_root_path
with the root path of your nginx logs.settings.yml.new
to settings.yml
On your monitoring server:
sudo python3 setup.py
settings.yml.new
file
sudo vi settings.yml.new
settings.yml.new
to settings.yml
sudo mv settings.yml.new settings.yml
chains.json
file - Mandatory
echo "[]" > clients/bcexporter/config/chains.json
sudo python3 setup.py
cd server && docker compose up -d
http://<host>:<port>
e.g http://192.168.1.2:3000Default username and password for Grafana is admin:admin
NOTE: Sudo is required here to set some folder permissions
On your exporter server:
chains.json
(./templates/chains.json) file. An example file is provided for you in chains.example.json
.Ensure that your access logs are formatted with the following format:
log_format json_combined escape=json
'{'
'"region":"us-east-1",'
'"time_local":"$time_local",'
'"request":"$scheme://$host$request_uri",'
'"method":"$request_method",'
'"protocol":"$server_protocol",'
'"status":"$status",'
'"request_body":"$request_body",'
'"body_bytes_sent":"$body_bytes_sent",'
'"http_referrer":"$http_referer",'
'"http_user_agent":"$http_user_agent",'
'"upstream_addr":"$upstream_addr",'
'"upstream_status":"$upstream_status",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_header_time":"$upstream_header_time",'
'"request_time":"$request_time"'
'}';
Note: Replace the region
value with a unique region identifer, i.e us-east-1
, eu-central-1
, etc based off where your nginx server is located.
Run setup.py again to save your configurations and set folder permissions.
sudo python3 setup.py
Edit the settings.yml.new
file
sudo vi settings.yml.new
Rename the file settings.yml.new
to settings.yml
sudo mv settings.yml.new settings.yml
Create the chains.json
file - Mandatory
echo "[]" > clients/bcexporter/config/chains.json
Execute the setup.py once again
sudo python3 setup.py
Modify clients.promtail.log_root_path
in the settings.yml
file to reflect your nginx log directory
Change directory into the clients subfolder, and boot up all client services
cd clients && docker compose up -d
NOTE: Sudo is required here to set some folder permissions
setup.py has an optional CLI flag that allows control over which clients are ran on the exporter stack. For example, if you don't want to run log aggregation, then you can disable the log shipper promtail
.
sudo python3 setup.py --clients blockchain_exporter cadvisor node_exporter
Default datasources have been provisioned in ./templates/datasources
To add additional datasources, please refer to grafana datasource documentation
Default dashboards have been provisioned in ./server/grafana/dashboards
To add additional dashboards, please refer to grafana dashboard documentation
Default alerting has been provisioned in ./server/grafana_provisioning/alerting
To add additional alerting, please refer to grafana alerting documentation
For any inquiries, please reach out to PoktBlade(PoktBlade#5970) or poktdachi(dachi#0005) on the pokt discord