Vision

Description

Vision is a stack for monitoring and logging. It provides :

• Elasticsearch web interface : http://xxx:9200
• Kibana web interface : http://xxx:9393
• Grafana web interface : http://xxx:9999/
• InfluxDB web interface : http://xxx:8083
• Heka dashboard : http://xxx:4352

Installation

• Download and install a release :

  $ curl https://github.com/nlamirault/vision/releases/download/x.y.z/vision-x.y.z-linux.tar.gz
  $ tar zxvf vision-x.y.z
  $ cd vision-x.y.z
  $ ./init.sh

• Start services using Docker Compose:

  $ docker-compose up -d

• Start services using SystemD:

  $ cp -r vision-*.service /lib/systemd/system/

Usage

Monitoring servers : Elasticsearch/Kibana/Beats

• Install Topbeat and Packetbeat

• Launch Elasticsearch and Kibana services

• Loading the templates into Elasticsearch

  $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
      -d@beats/topbeat.template.json
  
  $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
      -d@beats/packetbeat.template.json

• Running topbeat metrics :

  $ topbeat -c beats/topbeat.yml

• Running packetbeat metrics :

  $ packetbeat -c beats/packetbeat.yml

• Testing the installation:

  $ curl -XGET 'http://localhost:9200/topbeat-*/_search?pretty'
  $ curl -XGET 'http://localhost:9200/packetbeat-*/_search?pretty'

• Loading Kibana dashboards:

  $ curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.0.0.tar.gz
  $ tar zxvf beats-dashboards-1.0.0.tar.gz
  $ cd beats-dashboards-1.0.0
  $ ./load.sh

• Then open the Kibana website (http://localhost:9393), then select Topbeat index, and open Topbeat dashboard. Do same with Packetbeat index and dashboard.

Monitoring servers : Telegraf/InfluxDB/Grafana

• Install Telegraf

• Launch InfluxDB and Grafana services

• Running telegraf metrics :

  $ telegraf -config telegraf/telegraf.conf

• Then open the Grafana dashboard (http://localhost:9191) and import the Vision Telegraf dashboard from (grafana/grafana-telegraf.json)

• You could explore metrics into the InfluxDB UI on http://localhost:8083 with the query :

  SHOW MEASUREMENTS

Monitoring servers : Prometheus/Grafana

• Install Prometheus and Prometheus node exporter

• Launch services :

  $ ./prometheus -config.file=prometheus/vision.yml
  $ ./node_exporter

• Check Prometheus installation on : http://localhost:9090 and http://localhost:9090/consoles/node.html

• hen open the Grafana dashboard (http://localhost:9191) and import the Vision Prometheus dashboard from (grafana/grafana-prometheus.json)

Log analysis (Elasticsearch/Filebeat/Kibana)

• Install Filebeat

• Launch Elasticsearch and Kibana services

• Loading the Index Template into Elasticsearch

  $ curl -XPUT 'http://localhost:9200/_template/packetbeat' \
      -d@beats/filebeat.template.json

• Running filebeat metrics :

  $ filebeat -c beats/filebeat.yml

SystemD

You could use services files to launch Vision monitoring tools using SystemD.

$ sudo cp systemd/*.service /lib/systemd/system/
$ sudo systemctl enable vision-telegraf
$ sudo systemctl enable vision-topbeat
$ sudo systemctl enable vision-packetbeat

Development

• Creates a virtual machine called vision-dev for the development environment :

  $ ./docker-machine create -d virtualbox vision-dev
  $ eval "$(./docker-machine env vision-dev)"

• Check vision machine runnning :

  $ ./docker-machine ls

• Launch vision :

  $ ./docker-compose up

• Open your browser and navigate to the IP address associated with the vision virtual machine :

  $ ./docker-machine ip

• To see which environment variables are available to the web service, run:

  $ ./docker-compose run web env

Deployment

With our app running locally, we can now push this exact same environment to a cloud hosting provider with Docker Machine

Set your credentials in your environment :

$ source XXXXXXX.sh

Deploy a new instance :

$ docker-machine -D create -d digitalocean \
    --digitalocean-access-token $DIGITALOCEAN_TOKEN \
    vision-prod

Now we have two Machines running, one locally and one on Digital Ocean:

$ docker-machine ls
NAME            ACTIVE     DRIVER         STATE     URL
vision-dev      *          virtualbox     Running   tcp://w.x.y.z:2376
vision-prod                digitalocean   Running   tcp://a.b.c.d:2376

Set vision-prod as the active machine and load the Docker environment :

$ ./docker-machine active vision-prod
$ eval "$(./docker-machine env vision-prod)"

Finally, let's build the application in the Cloud :

$ ./docker-compose build
$ ./docker-compose up -d

Support

Feel free to ask question or make suggestions in our [Issue Tracker][].

License

See LICENSE for the complete license.

Changelog

A changelog is available.

Contact

Nicolas Lamirault nicolas.lamirault@gmail.com