Is it possible to monitor the remote container?

[hack3rcon]

Hello, Can this tool only monitor the containers in the host? Can this tool be used as a central monitoring server? I mean to monitor the containers on other hosts with this tool.

Thank you.

[GuyKh]

To monitor remote containers, you need an agent to pull the data from. on your remote host, setup a cadvisor container (similar to the one in docker-compose.yaml). On the prometheus configuration, add a target similar to cadvisor just with the ip of the remote host and the data would be pulled from therr

[hack3rcon]

To monitor remote containers, you need an agent to pull the data from. on your remote host, setup a cadvisor container (similar to the one in docker-compose.yaml). On the prometheus configuration, add a target similar to cadvisor just with the ip of the remote host and the data would be pulled from therr

Hello, Thank you so much for your reply. On the remote host, I created a docker-compose.yml file with the following contents:

services:
  cadvisor:
    container_name: cadvisor
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: "host"
    ports:
      - "8080:8080"
    volumes: 
      - "/:/rootfs:ro"
      - "/var/run:/var/run:ro"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - "/dev/disk/:/dev/disk:ro"
    privileged: true
    devices: 
      - "/dev/kmsg"

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    network_mode: "host"
    ports:
      - "9090:9090"
    volumes: 
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    privileged: true
    depends_on:
      - cadvisor

Then I created a prometheus.yml file with the following contents:

global:
  scrape_interval: 15s #Scrape interval to every 15 seconds.
  evaluation_interval: 15s #Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["localhost:8080"]

Now, in the targets section on the monitoring server, should I enter the IP address of the remote server? Something like the below:

global:
  scrape_interval: 15s #Scrape interval to every 15 seconds.
  evaluation_interval: 15s #Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    static_configs:
      - targets: ["localhost:9090"]
      - targets: ["Remote_Host:8080"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["localhost:8080"]
      - targets: ["Remote_Host:8080"]

[GuyKh]

@hack3rcon Wait, no. You have a local host (i.e. with the stack of the repo) and a remote host you want to monitor; You need to add to the remote host only the cadvisor docker-compose part.

The middle part is redundant.

Then, on your local prometheus.yaml file, you can add one of the following:

new scraper

  - job_name: "remote-cadvisor"
    static_configs:
      - targets: ["remote_host:8080"]

use a scraper that scrapes from two targets:

  - job_name: "cadvisor"
    static_configs:
      - targets: ["localhost:8080", "remote_host:8080"]

Save, restart prometheus and you're good to go

[hack3rcon]

Hello, Thanks again. On my monitoring server, I have two files with the following contents:

docker-compose.yml

services:
  cadvisor:
    container_name: cadvisor
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: "host"
    ports:
      - "8080:8080"
    volumes: 
      - "/:/rootfs:ro"
      - "/var/run:/var/run:ro"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - "/dev/disk/:/dev/disk:ro"
    privileged: true
    devices: 
      - "/dev/kmsg"

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    network_mode: "host"
    ports:
      - "9090:9090"
    volumes: 
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    privileged: true
    depends_on:
      - cadvisor

And:

prometheus.yml

global:
  scrape_interval: 15s #Scrape interval to every 15 seconds.
  evaluation_interval: 15s #Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: "prometheus"
    # metrics_path defaults to '/metrics'
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["localhost:8080","Remote_Host:8080"]

On the remote server, the contents of the docker-compose.yml file are as follows:

services:
  cadvisor:
    container_name: cadvisor
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: "host"
    ports:
      - "8080:8080"
    volumes: 
      - "/:/rootfs:ro"
      - "/var/run:/var/run:ro"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - "/dev/disk/:/dev/disk:ro"
    privileged: true
    devices: 
      - "/dev/kmsg"

When I go to the Monitoring_Server:8080 address, I only see the containers on the Monitoring Server! What is wrong?

[GuyKh]

8080 is the port of the cadvisor service, you have one on the monitoring and one on the remote. 9090 is the port of Prometheus service, which fetches the data from both. All sounds correct

[hack3rcon]

Hi, So, on the remote server I don't need Prometheus. Is it correct?

[GuyKh]

Correct

[hack3rcon]

Hi, Thanks again. In the dashboard section and in the Docker Containers section I see a list of services, can I segment them by host name?

[GuyKh]

It's a bit more complicates as the dashboards are designed to work with a single cadvisor instance - and as such they squash the multiple hosts into a single graph.

You should edit every graph and edit the view

[hack3rcon]

Thank you so much.

[GuyKh]

@hack3rcon anytime :)