Metricbeat host network monitor over docker swarm

[gianpietro1]

Hello,

A containerized Metricbeat requires to use network_mode: host to be able to detect and monitor the host's interfaces, however, if deploying a stack in swarm mode with a version 3 compose file, this mode is not available (see: https://docs.docker.com/compose/compose-file/#network_mode)

Would like to request the Elastic team to, if possible, implement an enhancement to overcome this limitation.

Thanks, Gianpietro

[jsoriano]

Hi @gianpietro1,

I think this is more a limitation on docker swarm than on metricbeat, and indeed there is an open discussion in this project about that. Other container orchestrators as Kubernetes support host network mode.

This mode is required to access the same network namespace as the host, what is needed to monitor its network interfaces, so I don't think it is possible to overcome this limitation on our side, this would be breaking namespaces boundaries.

One thing you can do if you want to monitor host network interfaces in your docker swarm cluster is to deploy metricbeat directly on all your nodes using some other configuration management solution instead of deploying them on swarm.

I am going to close this issue as I don't think we can do much here, if you have more doubts about metricbeat configuration or deployment options, please use the discuss forum.

[jsoriano]

Hi @gianpietro1 again,

I replied too fast :slightly_smiling_face:, after talking with @exekias offline about this issue he pointed that we could indeed obtain the information we use mounting the host proc filesystem into the container. We already do something like that to collect process information from the host. Here it'd be a bit different because we use a different library, but it'd be definitely possible. I'm reopening this.

[OlivierCuyp]

@gianpietro1 Did you try something like this ?

version: '3.3'

services:
  web:
    image: "nginx:alpine"
    networks:
      - outside

networks:
  outside:
    external:
      name: "host"

From: https://github.com/moby/moby/issues/25873#issuecomment-319109840

[OlivierCuyp]

@jsoriano we tested the solution I proposed it works like a charm. So I believe you might close this issue.

[jsoriano]

Great, thanks for trying it and letting us know!

[aldencolerain]

@jsoriano @exekias Would you please consider re-opening this feature request? In docker swarm its possible to join a container to the host network, like @OlivierCuyp mentioned, but it makes practical deployment very difficult for a number of reasons. Most critically the container is unable to join any overlay networks, so deploying in this manner means your elastic search instance is not discoverable by metricbeat on the swarm if it is secured behind a standard swarm overlay network.

[jsoriano]

Ok, let's reopen it to get the host network information from procfs even when metricbeat is not running from the host network.

[OlivierCuyp]

@aldencolerain I don't know your infra but in our case our nodes have a private & a public interface. For Elasticsearch we mapped the port 9200 & 9300 on the private ips, which is only accessible through our VPN and for other nodes of the cluster.

We put an "elasticsearch" label on the nodes dedicated for Elasticsearch & a service constraint on the elasticsearch service like this:

...
    placement:
        mode: global
        constraints:
          - node.labels.elasticsearch == true

We also mapped the private ips in our DNS (for the sake of flexibility) like this:

...
elastic1.my-company.com.  A  10.0.0.1
elastic2.my-company.com.  A  10.0.0.2
elastic3.my-company.com.  A  10.0.0.3

Then in the metricbeat configuration you can just add it like this:

...
output.elasticsearch:
  hosts: ["elastic1.my-company.com:9200", "elastic3.my-company.com:9200", "elastic3.my-company.com:9200"]

This is not perfect but it works. Your main issue, comes from Docker because you can't be part of the host network and an overlay network at the same time. I don't really see how you could mix both network interfaces. And I don't how you collect metrics from host network not being part of it.

I hope, it helped.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[jsoriano]

This issue has reappeared in discuss, reopening.

https://discuss.elastic.co/t/metricbeat-docker-network-summary-does-not-work-from-within-a-container/339226

[jerrac]

For the record, if possible, I'd like the docker module to support pulling info from a bind mounted host directory. Something like what the system module does.

Thanks for reopening this. :)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[jerrac]

To keep this open...

The current config from main is: https://github.com/elastic/beats/blob/af33fad4d50bbe7348a43e613e11c4953a82d939/metricbeat/metricbeat.reference.yml#L241-L275

As far as I can tell, that's the same config we had over a year ago that didn't support getting metrics from the host.

I haven't tested things, though, so if this is fixed, could we get an update on the topic?

Thanks.