Native DockerSwarm Support

[roidelapluie]

Dear community,

The coming prometheus release, Prometheus 2.20, contains native support for Docker Swarm.

Please try it and provide us with your feedback. This is highly appreciated.

https://prometheus.io/docs/prometheus/2.20/configuration/configuration/#dockerswarm_sd_config

[joonas-fi]

Thanks for reaching out, and thanks for adding Swarm support to Prometheus!

I have added notification to top of this project's README with a link to this issue. I'll close this as it's not an issue in promswarmconnect per se, but the conversation can of course go on. :)

In the near term I don't have time to try Prometheus's support, but when I get the time I will definitely test it out and switch to it, if it has everything I need.

[roidelapluie]

Maybe linking to the guide would also be great and more user friendly: https://prometheus.io/docs/guides/dockerswarm/

[joonas-fi]

You're right 👍 , fixed

[pschichtel]

I'm working on migrating a production application from promswarmconnect to the native support. I could share before/after configs as a reference for others if that is of any interest.

[roidelapluie]

Please note that the 2.21 of prometheus will also expose tasks/service without explicit exposed port. That can be tested with the docker image prom/prometheus:master until the release (planned end of august)

[pschichtel]

Had another shot at it today, we have a lot of services that don't have exposed ports. I have other priorities at the moment, but I'll revisit this with 2.21

[pschichtel]

I actually came around and migrated to the native support now, using prometheus 2.24:

scrape_configs:
  - job_name: 'general_swarm_tasks'
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        filters:
          - name: desired-state
            values: [running]
    relabel_configs:
      # only tasks that specify a metrics port
      - source_labels: [ __meta_dockerswarm_service_label_metrics_port ]
        regex: .+
        action: keep
      # only tasks that are in reachable networks
      - source_labels: [ __meta_dockerswarm_network_name ]
        regex: <networks regex>
        action: keep
      # drop tasks with missing metadata
      - source_labels: [ __meta_dockerswarm_task_id ]
        regex: .+
        action: keep
      - source_labels: [ __meta_dockerswarm_service_name ]
        regex: .+
        action: keep
      # unpack __address__
      - source_labels: [ __address__ ]
        regex: '([^:]+)(?::\d+)?'
        target_label: __tmp_metrics_host
      - source_labels: [ __address__ ]
        regex: '(?:[^:]+):(\d+)'
        target_label: __tmp_metrics_port
      # map metrics-* labels
      - source_labels: [ __meta_dockerswarm_service_label_metrics_port ]
        target_label: __tmp_metrics_port
      - source_labels: [ __meta_dockerswarm_service_label_metrics_host ]
        regex: (.+)
        target_label: __tmp_metrics_host
      - source_labels: [ __meta_dockerswarm_service_label_metrics_path ]
        regex: (.+)
        target_label: __metrics_path__
      - source_labels: [ __meta_dockerswarm_service_label_metrics_scheme ]
        regex: (.+)
        target_label: __scheme__
      - regex: __meta_dockerswarm_service_label_metrics_param_(.+)
        action: labelmap
        replacement: '__param_${1}'
      # map required labels
      - source_labels: [ __meta_dockerswarm_service_name ]
        target_label: job
      - source_labels: [ __meta_dockerswarm_task_id ]
        target_label: instance
      - source_labels: [ __tmp_metrics_host, __tmp_metrics_port ]
        separator: ':'
        target_label: __address__
      - source_labels: [ __meta_dockerswarm_service_label_metrics_address ]
        regex: (.+)
        target_label: __address__
      # map extra labels
      - source_labels: [ __meta_dockerswarm_service_label_com_docker_stack_namespace ]
        target_label: stack
      - source_labels: [ __meta_dockerswarm_service_name ]
        target_label: service
      - source_labels: [ __meta_dockerswarm_task_container_id ]
        target_label: container
      - source_labels: [ __meta_dockerswarm_node_hostname ]
        target_label: node

This works fairly similar to what I did previously with promswarmconnect. Instead of using the env var METRICS_ENDPOINT you have a couple of metrics-* labels. All services that have at least the label metrics-port will be kept, other labels can be used to overwrite defaults.

[joonas-fi]

Thanks for taking the time to give an example of what works! :+1: