dicovery.docker only returns containers with attached network

[regner]

What's wrong?

dicovery.docker only returns containers with an attached network. This presents two problems:

• Not all of our running containers have a network attached, batch jobs for example
• Containers run from Nomad with CNI bridge networking don't appear either

This appears to come from here: https://github.com/prometheus/prometheus/blob/main/discovery/moby/docker.go#L207

The disovery.docker component uses the Prometheus Moby Docker discovery tooling. That tooling assumes you're looking for containers with an exposed port for metrics collection. So even disregarding the second point above, and Nomad wasn't involved at all, containers that run simple batch jobs and don't have a network won't be discovered.

Steps to reproduce

Run a container with no network.

The following examples come from docker inspect ....

Here are the results of a container that is found and added:

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": ".......",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/38ceec05df8f",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "none": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "......",
                    "EndpointID": "......",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }

And here is the same section from a container which does not appear in the discovery targets:

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }

System information

Ubuntu

Software version

v0.35.4

Configuration

argument "write_pipeline" {
  optional = false
}

discovery.docker "local" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "docker" {
    targets = discovery.docker.local.targets

    rule {
    action       = "replace"
        target_label = "job"
    replacement  = "loki.source.docker"
    }

    rule {
    action       = "replace"
        target_label = "instance"
    replacement  = constants.hostname
    }
}

loki.source.docker "local" {
  host    = "unix:///var/run/docker.sock"
  targets = discovery.relabel.docker.output

  forward_to = [
    argument.write_pipeline.value,
  ]
}

Logs

No response

[regner]

I took a look at the nomad discovery component as well, and it also only returns targets which have services attached to them.

https://github.com/prometheus/prometheus/blob/main/discovery/nomad/nomad.go#L180

[rfratto]

Thank you the report.

Traditionally the service discovery mechanisms are intended for metrics collection, which is why the logic around looking for network addresses to collect from is involved.

I guess you're only targeting use cases where a connection to the container isn't required (i.e., loki.source.docker), right?

This should be something we'd be able to support, but we'll need to discuss with the Prometheus folks upstream to add the capability.

[regner]

I guess you're only targeting use cases where a connection to the container isn't required (i.e., loki.source.docker), right?

Correct.

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[regner]

@rfratto hey, just wanted to follow up on this. Were you able to open a discussion with the Prometheus team? Happy to help if I can.

[rfratto]

@regner Thanks for the reminder, and sorry for the delay. I opened an upstream issue here: https://github.com/prometheus/prometheus/issues/12965

[rfratto]

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)