Host metrics not including filesystem when specifying `/` as mount point

kvaggelakos commented 2 months ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

My vector seems to have an issue when I specify the mount point to be /. The host metrics appear for the filesystem when I specify * instead.

So in the working case (specifying *), it looks like this:

host_filesystem_free_bytes{collector="filesystem",filesystem="virtiofs",host="592075aabe13",mountpoint="/"} = 21131915264

In the broken case there are no host_filesystem_free_bytes metrics.

Configuration

[sources.host_metrics]
type = "host_metrics"
collectors = ["cpu", "filesystem", "memory", "network"]
filesystem.mountpoints.includes = ["/"]

Version

0.40.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

This is the only thing that seemed relevant: https://github.com/vectordotdev/vector/issues/6923

jszwedko commented 1 month ago

Thanks for this report! As a workaround you could collect metrics for all mount points and then filter afterwards with a filter transform.

Can you share more details about the environment Vector is running in? I tried to reproduce on MacOS but I wasn't able to. I used this config:

sources:
  source0:
    collectors:
    - cpu
    - filesystem
    - memory
    - network
    namespace: host
    scrape_interval_secs: 1
    type: host_metrics
    filesystem:
      mountpoints:
        includes: ["/"]
sinks:
  sink0:
    inputs:
    - source0
    type: console
    encoding:
      codec: json

And I saw:

$ ❯ vector-0.40.0 --config /tmp/tmp.yaml | grep filesystem_free_bytes
2024-09-06T22:34:08.789705Z  INFO vector::app: Log level is enabled. level="info"
2024-09-06T22:34:08.790521Z  INFO vector::app: Loading configs. paths=["/tmp/tmp.yaml"]
2024-09-06T22:34:08.791836Z  INFO vector::topology::running: Running healthchecks.
2024-09-06T22:34:08.791928Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-06T22:34:08.791982Z  INFO vector: Vector has started. debug="false" version="0.40.0" arch="aarch64" revision=""
2024-09-06T22:34:08.792004Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
{"name":"filesystem_free_bytes","namespace":"host","tags":{"collector":"filesystem","device":"/dev/disk3s1s1","filesystem":"apfs","host":"xxx","mountpoint":"/"},"timestamp":"2024-09-06T22:34:08.793033Z","kind":"absolute","gauge":{"value":28292816896.0}}

jszwedko commented 1 month ago

I also tried on an Ubuntu 20.04 system and it worked fine too:

➜  vector git:(master) ✗ vector --config /tmp/tmp.yaml | grep filesystem_free_bytes
2024-09-06T22:35:18.391018Z  INFO vector::app: Log level is enabled. level="info"
2024-09-06T22:35:18.393318Z  INFO vector::app: Loading configs. paths=["/tmp/tmp.yaml"]
2024-09-06T22:35:18.399703Z  INFO source{component_kind="source" component_id=source0 component_type=host_metrics}: vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root.
2024-09-06T22:35:18.399798Z  INFO source{component_kind="source" component_id=source0 component_type=host_metrics}: vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root.
2024-09-06T22:35:18.400462Z  INFO vector::topology::running: Running healthchecks.
2024-09-06T22:35:18.400638Z  INFO vector::topology::builder: Healthcheck passed.
2024-09-06T22:35:18.400915Z  INFO vector: Vector has started. debug="true" version="0.41.0" arch="x86_64" revision=""
{"name":"filesystem_free_bytes","namespace":"host","tags":{"collector":"filesystem","device":"overlay","filesystem":"overlay","host":"vector3","mountpoint":"/"},"timestamp":"2024-09-06T22:35:18.401654752Z","kind":"absolute","gauge":{"value":148724862976.0}}

🤔

I think more details about your environment would be helpful like OS version.

kvaggelakos commented 1 month ago

Thanks for looking into this.

My host uses kata containers. The filesystem documentation is here: https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-virtio-fs-with-kata.md

This is the results of df -h.

Filesystem      Size  Used Avail Use% Mounted on
none             20G  266M   20G   2% /
tmpfs            64M     0   64M   0% /dev
tmpfs           350M     0  350M   0% /sys/fs/cgroup
shm             350M     0  350M   0% /dev/shm
kataShared      7.3T  613G  6.7T   9% /etc/hosts

The container is based on ubuntu 22.

uname -r
6.1.38

vectordotdev / vector