Invalid fsstat used/free disk space docker wsl2

[cdalexndr]

Metricbeat reports invalid free size. Total size reported is total disk size (correct), but used size seems to be only docker related files, not rest of files on disk. So the resulting free space is very large, but in reality, only ~70GB are free.

ES document:

Discover
metricbeat-7.9.3-2020.11.25-000001#mULvAHYBYfcEbWf2DZ1R

{
  "_index": "metricbeat-7.9.3-2020.11.25-000001",
  "_type": "_doc",
  "_id": "mULvAHYBYfcEbWf2DZ1R",
  "_score": 1,
  "_source": {
    "@timestamp": "2020-11-25T19:43:54.215Z",
    "event": {
      "duration": 280200,
      "dataset": "system.fsstat",
      "module": "system"
    },
    "metricset": {
      "name": "fsstat",
      "period": 60000
    },
    "service": {
      "type": "system"
    },
    "system": {
      "fsstat": {
        "total_files": 16777216,
        "total_size": {
          "total": 269490393088,
          "free": 263948574720,
          "used": 5541818368
        },
        "count": 1
      }
    },
    "ecs": {
      "version": "1.5.0"
    },
    "host": {
      "name": "49dfe54a424c"
    },
    "agent": {
      "type": "metricbeat",
      "version": "7.9.3",
      "hostname": "49dfe54a424c",
      "ephemeral_id": "5565dcf7-e371-4655-8d8c-ae5026fa9cc7",
      "id": "0d988131-c303-49c5-9e64-ed8caa5ed884",
      "name": "49dfe54a424c"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-11-25T19:43:54.215Z"
    ]
  }
}

metricbeat.yml:

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
metricbeat.modules:
  - module: docker
    metricsets: ["container", "cpu", "diskio", "memory", "info", "network"]
    cpu.metrics: ["normalized_percentages"]
    hosts: ["unix:///var/run/docker.sock"]
    period: 60s
  - module: system
    metricsets:
      - cpu
      - memory
      - network
      - uptime
      - diskio
      - fsstat
    period: 60s
    cpu.metrics: ["normalized_percentages"]

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

docker service:

metricbeat:
    image: store/elastic/metricbeat:7.9.3
    restart: unless-stopped
    mem_limit: 64m
    volumes:
      - metricbeat-data:/usr/share/metricbeat/data
      - ./config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    user: root #docker sock access
    cap_add:
      - SYS_PTRACE
      - DAC_READ_SEARCH
    labels:
      app: metricbeat
    command: --strict.perms=false -e

For confirmed bugs, please report:

• Version: 7.9.3
• Operating System: docker windows 2.5.0.1
• Discuss Forum URL:
• Steps to Reproduce:

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[fearful-symmetry]

@cdalexndr So, just to be clear, you're running docker inside WSL? What other metrics are you looking at to get the correct disk usage?

[cdalexndr]

Yes, docker has WSL2 based engine enabled, with Alpine image. This invalid disk usage is reported on my developer machine. I can see the correct disk usage in windows explorer (outside of docker).

[fearful-symmetry]

but used size seems to be only docker related files

So, if you want to monitor a filesystem from inside docker, it needs to be mounted inside the container. There's an example here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html#monitoring-host

[cdalexndr]

Added hostfs and now the total space has doubled (invalid). Used space still invalid.

docker compose service:

  metricbeat:
    image: store/elastic/metricbeat:7.9.3
    mem_limit: 64m
    depends_on:
      - kibana #to register dashboards
    volumes:
      - metricbeat-data:/usr/share/metricbeat/data
      - ./config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/hostfs:ro
    user: root #docker sock access
    cap_add:
      - SYS_PTRACE
      - DAC_READ_SEARCH
    labels:
      app: metricbeat
    command: --strict.perms=false -e -system.hostfs=/hostfs

ES metricbeat doc:

{
  "_index": "metricbeat-7.9.3-2020.12.29-000002",
  "_type": "_doc",
  "_id": "I3Z_F3cBDXAhja8VSPy9",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-01-18T21:55:52.796Z",
    "host": {
      "name": "ff2cdac595b4"
    },
    "agent": {
      "name": "ff2cdac595b4",
      "type": "metricbeat",
      "version": "7.9.3",
      "hostname": "ff2cdac595b4",
      "ephemeral_id": "5796531f-290a-474e-a1ff-106dc7eeed87",
      "id": "0d988131-c303-49c5-9e64-ed8caa5ed884"
    },
    "system": {
      "fsstat": {
        "total_size": {
          "free": 519909797888,
          "used": 19788922880,
          "total": 539698720768
        },
        "count": 13,
        "total_files": 33554432
      }
    },
    "event": {
      "duration": 3637300,
      "dataset": "system.fsstat",
      "module": "system"
    },
    "metricset": {
      "name": "fsstat",
      "period": 60000
    },
    "service": {
      "type": "system"
    },
    "ecs": {
      "version": "1.5.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-01-18T21:55:52.796Z"
    ]
  },
  "highlight": {
    "metricset.name": [
      "@kibana-highlighted-field@fsstat@/kibana-highlighted-field@"
    ],
    "service.type": [
      "@kibana-highlighted-field@system@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1611006952796
  ]
}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)