Uninitialized userSet in metricbeatInit causing missing process metrics

[kevinsmithwrs]

• Version: Metricbeat 8.6 (but applicable to 8.2+ I believe)
• Operating System: Linux, k8s + docker
• Steps to Reproduce:
• Enable system module process metricset, set hostfs: /hostfs, do not set --system.hostfs=/hostfs CLI flag.

I think the Initialization of userSet is missing in metricbeatInit in metricbeat/internal/sysinit/init.go. My understanding is we shouldn't need to set the deprecated --system.hostfs=/hostfs parameter when launching metricbeat, in favour of using the hostfs settings in the individual module metricsets (process, linux, etc) in 8.x. If I do this however, we end up missing some process metrics. The cause is the uninitialized userSet bool, (which then defaults to false) . This is incorrect if the hostfs module setting has been set. In a docker/kubernetes environment, the below check in metricbeat/module/system/process/process.go uses this (incorrect) false for sys.IsSet() resulting in the value for LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE being used, which is set to '/' in the elastic metricbeat docker images thereby overriding the cgroup heirarchy.

As a workaround, I can get all process metrics to come out properly by setting the CLI flag, the proper /hostfs settings in the module, and setting LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE to "" when launching the metricbeat container.

metricbeat/module/system/process/process.go:

if !sys.IsSet() {
    override, isset := os.LookupEnv("LIBBEAT_MONITORING_CGROUPS_HIERARCHY_OVERRIDE")
        if isset {
            m.stats.CgroupOpts.CgroupsHierarchyOverride = override
        }
}

Fix needed in metricbeat/internal/sysinit/init.go:

// Deal with the legacy configs available to metricbeat
 func metricbeatInit(base mb.BaseModule, modulePath string, moduleSet bool) (mb.Module, error) {
        var hostfs = modulePath
-       var userSet bool
+       var userSet = moduleSet
        // allow the CLI to override other settings
        if hostfsCLI != nil && *hostfsCLI != "" {
                cfgwarn.Deprecate("8.0.0", "The --system.hostfs flag will be removed in the future and replaced by a config value.")
    once.Do(func() {
        InitModule(hostfs)
    })
    return &Module{BaseModule: base, HostFS: hostfs, UserSetHostFS: userSet}, nil

[botelastic[bot]]

This issue doesn't have a Team:<team> label.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!