Procstat plugin - add use_sudo option

[Tulux]

Feature Request

Some metrics such as num_fds or read_bytes are not available or return 0 unless the whole agent is run as root. However running a service as root which may interact with many components or service should be discouraged. Moreover official telegraf RPM creates a telegraf user to be run as. Therefore various plugins seem to be using a "use_sudo" parameter to work-around this limitation. My request is to add this parameter to this plugin too.

Proposal:

Adding "use_sudo" to avoid running Telegraf as root for various metrics.

Current behavior:

num_fds or I/O metrics may not return anything, while other such as memory_data or memory_swap return 0.

Desired behavior:

Getting all requested metrics with accurate values.

Use case:

Install official Telegraf RPM on Rhel 8, then use this configuration:

[[inputs.procstat]]
  name_suffix = "<MY_SUFFIX>"
  cgroup = "<MY_CGROUP>"
  pid_tag = true
  fieldpass = [ "cpu_usage", "memory_usage", "memory_data", "memory_rss", "memory_stack", "memory_swap", "memory_vms", "num_threads", "num_fds", "pid_count", "read_bytes", "read_count", "write_bytes", "write_count" ]

[powersj]

Hi,

Thanks for the PR. This sounds like a good idea for your use-case. Would you be interested in opening a PR with the config option to allow for the use of sudo? You could look at the lvm plugin as an example.

Thanks!

[Tulux]

Hi Joshua,

Thank you for your reply however I am not sure to understand your wish. I have created #11389 which contains a basic commit on branch procstat_sudo including the config option in the README for the end user. I have looked at the code but it doesn't seem to execute any commands (except for getting some information from a systemd unit) unlike the LVM plugin but it calls a library dependency instead (github.com/shirou/gopsutil/v3/process) so at this stage I don't really know how to get a privilege escalation.

Florent

[powersj]

Hi,

Thanks for taking a look and thank you very much for taking the time to start a PR!

At the bottom of the sample config, you will see that the default option when running is to use the pgrep command. That command is shelled out. Could try putting a sudo in front of that command, since you are not changing the pid_finder option and see if it works?

If this works, we would need to specify in the README that the use sudo would only apply with pgrep.

Thanks!

[Tulux]

Hi Joshua,

AFAIK pgrep only return process(es) PIDs not its related information. To be sure I have tried editing pgrep.go to force using sudo with appropriate sudoers and the result is strictly the same. Here is the result with out variable written to stdout:

2022-06-24T16:59:32Z I! Starting Telegraf 1.24.0-ffc5b667
2022-06-24T16:59:32Z I! Loaded inputs: procstat
2022-06-24T16:59:32Z I! Loaded aggregators: 
2022-06-24T16:59:32Z I! Loaded processors: 
2022-06-24T16:59:32Z W! Outputs are not used in testing mode!
2022-06-24T16:59:32Z I! Tags enabled: host=lunatik
1117
79035
79036

1117
79038
79039

> procstat,host=lunatik,pattern=rsyslogd,pid=1117,process_name=rsyslogd,user=root cpu_usage=0,memory_data=0i,memory_rss=5189632i,memory_stack=0i,memory_swap=0i,memory_usage=0.031019587069749832,memory_vms=227713024i,num_threads=4i 1656089973000000000
> procstat_lookup,host=lunatik,pattern=rsyslogd,pid_finder=pgrep,result=success pid_count=3i 1656089973000000000

As you can see, we only get PIDs there therefore we need privilege escalation somewhere else.

Florent

[reimda]

Since the associated PR is closed, should we close this issue too?

[Tulux]

I have created this issue to report a limitation with the procstat plugin not being run under root user (most distros telegraf package use a dedicated user instead). Although at this stage I don't know how to solve it, I don't think this issue should be closed.

[powersj]

@Tulux

Looping back around on this one. Your original config used the cgroup functionality of the procstat plugin. In this case, it appears we call filepath.Glob(procsPath) to gather the various files under the directory in question. Glob won't be able to read some of the files in your case, and it ignored errors when run.

Unfortunately, without permissions to read the files or folders it is not clear there is anything to do in this case.

[Tulux]

Indeed permissions are definitely required. I worked-around this limitation by creating a script that uses sudo (https://github.com/Tulux/cgroup-metrics). It is not ideal since it requires a sudo configuration, plus sudo is time-cpu intensive but it works.

[powersj]

It is not ideal since it requires a sudo configuration, plus sudo is time-cpu intensive but it works.

Agreed, however I think this is the only resolution for this issue.

Thanks for following up.