Open fredwangwang opened 3 years ago
Hi @fredwangwang I was able to verify this on the current master
on Linux as well. The following slightly more minimal job will demonstrate this against a dev-mode Nomad:
job "example" {
datacenters = [ "dc1" ]
group "group" {
task "task" {
driver = "raw_exec"
config {
command = "ping"
args = ["8.8.8.8"]
}
resources {
cpu = 200
memory = 200
}
}
}
}
I've also verified that this appears to be working as expected with the exec
driver, and that the particular workload doesn't seem to matter (note we do see metrics like rss
). One interesting thing here is that we have a nightly end-to-end test covering exactly this metric on both Linux and Windows that passes. (ref https://github.com/hashicorp/nomad/tree/master/e2e/metrics)
One interesting thing here is that we have a nightly end-to-end test covering exactly this metric on both Linux and Windows that passes.
I took a second look at that and I'm realizing we don't actually test that the results are non-zero though. So that might be where it's slipping through. I'll dig into this further.
Hey. I am also seeing discrepancies between values for the actual CPU and memory utilization of the exe on the node vs the values shown on the Nomad UI.
@dbkukku can you open a new issue for that explaining what you're seeing in more detail? That seems like it's a different problem.
I dug a little bit deeper and I believe I found the issue.
The flow approximately is:
Since the resource usage is reporting correctly for docker and exec for example, it has to be an issue in raw_exec driver. Which leads me into this line here and here:
ExecutorBasicMeasuredMemStats = []string{"RSS", "Swap"}
ms.RSS = memInfo.RSS
ms.Swap = memInfo.Swap
ms.Measured = ExecutorBasicMeasuredMemStats
The raw_exec only expose those two metrics. It makes sense since raw_exec depends greatly on the underlying OS to provide the metrics, which can be hugely different.
But since whats been measured is propagated back to the task runner by setting Measured []string
here, I think what probably makes more sense to me is to only show the metrics that are actually collected, instead of showing 0
.
@tgross could you provide some thoughts on this? Thanks!
Hi @fredwangwang looks like that's exactly it!
I compared the results and code path of the raw_exec
driver to the exec
driver running the exact same workload. When we collect the stats from exec
, we're hitting the libcontainer path in executor_linux.go#L353-L363
, whereas in raw_exec
we only ever hit the pid collector in the "universal executor" in executor.go#L595-L599
.
The pid_collector
calls into the gopsutil library 's MemoryInfo
. I tried to see if we could get that additional data out of that pid collector or derive it somehow. But it looks like that gopsutil is throwing away the platform-specific "extended" memory stats available from fillFromStatmWithContext
, which is probably just as well because there's a bug there. š (https://github.com/shirou/gopsutil/issues/277)
I think you're right that the best approach would be to not report the metrics that we're not collecting. It looks like the gauges are being written in the client, in the task runner: task_runner.go#L1296-L1323
. Next step is to figure out why that Measured
field isn't being referenced before writing those.
Thanks for following up @tgross!
I think we should be able to work with the metrics (RSS specifically) that we are getting, but still needs to understand that a bit more.
The confusion was mainly around that certain metrics reporting 0 where it should really just not showing up IMO. Looking forward to seeing it getting fixed or hearing more about why it is kept to report 0
! š
I had a conversation with folks here internally and that section of task_runner
should be checking and it just isn't. I'm going to double-check that there's no unexpected behavior in fixing it with the other task drivers. But otherwise should be a small fix.
Thanks @tgross!
A small note, the docs https://www.nomadproject.io/docs/telemetry/metrics#allocation-metrics probably needs to be updated as well to call out that memory metrics emitted depends on the driver type a task uses
Just for reference:
memory.usage
is the reporting by cgroups (resource container), thats why it not available for raw_exec
driver type.
memroy.usage == RSS (actual mem usage) + CACHE (page cache) + SWAP.
Using memory.rss
instead to get the actual mem usage, which is also the number used by nomad in its UI.
any update on this? Would love to have that available.
Hi @it-ben there are no updates currently. When an engineer is assigned to this and working on it, updates will be provided in the issue.
Nomad version
Nomad v0.12.3 (2db8abd9620dd41cb7bfe399551ba0f7824b3f61)
Operating system and Environment details
This happens on both windows nodes and linux nodes
Issue
Nomad reports wrong (
0
) nomad.client.allocs.memory.usage metric for jobs with raw_exec driverReproduction steps
deploy:
and see
http://host-with-above-alloc/v1/metrics
:Job file (if appropriate)
see above