Closed jseiser closed 2 years ago
So in the initial implementation we were told the output counter unit from the ECS metadata was in Jiffies. It's not documented at all in the AWS docs.
So I did some digging. It turns out the cpu_stats
returned by ECS are actually just pass-through CPU stats from docker's API.
That API is also not documented very well.
But I finally found this in the upstream moby source.
// CPUUsage stores All CPU stats aggregated since container inception.
type CPUUsage struct {
// Total CPU time consumed.
// Units: nanoseconds (Linux)
// Units: 100's of nanoseconds (Windows)
TotalUsage uint64 `json:"total_usage"`
// Total CPU time consumed per core (Linux). Not used on Windows.
// Units: nanoseconds.
PercpuUsage []uint64 `json:"percpu_usage,omitempty"`
// Time spent by tasks of the cgroup in kernel mode (Linux).
// Time spent by all container processes in kernel mode (Windows).
// Units: nanoseconds (Linux).
// Units: 100's of nanoseconds (Windows). Not populated for Hyper-V Containers.
UsageInKernelmode uint64 `json:"usage_in_kernelmode"`
// Time spent by tasks of the cgroup in user mode (Linux).
// Time spent by all container processes in user mode (Windows).
// Units: nanoseconds (Linux).
// Units: 100's of nanoseconds (Windows). Not populated for Hyper-V Containers
UsageInUsermode uint64 `json:"usage_in_usermode"`
}
So it turns out the data is in Nanoseconds, not Jiffies. This means the value is being converted incorrectly here.
I can not find a way to graph
ecs_cpu_seconds_total
like I can for any other exporter we have that is returning CPU seconds.Anything im trying, is returning very large numbers that I do not understand.
When i check the
/metrics
in my browser I getWhen i query in prom
If I query
rate(ecs_cpu_seconds_total[2m]) * 100
I get very large numbers,
304131.45438179886
and376788.3762791756
for core 0 and 1 for theheartbeat
container for instance.If it matters, im using something close to this: https://github.com/dwp/docker-ecs-service-discovery Which is allowing prom to
find
the container.The end goal is toi graph the CPU usage of each container.