SonalJain1707
commented
2 years ago Team:SAP
Open SonalJain1707 opened 2 years ago
SonalJain1707
commented
2 years ago Team:SAP
botelastic[bot]
commented
2 years ago Thank you very much for creating this issue. However, we would kindly like to ask you to post all questions and issues on the Discuss forum first. In addition to awesome, knowledgeable community contributors, core Beats developers are on the forums every single day to help you out as well. So, your questions will reach a wider audience there, and if we confirm that there is a bug, then you can reopen this issue with the new information or open a new one.
sebglon
commented
1 year ago Hi, we have the same issue on metricbeat 8.4.2. K8S: 1.22.2 Elastic cluster: v8.4.2
Here are my clusterRoles:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
- services
verbs: ["get", "list", "watch"]
# Enable this rule only if planing to use Kubernetes keystore
#- apiGroups: [""]
# resources:
# - secrets
# verbs: ["get"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- "/metrics"
verbs:
- getIf i run this cmd under the metricbeat daemonset pod:
curl -k https://${NODE_NAME}:10250/stats/summary --header "Authorization: Bearer $TOKEN"
I have my pod memory usage:
{
"podRef": {
"name": "my-pod",
"namespace": "default",
"uid": "7be45b82-0395-4ae9-a0ab-7c275c121565"
},
"startTime": "2022-10-27T15:42:12Z",
"containers": [
{
"name": "postgres",
"startTime": "2022-10-27T15:42:15Z",
"cpu": {
"time": "2022-10-27T16:02:59Z",
"usageNanoCores": 3272512,
"usageCoreNanoSeconds": 19228071000
},
"memory": {
"time": "2022-10-27T16:02:59Z",
"workingSetBytes": 148652032
},
"rootfs": {
"time": "2022-10-27T16:02:52Z",
"availableBytes": 110666403840,
"capacityBytes": 135102586880,
"usedBytes": 196608,
"inodesFree": 8147061,
"inodes": 8380416,
"inodesUsed": 60
},
"logs": {
"time": "2022-10-27T16:02:59Z",
"availableBytes": 110666403840,
"capacityBytes": 135102586880,
"usedBytes": 45056,
"inodesFree": 8147061,
"inodes": 8380416,
"inodesUsed": 1
}
}
],
"cpu": {
"time": "2022-10-27T16:02:58Z",
"usageNanoCores": 3221979,
"usageCoreNanoSeconds": 19417139000
},
"memory": {
"time": "2022-10-27T16:02:58Z",
"availableBytes": 7367151616,
"usageBytes": 190943232,
"workingSetBytes": 149041152,
"rssBytes": 41623552,
"pageFaults": 803187,
"majorPageFaults": 0
},
After small investigation container memory and CPU usage are removed from kubelet here
But on metricbeat, we calculate the pod memory usage based on the container memory usage here. That's an issue for me to calculate the pod.memory.usageBytes with a non existing metrics. we have to reuse the provided values from kubelet
sebglon
commented
1 year ago Hi, For me the dataset available here is not good: https://github.com/elastic/beats/blob/33df99af2f7eb5757dac6e849e133c1370303f26/metricbeat/module/kubernetes/_meta/test/stats_summary_multiple_containers.json#L85
it not match the kbelet data: https://github.com/kubernetes/kubernetes/blob/release-1.22/pkg/kubelet/metrics/collectors/resource_metrics.go
sebglon
commented
1 year ago Here is an exampe Kubelet call on K8S 1.17:
{
"podRef": {
"name": "filebeat-5bd7l",
"namespace": "beats",
"uid": "2410a8e2-8732-45d0-a168-4066147bcc4c"
},
"startTime": "2022-09-28T09:45:25Z",
"containers": [
{
"name": "filebeat",
"startTime": "2022-09-28T09:45:35Z",
"cpu": {
"time": "2022-10-28T07:57:56Z",
"usageNanoCores": 11373776,
"usageCoreNanoSeconds": 24321372471944
},
"memory": {
"time": "2022-10-28T07:57:56Z",
"availableBytes": 177180672,
"usageBytes": 95526912,
"workingSetBytes": 84963328,
"rssBytes": 80998400,
"pageFaults": 19847091,
"majorPageFaults": 0
},
"rootfs": {
"time": "2022-10-28T07:57:56Z",
"availableBytes": 78362660864,
"capacityBytes": 135247413248,
"usedBytes": 53248,
"inodesFree": 7446403,
"inodes": 8388608,
"inodesUsed": 13
},
"logs": {
"time": "2022-10-28T07:57:56Z",
"availableBytes": 78362660864,
"capacityBytes": 135247413248,
"usedBytes": 151998464,
"inodesFree": 7446403,
"inodes": 8388608,
"inodesUsed": 942205
}
}
],
"cpu": {
"time": "2022-10-28T07:57:53Z",
"usageNanoCores": 13635645,
"usageCoreNanoSeconds": 24321593387534
},
"memory": {
"time": "2022-10-28T07:57:53Z",
"availableBytes": 175927296,
"usageBytes": 96772096,
"workingSetBytes": 86216704,
"rssBytes": 80998400,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2022-10-28T07:57:56Z",
"name": "eth0",
"rxBytes": 23609641833189,
"rxErrors": 0,
"txBytes": 20488187806525,
"txErrors": 0,
"interfaces": [
{
"name": "cali931f1a1ec64",
"rxBytes": 10908725315,
"rxErrors": 0,
"txBytes": 67507172815,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 23609641833189,
"rxErrors": 0,
"txBytes": 20488187806525,
"txErrors": 0
},
{
"name": "calif160940b4c5",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 1420536,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 18158882920140,
"rxErrors": 0,
"txBytes": 17254991431206,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "nodelocaldns",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2022-10-27T12:44:03Z",
"availableBytes": 78390005760,
"capacityBytes": 135247413248,
"usedBytes": 12288,
"inodesFree": 7446592,
"inodes": 8388608,
"inodesUsed": 5,
"name": "config"
},
{
"time": "2022-10-27T12:44:03Z",
"availableBytes": 32193466368,
"capacityBytes": 32193474560,
"usedBytes": 8192,
"inodesFree": 7859728,
"inodes": 7859735,
"inodesUsed": 7,
"name": "vault-tls-secrets"
},
{
"time": "2022-10-28T07:57:30Z",
"availableBytes": 32193462272,
"capacityBytes": 32193474560,
"usedBytes": 12288,
"inodesFree": 7859731,
"inodes": 7859735,
"inodesUsed": 4,
"name": "vault-secrets"
},
{
"time": "2022-10-27T12:44:03Z",
"availableBytes": 32193462272,
"capacityBytes": 32193474560,
"usedBytes": 12288,
"inodesFree": 7859726,
"inodes": 7859735,
"inodesUsed": 9,
"name": "filebeat-token-jml4h"
},
{
"time": "2022-10-28T07:57:30Z",
"availableBytes": 32193466368,
"capacityBytes": 32193474560,
"usedBytes": 8192,
"inodesFree": 7859732,
"inodes": 7859735,
"inodesUsed": 3,
"name": "home-init"
}
],
"ephemeral-storage": {
"time": "2022-10-28T07:58:04Z",
"availableBytes": 78362660864,
"capacityBytes": 135247413248,
"usedBytes": 152064000,
"inodesFree": 7446403,
"inodes": 8388608,
"inodesUsed": 18
}
},For me the issue is on kubelet 1.22.2 and seems fixed on 1.22.9. it's because we use systemd as cgroupfs driver (https://github.com/kubernetes/kubernetes/issues/103366#issuecomment-885430574)
fmiqbal
commented
1 year ago stumble upon this issue, I dunno if mine is also related,
using microk8s v1.27, ES 8.7 self-managed, kubernetes integration on fleet agent
memory bytes is present on kubelet,
But missing on document,
From dashboard persepective, only overview -> node memory, pods -> network rx/tx, nodes -> network rx/tx, that missing
botelastic[bot]
commented
5 months ago Hi! We just realized that we haven't looked into this issue in a while. We're sorry!
We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!
Hi,
We are trying to collect metrics from cloud converge but we are not able to see metrics for cpu , memory and network