Closed carlosedp closed 2 years ago
@carlosedp I've run into an issue where the ingresses often change location, and I can no longer load the Grafana page when that happens:
[2020-10-31 09:17:31+2 ✘][~] [james@tpin1][$ sudo kubectl get ingress -o wide -n monitoring NAME CLASS HOSTS ADDRESS PORTS AGE grafana <none> grafana.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h alertmanager-main <none> alertmanager.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h prometheus-k8s <none> prometheus.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h
I tried to run the make target to update the ingress suffix, but wasn't quite sure if that was the right command to fix the problem?
You can edit the manifests themselves or change the vars.jsonnet and regenerate the manifests, than apply again to the cluster.
I have this up an running on my pi4 cluster and it works very well. (Thank you very much! Its awesome!) I have a microcontroller running in some custom hardware and it has a webserver that posts metrics on `192.168.xx.xx/metrics and am trying to add it as a target for Prometheus to get the data from.
Normally I think it would be set up in the prometheus.yml
file something like this:
scrape_configs:
- job_name: "somemetrics"
scrape_interval: 5s
scrape_timeout: 5s
static_configs:
- targets: ['192.168.xx.xx']
I tried to add the same configuration to the vars.jsonnet
file like this:
prometheus: {
retention: '15d',
scrapeInterval: '30s',
scrapeTimeout: '30s',
scrape_configs: [
{
job_name: 'somemetrics',
scrape_interval: '5s',
scrape_timeout: '5s',
static_configs: [
{
targets: [
'192.168.xx.xx',
],
},
],
},
],
},
However, this does not seem to work and it does not appear as a target under the /targets
page in the prometheus web interface. I might be missing something obvious here so please point me in the right direction if I am. My question is essentially ho do I set up my targets like above mentioned? Is it supposed to be done via the vars.jsonnet
file or am I going about this wrong?
Looking at this, your targets also need to be one the same container networks that your prometheus stack is on. OR You need to accept incoming container port forwarding. If you followed the steps in the Make Deploy, it created a monitoring network, you need to enable ingress into the prometheus controller.
On Sun, Nov 29, 2020 at 1:58 PM MrCravon notifications@github.com wrote:
I have this up an running on my pi4 cluster and it works very well. (Thank you very much! Its awesome!) I have a microcontroller running in some custom hardware and it has a webserver that posts metrics on `192.168.xx.xx/metrics and am trying to add it as a target for Prometheus to get the data from.
Normally I think it would be set up in the prometheus.yml file something like this:
scrape_configs:
- job_name: "somemetrics" scrape_interval: 5s scrape_timeout: 5s static_configs:
- targets: ['192.168.xx.xx']
I tried to add the same configuration to the vars.jsonnet file like this:
prometheus: { retention: '15d', scrapeInterval: '30s', scrapeTimeout: '30s', scrape_configs: [ { job_name: 'somemetrics', scrape_interval: '5s', scrape_timeout: '5s', static_configs: [ { targets: [ '192.168.xx.xx', ], }, ], }, ], },
However, this does not seem to work and it does not appear as a target under the /targets page in the prometheus web interface. I might be missing something obvious here so please point me in the right direction if I am. My question is essentially ho do I set up my targets like above mentioned? Is it supposed to be done via the vars.jsonnet file or am I going about this wrong?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/carlosedp/cluster-monitoring/issues/91#issuecomment-735439026, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC26VXLNYVKVURFE3ZA7BTSSKKXBANCNFSM4QGQ2QHA .
Thanks, I'll look more in to the ingress part of this tomorrow. However, I was under the impression that ingress was used to manage external access IN to the services in the containers. Like a client outside the cluster getting access to the web interfaces for prometheus and grafana. However in my case the 192.168.xx.xx
is a webserver outside the cluster on the local network. The page on the webserver called /metrics
would then be accessed by the prometheus scraper as a client. So would I need still need a ingress for the prometheus container scraper to access a webserver outside the cluster?
Carlos and team, i have added a Prometheus snmp-exporter into the same monitoring cluster to this awesome deployment in order to scrape snmp managed Cisco devices. I have deployed snmp-exporter as you suggested via helm charts on the same monitoring namespace, what i need assistance with mine particular situation is how can i update the prometheus.yml file to start scraping this SNMP managed devices? and the ability to add the snmp.yml into the prometheus-k8s-0 container?. i appreciated the support.
Hello,
How we can add a new dashboard with some panels at grafana-dashboardDefinitions? Could this be done by adding an extra JSON in the grafana-DashboardDefinitions.yaml or I have to do something more? What I was thinking is to deploy the stack adding new dashboard with the Web GUI and then copy the JSON that it produces to add it in grafana-dashboardDefinitions, is any other approach available ? When I follow this approach kubectl says that the appropriate config map is created but in the dashboards there is none of my custom dashboards
EDIT The solution was to add some extra line to the deployment file of grafana in the part of volume mounts and to declare the the config map below the volumes part
Update on the stuff below
I've switched from wifi to ethernet on the raspberry pi cluster and everything just works. So for anyone out there experiencing and issue like I mentioned below, you might try ethernet instead.
@rur0 Did you ever resolve the issue? I am getting the same thing. On a fresh Raspberry Pi cluster, running k3s on Ubuntu Server 20.10, I see the following:
root@main:~/cluster-monitoring# kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
node-exporter-p8qk8 2/2 Running 0 24s
prometheus-adapter-9c79c98f7-wfh2n 1/1 Running 0 22s
arm-exporter-h22xf 2/2 Running 0 35s
arm-exporter-rpbqb 0/2 CreateContainerError 0 35s
kube-state-metrics-857f95d994-crhd4 0/3 CrashLoopBackOff 3 25s
prometheus-operator-67586fc88-dlfbt 0/2 RunContainerError 6 53s
grafana-655d666bbb-lt7xv 0/1 RunContainerError 2 26s
node-exporter-7fg5z 0/2 CrashLoopBackOff 4 24s
root@main:~/cluster-monitoring# kubectl describe pod -n monitoring arm-exporter-rpbqb
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned monitoring/arm-exporter-rpbqb to worker-01
Warning Failed 2m22s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 939b8c9fb00b22dcfd7f10ef4815f330f3b56204b40a640d12127f0c7733ce2a not found: not found
Warning Failed 2m22s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 939b8c9fb00b22dcfd7f10ef4815f330f3b56204b40a640d12127f0c7733ce2a not found: not found
Warning Failed 2m18s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 197d303f31f3e16d5ca686ad3d783292d6d8e9607bc452bec3010d09319d8c7c not found: not found
Warning Failed 2m18s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 197d303f31f3e16d5ca686ad3d783292d6d8e9607bc452bec3010d09319d8c7c not found: not found
Warning Failed 2m15s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task ddfbdfe9a1372779e41ab400bcd0cf47722f722fb48060dc4d0e62609d87669e not found: not found
Warning Failed 2m15s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task ddfbdfe9a1372779e41ab400bcd0cf47722f722fb48060dc4d0e62609d87669e not found: not found
Normal Pulled 2m11s (x4 over 2m22s) kubelet, worker-01 Successfully pulled image "carlosedp/arm_exporter:latest"
Warning Failed 2m11s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task b87a5e0a9eae8daf52cabe9cf7c838f57cd2f738e47bfc81e88b99d789d0b698 not found: not found
Normal Pulled 2m11s (x4 over 2m22s) kubelet, worker-01 Container image "carlosedp/kube-rbac-proxy:v0.5.0" already present on machine
Warning Failed 2m11s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task b87a5e0a9eae8daf52cabe9cf7c838f57cd2f738e47bfc81e88b99d789d0b698 not found: not found
Normal SandboxChanged 2m11s (x4 over 2m21s) kubelet, worker-01 Pod sandbox changed, it will be killed and re-created.
Normal Pulling 2m6s (x5 over 2m23s) kubelet, worker-01 Pulling image "carlosedp/arm_exporter:latest"
root@main:~/cluster-monitoring# kubectl describe pod -n monitoring kube-state-metrics-857f95d994-crhd4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned monitoring/kube-state-metrics-857f95d994-crhd4 to worker-01
Warning FailedMount 4m4s (x2 over 4m6s) kubelet, worker-01 MountVolume.SetUp failed for volume "kube-state-metrics-token-t5fx4" : failed to sync secret cache: timed out waiting for the condition
Warning Failed 4m kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task a164f027602367fe02063e752a2f57874cf803c21033ff40b0163b00d326e41f not found: not found
Warning Failed 4m kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task a164f027602367fe02063e752a2f57874cf803c21033ff40b0163b00d326e41f not found: not found
Warning Failed 4m kubelet, worker-01 Error: failed to create containerd task: OCI runtime create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1923: running lstat on namespace path \"/proc/1072790/ns/ipc\" caused \"lstat /proc/1072790/ns/ipc: no such file or directory\"": unknown
Warning Failed 3m58s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 4c9449c09877664bcebb21b0be7f7d1074b2069071c88296ad7dd193cf986f18 not found: not found
Normal Pulled 3m58s (x2 over 4m1s) kubelet, worker-01 Container image "carlosedp/kube-state-metrics:v1.9.6" already present on machine
Normal Created 3m58s (x2 over 4m1s) kubelet, worker-01 Created container kube-state-metrics
Warning Failed 3m58s kubelet, worker-01 Error: sandbox container "4c9449c09877664bcebb21b0be7f7d1074b2069071c88296ad7dd193cf986f18" is not running
Warning Failed 3m58s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 4c9449c09877664bcebb21b0be7f7d1074b2069071c88296ad7dd193cf986f18 not found: not found
Warning BackOff 3m56s kubelet, worker-01 Back-off restarting failed container
Normal Pulled 3m56s (x3 over 4m) kubelet, worker-01 Container image "carlosedp/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 3m56s kubelet, worker-01 Created container kube-rbac-proxy-main
Warning Failed 3m56s kubelet, worker-01 Error: sandbox container "73acb31399df06c63f892a58a23ccc68ff9392f55184c28bdf6b03925161adbd" is not running
Normal Pulled 3m56s (x3 over 4m) kubelet, worker-01 Container image "carlosedp/kube-rbac-proxy:v0.5.0" already present on machine
Warning Failed 3m56s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 73acb31399df06c63f892a58a23ccc68ff9392f55184c28bdf6b03925161adbd not found: not found
Normal SandboxChanged 3m55s (x3 over 4m) kubelet, worker-01 Pod sandbox changed, it will be killed and re-created.
root@main:~/cluster-monitoring# kubectl describe pod -n monitoring prometheus-operator-67586fc88-dlfbt
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned monitoring/prometheus-operator-67586fc88-dlfbt to worker-01
Warning Failed 5m25s kubelet, worker-01 Error: sandbox container "9e6b794cfb528efdd28ca2c43de3db000ca3666891358bed726d4ddab3551662" is not running
Warning Failed 5m25s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 9e6b794cfb528efdd28ca2c43de3db000ca3666891358bed726d4ddab3551662 not found: not found
Normal Pulled 5m24s (x2 over 5m25s) kubelet, worker-01 Container image "carlosedp/prometheus-operator:v0.40.0" already present on machine
Normal Created 5m24s (x2 over 5m25s) kubelet, worker-01 Created container prometheus-operator
Warning Failed 5m24s kubelet, worker-01 Error: sandbox container "74fda8c4bf09286f857d695335e6818ef8694453ccec55743f3d124071ef74d0" is not running
Warning Failed 5m24s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 74fda8c4bf09286f857d695335e6818ef8694453ccec55743f3d124071ef74d0 not found: not found
Warning Failed 5m20s kubelet, worker-01 Error: failed to create containerd task: OCI runtime create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1923: running lstat on namespace path \"/proc/1062821/ns/ipc\" caused \"lstat /proc/1062821/ns/ipc: no such file or directory\"": unknown
Normal Pulled 5m18s (x4 over 5m25s) kubelet, worker-01 Container image "carlosedp/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 5m18s (x2 over 5m20s) kubelet, worker-01 Created container kube-rbac-proxy
Warning Failed 5m18s kubelet, worker-01 Error: sandbox container "55e4c32d4561226673a78af5ea4023544bada8e491924545ffa87ff5d9ef0799" is not running
Warning BackOff 5m16s kubelet, worker-01 Back-off restarting failed container
Warning BackOff 5m16s (x3 over 5m20s) kubelet, worker-01 Back-off restarting failed container
Normal SandboxChanged 22s (x61 over 5m25s) kubelet, worker-01 Pod sandbox changed, it will be killed and re-created.
root@main:~/cluster-monitoring# kubectl describe pod -n monitoring grafana-655d666bbb-lt7xv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned monitoring/grafana-655d666bbb-lt7xv to worker-01
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-controller-manager" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-traefik-dashboard" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-prometheus-dashboard" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-statefulset" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-persistentvolumesusage" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-prometheus" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-cluster" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m45s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-workload" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m44s (x2 over 5m45s) kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-apiserver" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m44s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-nodes" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m44s kubelet, worker-01 MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-workloads-namespace" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 5m42s (x12 over 5m44s) kubelet, worker-01 (combined from similar events): MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-pod" : failed to sync configmap cache: timed out waiting for the condition
Normal Pulled 5m41s kubelet, worker-01 Container image "grafana/grafana:7.0.3" already present on machine
Warning BackOff 44s (x61 over 5m35s) kubelet, worker-01 Back-off restarting failed container
root@main:~/cluster-monitoring# kubectl describe pod -n monitoring node-exporter-7fg5z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned monitoring/node-exporter-7fg5z to worker-01
Warning FailedMount 6m15s (x2 over 6m16s) kubelet, worker-01 MountVolume.SetUp failed for volume "node-exporter-token-ntcrf" : failed to sync secret cache: timed out waiting for the condition
Warning Failed 6m12s kubelet, worker-01 Error: sandbox container "5a48a1bacbd4974d3470302bce815a842e08dc9906589a8fcc2b2e025766891c" is not running
Warning Failed 6m12s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 5a48a1bacbd4974d3470302bce815a842e08dc9906589a8fcc2b2e025766891c not found: not found
Normal Pulled 6m10s (x2 over 6m13s) kubelet, worker-01 Container image "prom/node-exporter:v0.18.1" already present on machine
Normal Created 6m10s (x2 over 6m12s) kubelet, worker-01 Created container node-exporter
Warning Failed 6m10s kubelet, worker-01 Error: sandbox container "6bc941208b1c99a891c941ec7bb21bb5b3c3419758eba92b0d7e3d33e2d6536a" is not running
Warning Failed 6m10s kubelet, worker-01 Error: failed to get sandbox container task: no running task found: task 6bc941208b1c99a891c941ec7bb21bb5b3c3419758eba92b0d7e3d33e2d6536a not found: not found
Warning Failed 6m8s kubelet, worker-01 Error: failed to create containerd task: OCI runtime create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1923: running lstat on namespace path \"/proc/1074283/ns/ipc\" caused \"lstat /proc/1074283/ns/ipc: no such file or directory\"": unknown
Normal Pulled 6m7s (x4 over 6m12s) kubelet, worker-01 Container image "carlosedp/kube-rbac-proxy:v0.5.0" already present on machine
Normal Created 6m7s (x2 over 6m9s) kubelet, worker-01 Created container kube-rbac-proxy
Warning Failed 6m6s kubelet, worker-01 Error: failed to create containerd task: OCI runtime create failed: container_linux.go:341: creating new parent process caused "container_linux.go:1923: running lstat on namespace path \"/proc/1075142/ns/ipc\" caused \"lstat /proc/1075142/ns/ipc: no such file or directory\"": unknown
Normal SandboxChanged 6m6s (x4 over 6m12s) kubelet, worker-01 Pod sandbox changed, it will be killed and re-created.
Warning BackOff 6m5s (x3 over 6m9s) kubelet, worker-01 Back-off restarting failed container
Warning BackOff 74s (x211 over 6m5s) kubelet, worker-01 Back-off restarting failed container
I just went through a power outage and would like to enable the APCUPSd exporter. However, I don't see a deployment. Is this designed to work with a SmartUPS and you are querying the UPS directly? If you are using apcupsd exporter, where does it run?
Hi i have some problems with a cluster of raspberry pi with k3s, i m not be able to reach the nodes from outside. I used k3s built in with trafik and coredns, why im not be able to reach for example a simple hello world pod from outside? Any suggests?
I tried even to install this monitoring but the address shown in the ingress didn't work.
This is my k3s cluster configuration, something is wrong?
I have recently deployed this to a k3s 1.20.0+k3s2 cluster with 3 Raspberry Pis. The only two enabled exporters are the armExporter and the traefikExporter.
The most important metrics all show up(Cluster Memory Usage, Cluster CPU Usage, Memory Usage, CPU Usage, CPU Temperature and a few more), however some other ones do not get any values. The not working ones are Pod CPU Usage, Pod Memory Usage, Sent Network Traffic per Container, Received Network Traffic per Container, and Pod Network i/o. They all show "No data".
I have looked at all the logs, and the only one that seemed like it might want to say anything but "everythings ok", was the one of the node-exporter containers, however even that one seemed pretty ok. Here it is:
kubectl logs -f -n monitoring node-exporter-qkxzr -c node-exporter
time="2020-12-27T11:46:51Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156"
time="2020-12-27T11:46:51Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:43:22)" source="node_exporter.go:157"
time="2020-12-27T11:46:51Z" level=info msg="Enabled collectors:" source="node_exporter.go:97"
time="2020-12-27T11:46:51Z" level=info msg=" - arp" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - bcache" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - bonding" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - conntrack" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - cpu" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - cpufreq" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - diskstats" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - edac" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - entropy" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - filefd" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - filesystem" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - infiniband" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - ipvs" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - loadavg" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - mdadm" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - meminfo" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - netclass" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - netdev" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - netstat" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - nfs" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - nfsd" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - pressure" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - sockstat" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - stat" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - textfile" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - time" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - timex" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - uname" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - vmstat" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - xfs" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg=" - zfs" source="node_exporter.go:104"
time="2020-12-27T11:46:51Z" level=info msg="Listening on 127.0.0.1:9100" source="node_exporter.go:170"
I have tried running make deploy
again, but that didn't change anything.
All the pods are shown as RUNNING.
These two targets are shown as DOWN, however k3s doesn't show any pod with names like that.
I also noticed some time later that the title is monitoring/kube-*, while the labels say these pods should be in the kube-system namespace.
Edit: I just tried running make vendor
, make
, and make deploy
again, but this also didn't change anything.
This and the ingresses seem to be the only thing that showed configured, rather then unchanged:
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured
Also the monitoring namespace has a pod named prometheus-k8s-0
, even tho its k3s, but i am not sure whether thats an issue.
I solve this issue. Thank for advice, at the end I just installed nginx, configured it and after that I was able to access to prometheus and grafana. Thanks a lot!
Hi, can u please provide the detail step to resolve this same issue ? thanks
Is my issue maybe related to this?
Kube-apiserver: the componentstatus API is deprecated. This API provided status of etcd, kube-scheduler, and kube-controller-manager components, but only worked when those components were local to the API server, and when kube-scheduler and kube-controller-manager exposed unsecured health endpoints. Instead of this API, etcd health is included in the kube-apiserver health check and kube-scheduler/kube-controller-manager health checks can be made directly against those components' health endpoints. (#93570, @liggitt) [SIG API Machinery, Apps and Cluster Lifecycle] - https://kubernetes.io/docs/setup/release/notes/#deprecation-5
The node exporters are the only pods that run on the local Ip of the device they run on (192.168.X.X), instead of a cluster internal Ip (10.42.X.X).
Edit: my ingress problem solved itself after another restart.
Also here is the result of kubectl get pods --all-namespaces -o wide
:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system helm-install-traefik-ld2nr 0/1 Completed 0 14h 10.42.0.2 raspberrypi-master <none> <none>
kube-system svclb-traefik-8pr5w 2/2 Running 4 14h 10.42.2.6 raspberrypi-worker1 <none> <none>
monitoring node-exporter-cnzrd 2/2 Running 4 14h 192.168.2.104 raspberrypi-worker1 <none> <none>
monitoring arm-exporter-hlgpg 2/2 Running 4 14h 10.42.2.7 raspberrypi-worker1 <none> <none>
kube-system metrics-server-86cbb8457f-hvckl 1/1 Running 2 14h 10.42.0.22 raspberrypi-master <none> <none>
kube-system local-path-provisioner-7c458769fb-f7fn6 1/1 Running 2 14h 10.42.0.21 raspberrypi-master <none> <none>
monitoring node-exporter-b24st 2/2 Running 4 14h 192.168.2.106 raspberrypi-master <none> <none>
kube-system svclb-traefik-7754c 2/2 Running 4 14h 10.42.0.23 raspberrypi-master <none> <none>
kube-system coredns-854c77959c-v67jt 1/1 Running 2 14h 10.42.0.19 raspberrypi-master <none> <none>
monitoring arm-exporter-c28hq 2/2 Running 4 14h 10.42.0.20 raspberrypi-master <none> <none>
kube-system svclb-traefik-lf4ck 2/2 Running 4 14h 10.42.1.21 raspberrypi-worker2 <none> <none>
monitoring node-exporter-w56w2 2/2 Running 4 14h 192.168.2.121 raspberrypi-worker2 <none> <none>
monitoring arm-exporter-q7wx2 2/2 Running 4 14h 10.42.1.23 raspberrypi-worker2 <none> <none>
monitoring prometheus-operator-67755f959-zghjt 2/2 Running 4 14h 10.42.1.22 raspberrypi-worker2 <none> <none>
monitoring kube-state-metrics-6cb6df5d4-dvw4k 3/3 Running 6 14h 10.42.1.26 raspberrypi-worker2 <none> <none>
kube-system traefik-6f9cbd9bd4-ldv68 1/1 Running 2 14h 10.42.1.20 raspberrypi-worker2 <none> <none>
monitoring alertmanager-main-0 2/2 Running 4 14h 10.42.1.24 raspberrypi-worker2 <none> <none>
monitoring prometheus-adapter-585b57857b-lp6vl 1/1 Running 2 14h 10.42.1.27 raspberrypi-worker2 <none> <none>
monitoring grafana-7cccfc9b5f-bd4sh 1/1 Running 2 14h 10.42.1.25 raspberrypi-worker2 <none> <none>
monitoring prometheus-k8s-0 3/3 Running 0 2m40s 10.42.1.28 raspberrypi-worker2 <none> <none>
@carlosedp - Thanks. So people run their K8s clusters without any firewalls on the servers? That’s an interesting paradigm shift, indeed! 60 % of the time, it works every time … On 28 Oct 2020, at 19:27, Carlos Eduardo @.***> wrote: @jjo93sa usually on Kubernetes clusters, we don't set IPTables rules so they don't mess with Kubernetes rules and block required ports. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Between the nodes, yes. Protect the cluster itself not between cluster nodes.
@carlosedp I've run into an issue where the ingresses often change location, and I can no longer load the Grafana page when that happens:
[2020-10-31 09:17:31+2 ✘][~] [james@tpin1][$ sudo kubectl get ingress -o wide -n monitoring NAME CLASS HOSTS ADDRESS PORTS AGE grafana <none> grafana.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h alertmanager-main <none> alertmanager.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h prometheus-k8s <none> prometheus.10.10.50.24.nip.io 10.10.50.23 80, 443 3d14h
I tried to run the make target to update the ingress suffix, but wasn't quite sure if that was the right command to fix the problem?
@jjo93sa you need to update the config and re-apply the ingress manifests.
Carlos and team, i have added a Prometheus snmp-exporter into the same monitoring cluster to this awesome deployment in order to scrape snmp managed Cisco devices. I have deployed snmp-exporter as you suggested via helm charts on the same monitoring namespace, what i need assistance with mine particular situation is how can i update the prometheus.yml file to start scraping this SNMP managed devices? and the ability to add the snmp.yml into the prometheus-k8s-0 container?. i appreciated the support.
@robmit68 you need to create a custom ServiceMonitor with the prometheus config pointing to the SNMP exporter and targeting your device. It's something like https://github.com/carlosedp/ddwrt-monitoring/blob/02f3013c3acb80ec048c77469f64f76c5e2406e3/prometheus/prometheus.yml#L37
I just went through a power outage and would like to enable the APCUPSd exporter. However, I don't see a deployment. Is this designed to work with a SmartUPS and you are querying the UPS directly? If you are using apcupsd exporter, where does it run?
@simonspg In this deployment, apcupsd and the exporter are running on an external host and Prometheus just queries it.
You could also run the ups-exporter in the cluster querying the apcupsd host (that also has the UPS running) but it would require adding the deployment in modules/ups_exporter.jsonnet.
Hi i have some problems with a cluster of raspberry pi with k3s, i m not be able to reach the nodes from outside. I used k3s built in with trafik and coredns, why im not be able to reach for example a simple hello world pod from outside? Any suggests? I tried even to install this monitoring but the address shown in the ingress didn't work.
![]()
This is my k3s cluster configuration, something is wrong?
@MovieMaker93 this is a question for K3s and not the monitoring stack, sorry, check the installation steps, your hosts and start small, then deploy the monitoring.
To all, in a couple weeks I'll check this stack in a fresh Kubernetes and K3s clusters to see if there are any required updates. Also will update libraries, dependencies and dashboards.
Outstanding! Thank you for clarifying where ACPUPSd is running. It all makes sense now. Another question: I would like to add to and modify the supplied Grafana dashboards, but it seems that is not possible due to the way they are stored. How hard would it be to move the dashboard storage to a PVC? And one more question: I see timeouts on database access. I have MariaDB in a container for another project. How hard would it be to move to a MariaDB backend? THANK YOU for this fabulous monitoring solution!
Hi Carlos, thanks for a great piece of software! I found it via Jeff Geerling's blog and installed in on my TuringPi (K3s) with 7 CM3 installed. That said, with only your package installed I have rather high memory utilization (cluster memory usage 41%) on two nodes (700-800MB) with prometheus-k8s-0 eating close to 500 MB. Is this normal or did I miss a setting somewhere? Also I constantly get CPUThrottlingHigh alerts.
Hi Carlos, thanks for a great piece of software! I found it via Jeff Geerling's blog and installed in on my TuringPi (K3s) with 7 CM3 installed. That said, with only your package installed I have rather high memory utilization (cluster memory usage 41%) on two nodes (700-800MB) with prometheus-k8s-0 eating close to 500 MB. Is this normal or did I miss a setting somewhere? Also I constantly get CPUThrottlingHigh alerts.
I've exactly the same configuration with the same issues
@Gooseman42 and @thorsten-l, monitoring infrastructure is not a "cheap" job and requires processing and memory resources proportional to the amount of nodes, events and a multitude of parameters collected from the cluster.
Although the stack works fine on Raspberry Pi nodes, it's meant to monitor from single node to multi-node production clusters. It's not optimized (and never meant to be) to run on small board computers.
All good, thanks for the clarification. I just wanted to make sure it wasn't due to some configuration issue.
I have recently deployed this to a k3s 1.20.0+k3s2 cluster with 3 Raspberry Pis. The only two enabled exporters are the armExporter and the traefikExporter.
The most important metrics all show up(Cluster Memory Usage, Cluster CPU Usage, Memory Usage, CPU Usage, CPU Temperature and a few more), however some other ones do not get any values. The not working ones are Pod CPU Usage, Pod Memory Usage, Sent Network Traffic per Container, Received Network Traffic per Container, and Pod Network i/o. They all show "No data".
I have looked at all the logs, and the only one that seemed like it might want to say anything but "everythings ok", was the one of the node-exporter containers, however even that one seemed pretty ok. Here it is:
kubectl logs -f -n monitoring node-exporter-qkxzr -c node-exporter time="2020-12-27T11:46:51Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" time="2020-12-27T11:46:51Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:43:22)" source="node_exporter.go:157" time="2020-12-27T11:46:51Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" time="2020-12-27T11:46:51Z" level=info msg=" - arp" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - bcache" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - bonding" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - conntrack" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - cpu" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - cpufreq" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - diskstats" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - edac" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - entropy" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - filefd" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - filesystem" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - infiniband" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - ipvs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - loadavg" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - mdadm" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - meminfo" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netclass" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netdev" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - nfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - nfsd" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - pressure" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - sockstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - stat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - textfile" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - time" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - timex" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - uname" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - vmstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - xfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - zfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg="Listening on 127.0.0.1:9100" source="node_exporter.go:170"
I have tried running
make deploy
again, but that didn't change anything. All the pods are shown as RUNNING.These two targets are shown as DOWN, however k3s doesn't show any pod with names like that.
I also noticed some time later that the title is monitoring/kube-*, while the labels say these pods should be in the kube-system namespace.
Edit: I just tried running
make vendor
,make
, andmake deploy
again, but this also didn't change anything. This and the ingresses seem to be the only thing that showed configured, rather then unchanged:customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured
Also the monitoring namespace has a pod named
prometheus-k8s-0
, even tho its k3s, but i am not sure whether thats an issue.
I have the same problem. How did you manage to solve?
I did not manage to solve it yet. I hope that the dependency update will fix this, whenever that happens.
Hello,
Is there a way to declare in the deployment file of Grafana the admin password and some group of users with their roles/permissions ?
I have recently deployed this to a k3s 1.20.0+k3s2 cluster with 3 Raspberry Pis. The only two enabled exporters are the armExporter and the traefikExporter. The most important metrics all show up(Cluster Memory Usage, Cluster CPU Usage, Memory Usage, CPU Usage, CPU Temperature and a few more), however some other ones do not get any values. The not working ones are Pod CPU Usage, Pod Memory Usage, Sent Network Traffic per Container, Received Network Traffic per Container, and Pod Network i/o. They all show "No data". I have looked at all the logs, and the only one that seemed like it might want to say anything but "everythings ok", was the one of the node-exporter containers, however even that one seemed pretty ok. Here it is:
kubectl logs -f -n monitoring node-exporter-qkxzr -c node-exporter time="2020-12-27T11:46:51Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" time="2020-12-27T11:46:51Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:43:22)" source="node_exporter.go:157" time="2020-12-27T11:46:51Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" time="2020-12-27T11:46:51Z" level=info msg=" - arp" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - bcache" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - bonding" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - conntrack" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - cpu" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - cpufreq" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - diskstats" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - edac" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - entropy" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - filefd" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - filesystem" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - infiniband" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - ipvs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - loadavg" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - mdadm" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - meminfo" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netclass" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netdev" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - netstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - nfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - nfsd" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - pressure" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - sockstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - stat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - textfile" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - time" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - timex" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - uname" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - vmstat" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - xfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg=" - zfs" source="node_exporter.go:104" time="2020-12-27T11:46:51Z" level=info msg="Listening on 127.0.0.1:9100" source="node_exporter.go:170"
I have tried running
make deploy
again, but that didn't change anything. All the pods are shown as RUNNING. These two targets are shown as DOWN, however k3s doesn't show any pod with names like that.I also noticed some time later that the title is monitoring/kube-*, while the labels say these pods should be in the kube-system namespace. Edit: I just tried running
make vendor
,make
, andmake deploy
again, but this also didn't change anything. This and the ingresses seem to be the only thing that showed configured, rather then unchanged:customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured
Also the monitoring namespace has a pod named
prometheus-k8s-0
, even tho its k3s, but i am not sure whether thats an issue.I have the same problem. How did you manage to solve?
+1
When I perform kubectl apply manifests and manifests/setup I get this error for kube-state-metrics and prometheus operator
kubectl logs -n monitoring prometheus-operator-67755f959-rctgl kube-rbac-proxy I0318 14:07:13.720068 1 main.go:186] Valid token audiences: I0318 14:07:13.720148 1 main.go:232] Generating self signed cert as no cert is provided runtime: mlock of signal stack failed: 12 runtime: increase the mlock limit (ulimit -l) or runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ fatal error: mlock failed
To all, in a couple weeks I'll check this stack in a fresh Kubernetes and K3s clusters to see if there are any required updates. Also will update libraries, dependencies and dashboards.
This was now 2⅔ months ago, is there a rough estimate as to when this update will take place?
Updating k3s from 1.20.0+k3s2 to 1.20.4+k3s1 solved the graphs showing "No Data" for me. The two targets i mentioned are still shown as down, however this doesn't seem to affect much? Also for some prometheus is now using 2GB of ram, which is an issue on a Raspberry Pi, but i hope it only does so while cleaning up and stops that later. @Gory19 @Northwood128 @alexwilliams0712
Firstly excellent project. Very well done. I do not have a real issue but I am seeking some guidance. I am pretty new to all this. I know Docker well, but this is my first stab at kubernetes and I only put my pi cluster together a couple of weeks ago. I have everything up and running and have added additional exporters on my router for UPS, network and NAS metrics, and I have created grafana dashboards to view them. I also have additional data sources to monitor my websites. Finally I have also added in the ELK stack with filebeat and metricbeat in the monitoring namespace and pointed a few things at each other.
However, I did not set up grafana or prometheus with persistent storage as I did not have anything set up. I have now purchased an additional Pi, SSDs and networking equipment to set up a SAN as an iSCSI target. I would now like to backup my grafana datasources and dashboards and redeploy the stack with persistent storage.
Backing up the json for the dashboards is easy. Data sources less so. What I would like to do of course is add the dashboards and datasources to the deployment. It looks like I just need to drop the dashboard json I export into grafana-dashboardDefinitions.yaml? Is there a way datasourses are injected too? according to https://grafana.com/docs/grafana/latest/administration/provisioning/#datasources I should be able to define them somewhere but grafana-dashboardDatasources.yaml is a secret, which doesn't seem to be deployed to my cluster (because I don't have persistent storage?) How can I go about capturing my data sources so I do not need to re-create them?
Just seeking a little guidance. And thanks again for the awesome work.
Hey @radicalgeek awesome progress on your learning experience.
First thing as you mentioned is exporting all things you created (dashboards, datasources and etc) to avoid losing them.
To have the dashboards deployed by the stack, I'd recommend creating something like the jsonnet modules I defined in modules
dir and add it to the vars.jsonnet. This way, the deployment would add it to your stack.
The datasource needs a bit of investigation but I believe you might need to create another manifest like the grafana-dashboardDatasources.yaml
. The content of it (the datasources.yaml
line) is a base64 encoded yaml file with the content. You can see it by echoing it's contents to |base64 -d
.
Hope it helped.
To all, in a couple weeks I'll check this stack in a fresh Kubernetes and K3s clusters to see if there are any required updates. Also will update libraries, dependencies and dashboards.
This was now 2⅔ months ago, is there a rough estimate as to when this update will take place?
Sorry to all but I've been pretty busy with other projects and didn't have time (or focus) to update this.
As an open source project, I'd review and take PRs if someone is willing to do it.
I want to make alermanager send an alert to a python webhook, could you please inform me which yaml I have to edit to add new rules to alertmanager? Where should I declare the receiver ?
I want to make alermanager send an alert to a python webhook, could you please inform me which yaml I have to edit to add new rules to alertmanager? Where should I declare the receiver ?
You have to override AlertManager secret that contains the configuration. Webhook config documentation is at https://prometheus.io/docs/alerting/latest/configuration/#webhook_config.
To see how to override, look at the jsonnet sources from the libraries used here in the project.
How do you update web certificates without tearing down and re-installing? I have created a SAN certificate for grafana, prometheus, and alertmanager, and put the server.crt and server.key files in the root base directory with the rest of the files. If I follow the instruction for my platform (K3s) for installing, everything works as expected.
When I renew my cert, what is the correct method for installing the new cert without having the teardown and re-install everything?
To all, in a couple weeks I'll check this stack in a fresh Kubernetes and K3s clusters to see if there are any required updates. Also will update libraries, dependencies and dashboards.
This was now 2⅔ months ago, is there a rough estimate as to when this update will take place?
I have deployed cluster-monitoring on a raspberry pi 4 cluster running v1.20.7+k3s1 and things appear to be working.
v1.20.7+k3s1
I have a new setup, it was done following this points
OS
Linux k3master1 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Installation of K3s using k3sup
--k3s-channel stable \
--ip <ip> \
--user root \
--k3s-extra-args '--cluster-init --kube-apiserver-arg enable-aggregator-routing=true --disable traefik --no-deploy traefik --disable metrics-server --no-deploy metrics-server' \
--datastore postgres://dbconnection
This installs the stable release -- K3s v1.20.7+k3s1.
--cluster-init --kube-apiserver-arg enable-aggregator-routing=true --disable traefik --no-deploy traefik --disable metrics-server --no-deploy metrics-server
Followed the instructions of the manual deploy for monitoring.
Not getting data for many metrics. There was no error in the deployment part.
CPU usage, Memory Usage or anything related usage are not being shown.
EDIT: Just to make things easy to debug, when checked in events. Found issues of deployments not able to reach the ServiceAccounts redployed or started the deployment. The pods have started and logs suggest it is working but still no data for usage.
Hi,
First of all, thank you for this great project. I've successfully deployed the monitoring and now I would like to add another rpi to Prometheus scrapping. This rpi is outside the k8s cluster and already have node_exporter installed. How to I had this node to prometheus?
Firstly, thanks for all the work you put into this @carlosedp 👏🏻. Prometheus seems to be running into an error
panic: mmap: cannot allocate memory
, have you run into this before? Deleting the pod fixes the issue, and I do have memory available. Also - what is the best way to add additional targets? Thanks againroot@pi-master:/home/pi# kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5+k3s1", GitCommit:"58ebdb2a2ec5318ca40649eb7bd31679cb679f71", GitTreeState:"clean", BuildDate:"2020-05-06T23:42:31Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5+k3s1", GitCommit:"58ebdb2a2ec5318ca40649eb7bd31679cb679f71", GitTreeState:"clean", BuildDate:"2020-05-06T23:42:31Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm"} root@pi-master:/home/pi# root@pi-master:/home/pi# cat /etc/os-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs" root@pi-master:/home/pi#
Had the same issue. To solve I moved the OS to an 64bit version, in my case went to Ubuntu.
There is also a 64bit prerelease version of Raspbian if you prefer, but you are probably not going to get around reinstalling, if this is actually the issue.
Hi, Everyhting works fine. Thanks a lot for this cool repo!
One question. Where can I add additionalScrapeConfigs?
Best, Gregor
Thank you for this amazing project.
I've followed the tutorial here: https://kauri.io/#deploy-prometheus-and-grafana-to-monitor-a-kube/186a71b189864b9ebc4ef7c8a9f0a6b5/a
But I've found a fatal error while make deploy
.
I disabled the ingress in the vars.jsonnet but I still get the same error:
error validating "manifests/ingress-alertmanager.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "manifests/ingress-grafana.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "manifests/ingress-prometheus.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false
I have K3s version v1.20.7+k3s1 Thanks
@carlosedp Kubernetes maintainers changed Ingress
from extensions/v1beta1
to networking.k8s.io/v1
A quick and dirty way is to open Ingress-*.yaml
and change the networking.k8s.io/v1
to extensions/v1beta1
However, after K8s releases 1.22, this method will fail
For long term fix:
Ingress-alertmanager.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: alertmanager-main
namespace: monitoring
spec:
tls:
- hosts:
- alertmanager.192.168.1.15.nip.io
rules:
- host: alertmanager.192.168.1.15.nip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: alertmanager-main
port:
name: web
Ingress-grafana.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: grafana
namespace: monitoring
spec:
tls:
- hosts:
- grafana.192.168.1.15.nip.io
rules:
- host: grafana.192.168.1.15.nip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
name: http
Ingress-promethus.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-k8s
namespace: monitoring
spec:
tls:
- hosts:
- prometheus.192.168.1.15.nip.io
rules:
- host: prometheus.192.168.1.15.nip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-k8s
port:
name: web
Make sure you change the host
Hi there,
I am learning kubernetes, so I deployed a small cluster on some Raspberry Pis. However, I cannot reach my grafana instance (grafana.192.168.1.100.nip.io). I updated the ingress files following the post above, but to no avail.
Does anyone have any suggestions on what to do?
Cheers!
Hi,
I configured K3s with MetalLB and for some reason the ingress now doesn't work, is there a way to make the prometheus-k8s-0 to use the hostNetwork: true option? I have added in the prometheus-prometheus.yaml in the spec section but it doesn't seem to work
Hello all,
Thank you for making this project, especially for ARM users! I'm learning/running microk8s and have managed to get all nodes deployed and running. microk8s.kubectl get ingress --all-namespaces
outputs that the hosts should be up and running.
However i'm not able to connect in browser. I'm aware microk8s isn't officially supported so unsure if it is an issue with this version of kubernetes. This has happened with and without applying the fixes for ingress-*.yml
files.
Thank you!
Thank you for this amazing project.
I've followed the tutorial here: https://kauri.io/#deploy-prometheus-and-grafana-to-monitor-a-kube/186a71b189864b9ebc4ef7c8a9f0a6b5/a
But I've found a fatal error while
make deploy
. I disabled the ingress in the vars.jsonnet but I still get the same error:error validating "manifests/ingress-alertmanager.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false error validating "manifests/ingress-grafana.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false error validating "manifests/ingress-prometheus.yaml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "servicePort" in io.k8s.api.networking.v1.IngressBackend]; if you choose to ignore these errors, turn validation off with --validate=false
I have K3s version v1.20.7+k3s1 Thanks
I had a same error on k3s version v1.21.2+k3s1 (5a67e8dc) go version go1.16.4
There are 3 master node on EC2 and jetson nano(arm64) via the VPN network, and else are Raspberry Pi arm64.
$sudo kubectl get node
NAME STATUS ROLES AGE VERSION
ip-xxx-xxx-xxx-xxx Ready control-plane,master 14d v1.21.1+k3s1
ip-yyy-yyy-yyy-yyy Ready control-plane,master 14d v1.21.1+k3s1
pi4-node2 Ready <none> 33m v1.21.2+k3s1
jetson-master Ready control-plane,master 14d v1.21.2+k3s1
pi4-node1 Ready <none> 37m v1.21.2+k3s1
Thanks
I don't see how this can work with later version of k3s, since they disabled metrics listening on any interface other than 127.0.0.1
https://github.com/k3s-io/k3s/issues/425 https://github.com/k3s-io/k3s/commit/4808c4e7d53db310fb324b2157386e50ebef5167
Since I don't have too many resources and time to address all questions regarding the deployments, the Issues section is a place to report problems or improvements to the stack.
This issue is a place where you can add a comment in case of a question where me or any community member can answer in a best effort manner.
If you deployed the monitoring stack and some targets are not available or showing no metrics in Grafana, make sure you don't have IPTables rules or use a firewall on your nodes before deploying Kubernetes.
If you don't want to receive further notifications, click "Unsubscribe" in the right bar, right above the participants list.