search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.49k stars 773 forks source link

Nothing to do for Prometheus #1576

Closed m-jayson closed 3 years ago

m-jayson commented 4 years ago

I tried enabling prometheus on my microk8s cluster on raspberry pi however it is giving me this below result. image

It seems prometheus is not part of the default addons image

I can access grafana image

There is influxdb however, we like to use prometheus as we have experience on that.

I'm following this example but since there is no prometheus it looks like i cant https://ubuntu.com/blog/monitoring-at-the-edge-with-microk8s

lastly this is my microk8s version --classic --channel=1.18/stable

balchua commented 4 years ago

@mgonzaga1990 thanks for reporting the issue. Back then like a couple of months ago, the kube-prometheus is having some problem on arm64. Based on this issue https://github.com/prometheus-operator/kube-prometheus/issues/545 it looks to be good now. @ktsakalozos is there anyway we can try this on an arm arch?

m-jayson commented 4 years ago

@balchua thanks,

meanwhile, i tried accessing influxdb https://192.168.1.150:16443/api/v1/namespaces/kube-system/services/monitoring-influxdb:http/proxy/

from image

however it is giving me. Error trying to reach service: 'dial tcp 10.1.97.129:8083: connect: connection refused'

but i can access grafana

balchua commented 4 years ago

Fyi the influxdb and heapster were removed from MicroK8s 1.19 channel. The kube dashboard comes with the scraper and dashboard alone.

Coming back to your issue, what about port forwarding?

m-jayson commented 4 years ago

@balchua scraper? sounds like prometheus to me? does that mean it has built in prometheus? how about grafana? will still be there? if that is the case i'll update my version to 1.19 channel. but i need to verify it first

balchua commented 4 years ago

Sorry i may have confused you. The dashboard scraper is not prometheus. It is part of the dashboard pods to get some metrics. It doesn't come with prometheus nor grafana.

The monitoring stack will still be the prometheus addon. Which is based on kube-prometheus. Its unfortunate that i dont have an arm64 box to test it.

m-jayson commented 4 years ago

i see. but let me verify my understanding.. so will there be prometheus on 1.19? how about grafana will still be there?. Sorry I'm still exploring microk8s and planning to stick with it

m-jayson commented 4 years ago

@balchua if you could tell me what to do i could help you test it in arm64

balchua commented 4 years ago

@mgonzaga1990 there is a file named /snap/microk8s/current/microk8s-resources/wrappers/addon-list.yaml where we specify whether the addon is available for which architecture. For example:

    - name: "prometheus"
      description: "Prometheus operator for monitoring and logging"
      version: "v0.6.0"
      check_status: "pod/prometheus-k8s-0"
      supported_architectures:
      - amd64

But this file is read only. I don't know how to mount it to a writable filesystem.

m-jayson commented 4 years ago

@balchua

I think that is different in my end.

image

cant see microk8s-resources

m-jayson commented 4 years ago

@balchua i think as of now i may have to manually deploy prometheus in my cluster.. what image would you recommend for arm64?

balchua commented 4 years ago

@mgonzaga1990 this is how we package the kube-prometheus https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.prometheus.sh Basically we just reduce the replicas.

m-jayson commented 4 years ago

@balchua thanks for this.. i'm planning to upgrade now.. btw. what is the difference between classic and stable?

what would you recommend for raspberry pi?

balchua commented 4 years ago

@mgonzaga1990 you can find here more information about the channels.

m-jayson commented 4 years ago

@balchua i think i have to stick with 1.18 as of now.. i tried using the 1.19 but at some point my raspberry pi freezes when running pipeline (drone and gitea) which i don't experience with 1.18

balchua commented 4 years ago

If it is not too much, will it be possible to get the inspect tarball? Will really help to troubleshoot it.

m-jayson commented 4 years ago

@balchua sure sure.. i'll geenrate it and let you know once available

balchua commented 4 years ago

@mgonzaga1990 perhaps opening a new issue with the 1.19 freezing up your rpis will be a good way to track that. Wdyt?

m-jayson commented 4 years ago

yup sure... ill set up my cluster back to 1.19

On Tue, Sep 22, 2020, 6:58 PM balchua notifications@github.com wrote:

@mgonzaga1990 https://github.com/mgonzaga1990 perhaps opening a new issue with the 1.19 freezing up your rpis will be a good way to track that. Wdyt?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ubuntu/microk8s/issues/1576#issuecomment-696649304, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBKRM7IETYGC67FPY2JNF3SHB7PDANCNFSM4RPGQ7BQ .

nderraugh commented 4 years ago

I have the same problem on the 1.19 channel. I'm on a pi 4b and ubuntu 20.04.1.

I think the arm64 issue may be a red herring. I am new to microk8s and snap, but looking through /snap/microk8s/1669/microk8s-enable.wrapper it looks like there is neither a "${SNAP}/actions/enable.$action.sh" nor a "${SNAP}/actions/$action.yaml" for prometheus. Hence the Nothing to do for prometheus message.

I ran snap run --shell microk8s.status to set my env vars and then ran /snap/microk8s/1669/microk8s-resources/actions/enable.prometheus.sh and made considerable progress.

Here's the output:

Enabling DNS
Applying manifest
serviceaccount/coredns unchanged
configmap/coredns unchanged
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Fetching kube-prometheus version v0.6.0.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   143  100   143    0     0    242      0 --:--:-- --:--:-- --:--:--   242
100  267k    0  267k    0     0   156k      0 --:--:--  0:00:01 --:--:-- 1342k
kube-prometheus-0.6.0/
kube-prometheus-0.6.0/.github/
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/bug.md
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/feature.md
kube-prometheus-0.6.0/.github/ISSUE_TEMPLATE/support.md
kube-prometheus-0.6.0/.gitignore
kube-prometheus-0.6.0/.travis.yml
kube-prometheus-0.6.0/DCO
kube-prometheus-0.6.0/LICENSE
kube-prometheus-0.6.0/Makefile
kube-prometheus-0.6.0/NOTICE
kube-prometheus-0.6.0/OWNERS
kube-prometheus-0.6.0/README.md
kube-prometheus-0.6.0/build.sh
kube-prometheus-0.6.0/code-of-conduct.md
kube-prometheus-0.6.0/docs/
kube-prometheus-0.6.0/docs/EKS-cni-support.md
kube-prometheus-0.6.0/docs/GKE-cadvisor-support.md
kube-prometheus-0.6.0/docs/developing-prometheus-rules-and-grafana-dashboards.md
kube-prometheus-0.6.0/docs/exposing-prometheus-alertmanager-grafana-ingress.md
kube-prometheus-0.6.0/docs/kube-prometheus-on-kubeadm.md
kube-prometheus-0.6.0/docs/monitoring-external-etcd.md
kube-prometheus-0.6.0/docs/monitoring-other-namespaces.md
kube-prometheus-0.6.0/docs/weave-net-support.md
kube-prometheus-0.6.0/example.jsonnet
kube-prometheus-0.6.0/examples/
kube-prometheus-0.6.0/examples/additional-namespaces-servicemonitor.jsonnet
kube-prometheus-0.6.0/examples/additional-namespaces.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config-external.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config.jsonnet
kube-prometheus-0.6.0/examples/alertmanager-config.yaml
kube-prometheus-0.6.0/examples/auth
kube-prometheus-0.6.0/examples/basic-auth/
kube-prometheus-0.6.0/examples/basic-auth/secrets.yaml
kube-prometheus-0.6.0/examples/basic-auth/service-monitor.yaml
kube-prometheus-0.6.0/examples/eks-cni-example.jsonnet
kube-prometheus-0.6.0/examples/etcd-client-ca.crt
kube-prometheus-0.6.0/examples/etcd-client.crt
kube-prometheus-0.6.0/examples/etcd-client.key
kube-prometheus-0.6.0/examples/etcd-skip-verify.jsonnet
kube-prometheus-0.6.0/examples/etcd.jsonnet
kube-prometheus-0.6.0/examples/example-app/
kube-prometheus-0.6.0/examples/example-app/example-app.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-alertmanager-discovery-role-binding.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-alertmanager-discovery-role.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-role-binding.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-role.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-service-account.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend-svc.yaml
kube-prometheus-0.6.0/examples/example-app/prometheus-frontend.yaml
kube-prometheus-0.6.0/examples/example-app/servicemonitor-frontend.yaml
kube-prometheus-0.6.0/examples/example-grafana-dashboard.json
kube-prometheus-0.6.0/examples/existingrule.json
kube-prometheus-0.6.0/examples/existingrule.yaml
kube-prometheus-0.6.0/examples/grafana-additional-jsonnet-dashboard-example.jsonnet
kube-prometheus-0.6.0/examples/grafana-additional-rendered-dashboard-example-2.jsonnet
kube-prometheus-0.6.0/examples/grafana-additional-rendered-dashboard-example.jsonnet
kube-prometheus-0.6.0/examples/ingress.jsonnet
kube-prometheus-0.6.0/examples/internal-registry.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-build-snippet/
kube-prometheus-0.6.0/examples/jsonnet-build-snippet/build-snippet.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/
kube-prometheus-0.6.0/examples/jsonnet-snippets/bootkube.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kops-coredns.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kops.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kube-aws.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kubeadm.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/kubespray.jsonnet
kube-prometheus-0.6.0/examples/jsonnet-snippets/node-ports.jsonnet
kube-prometheus-0.6.0/examples/ksonnet-example.jsonnet
kube-prometheus-0.6.0/examples/kustomize.jsonnet
kube-prometheus-0.6.0/examples/minikube.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-alert-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-recording-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-additional-rendered-rule-example.jsonnet
kube-prometheus-0.6.0/examples/prometheus-name-override.jsonnet
kube-prometheus-0.6.0/examples/prometheus-pvc.jsonnet
kube-prometheus-0.6.0/examples/tolerations.libsonnet
kube-prometheus-0.6.0/examples/weave-net-example.jsonnet
kube-prometheus-0.6.0/experimental/
kube-prometheus-0.6.0/experimental/metrics-server/
kube-prometheus-0.6.0/experimental/metrics-server/auth-delegator.yaml
kube-prometheus-0.6.0/experimental/metrics-server/auth-reader.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-apiservice.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-cluster-role-binding.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-cluster-role.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-deployment.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-service-account.yaml
kube-prometheus-0.6.0/experimental/metrics-server/metrics-server-service.yaml
kube-prometheus-0.6.0/go.mod
kube-prometheus-0.6.0/go.sum
kube-prometheus-0.6.0/hack/
kube-prometheus-0.6.0/hack/example-service-monitoring/
kube-prometheus-0.6.0/hack/example-service-monitoring/deploy
kube-prometheus-0.6.0/hack/example-service-monitoring/teardown
kube-prometheus-0.6.0/hack/jsonnet-docker-image
kube-prometheus-0.6.0/jsonnet/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/.gitignore
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alertmanager/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alertmanager/alertmanager.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/alertmanager.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/alerts.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/general.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/node.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/prometheus-operator.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/alerts/tests.yaml
kube-prometheus-0.6.0/jsonnet/kube-prometheus/dropping-deprecated-metrics-relabelings.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/grafana-weave-net-cluster.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/grafana-weave-net.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/jsonnetfile.json
kube-prometheus-0.6.0/jsonnet/kube-prometheus/ksm-autoscaler/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/ksm-autoscaler/ksm-autoscaler.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-all-namespaces.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-anti-affinity.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-bootkube.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-config-mixins.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-custom-metrics.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-eks.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-insecure-kubelet.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kops-coredns.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kops.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-ksonnet.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kube-aws.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kubeadm.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-kubespray.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-managed-cluster.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-node-ports.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-static-etcd.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-strip-limits.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus-weave-net.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-prometheus.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-rbac-proxy/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-rbac-proxy/container.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-state-metrics/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/kube-state-metrics/kube-state-metrics.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/image.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/lib/lib.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/node-exporter/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/node-exporter/node-exporter.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus-adapter/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus-adapter/prometheus-adapter.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/prometheus/prometheus.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/general.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/node-rules.libsonnet
kube-prometheus-0.6.0/jsonnet/kube-prometheus/rules/rules.libsonnet
kube-prometheus-0.6.0/jsonnetfile.json
kube-prometheus-0.6.0/jsonnetfile.lock.json
kube-prometheus-0.6.0/kustomization.yaml
kube-prometheus-0.6.0/manifests/
kube-prometheus-0.6.0/manifests/alertmanager-alertmanager.yaml
kube-prometheus-0.6.0/manifests/alertmanager-secret.yaml
kube-prometheus-0.6.0/manifests/alertmanager-service.yaml
kube-prometheus-0.6.0/manifests/alertmanager-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/alertmanager-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardDatasources.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardDefinitions.yaml
kube-prometheus-0.6.0/manifests/grafana-dashboardSources.yaml
kube-prometheus-0.6.0/manifests/grafana-deployment.yaml
kube-prometheus-0.6.0/manifests/grafana-service.yaml
kube-prometheus-0.6.0/manifests/grafana-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/grafana-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-clusterRole.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-deployment.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-service.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/kube-state-metrics-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/node-exporter-clusterRole.yaml
kube-prometheus-0.6.0/manifests/node-exporter-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/node-exporter-daemonset.yaml
kube-prometheus-0.6.0/manifests/node-exporter-service.yaml
kube-prometheus-0.6.0/manifests/node-exporter-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/node-exporter-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-apiService.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRole.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleBindingDelegator.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-clusterRoleServerResources.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-configMap.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-deployment.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-roleBindingAuthReader.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-service.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/prometheus-adapter-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-clusterRole.yaml
kube-prometheus-0.6.0/manifests/prometheus-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/prometheus-operator-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-prometheus.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleBindingConfig.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleBindingSpecificNamespaces.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleConfig.yaml
kube-prometheus-0.6.0/manifests/prometheus-roleSpecificNamespaces.yaml
kube-prometheus-0.6.0/manifests/prometheus-rules.yaml
kube-prometheus-0.6.0/manifests/prometheus-service.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceAccount.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitor.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorApiserver.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorCoreDNS.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubeControllerManager.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubeScheduler.yaml
kube-prometheus-0.6.0/manifests/prometheus-serviceMonitorKubelet.yaml
kube-prometheus-0.6.0/manifests/setup/
kube-prometheus-0.6.0/manifests/setup/0namespace-namespace.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0podmonitorCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0prometheusruleCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0servicemonitorCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-clusterRole.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-clusterRoleBinding.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-deployment.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-service.yaml
kube-prometheus-0.6.0/manifests/setup/prometheus-operator-serviceAccount.yaml
kube-prometheus-0.6.0/scripts/
kube-prometheus-0.6.0/scripts/minikube-start-kvm.sh
kube-prometheus-0.6.0/scripts/minikube-start.sh
kube-prometheus-0.6.0/scripts/monitoring-deploy.sh
kube-prometheus-0.6.0/scripts/tools.go
kube-prometheus-0.6.0/sync-to-internal-registry.jsonnet
kube-prometheus-0.6.0/test.sh
kube-prometheus-0.6.0/tests/
kube-prometheus-0.6.0/tests/e2e/
kube-prometheus-0.6.0/tests/e2e/main_test.go
kube-prometheus-0.6.0/tests/e2e/prometheus_client.go
kube-prometheus-0.6.0/tests/e2e/travis-e2e.sh
Enabling Prometheus
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created
The Prometheus operator is enabled (user/pass: admin/admin)

In the pods that were created there are still several problems. Specifically the logs of several crashing pods mention exec format error (related to the arch?). But I think if somone who knows snap and microk8s better than me can connect some dots here we can resolve the original "Nothing to do for Prometheus" issue and look at solving these pod failures if they still exist.

nderraugh commented 4 years ago

Following up from the link to https://github.com/prometheus-operator/kube-prometheus/issues/545 I removed microk8s 1.19 and intalled 1.18 (I hadn't noticed that kube-prometheus support for 1.19 was still at HEAD and not 0.6). I then followed the same procedure as I outlined in my previous post and it failed with error: the path "/snap/microk8s/1705/actions/prometheus/setup" does not exist.

I ran through the commands in enable.prometheus.sh by hand and it fails at $KUBECTL apply -f "${SNAP}/actions/prometheus/setup". I changed it to $KUBECTL apply -f "${SNAP}/microk8s-resources/actions/prometheus/setup" and it ran successfully. 

namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created

The next step failed with the same path problem. But I changed it to ($KUBECTL apply -f "${SNAP}/microk8s-resources/actions/prometheus/") and it ran successfully.

alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-pods created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
role.rbac.authorization.k8s.io/kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

The pods that start up still have the same exec format eror in their logs though, except for grafana which has this error 2020-10-11T16:31:06.142685100Z /run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory.

FWIW, cloning the kube-prometheus repo and running through the Quickstart resulted in prometheus-operator coming up successfully, followed by node-exporter, grafana, prometheus-adapter, alert-manager-main-{0,1,2}, prometheus-k8s-{0,1}. However the kube-state-metrics container still fails with exec format error in the logs.

nderraugh commented 4 years ago

As a workaround on 1.19 you can do as suggested in https://github.com/kubernetes/kube-state-metrics/pull/1190#issuecomment-667026038 and modify https://github.com/prometheus-operator/kube-prometheus/blob/980e95de011319b88a3b9c0787a81dcdf338a898/manifests/kube-state-metrics-deployment.yaml#L26 from

image: quay.io/coreos/kube-state-metrics:v1.9.7

to

image: gcr.io/k8s-staging-kube-state-metrics/kube-state-metrics-arm64:v1.9.7

and everything comes up as expected.

seangustafson commented 3 years ago

I can confirm that the prometheus add-on installed successfully in microk8s 1.20 on my raspberry pi cluster running Ubuntu 20.04.1 (like the others in this thread I had problems with 1.19). Given that, I think this issue can now be closed.

balchua commented 3 years ago

Thanks @seangustafson, this addon also works on mixed clusters. I.e a mix of arm64 and amd64 nodes.