run Prometheus no node

[ppabloescobar]

hello.

How to run a prometheus outside the node,there is a possibility.

[jtlisi]

You can run your Prometheus external to the the cluster. You can find more information on how to do that here

You will have to update the kubernetes sd config for prometheus to access your cluster.

More information on that can be found here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

[ppabloescobar]

msg="Cannot create Kubernetes discovery" err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined"

[ppabloescobar]

scrape_configs:

• job_name: 'kubernetes-kubelet' scheme: https tls_config: ca_file: /opt/prometheus/k8s_cert_pre/ca.crt insecure_skip_verify: true bearer_token_file: /opt/prometheus/k8s_cert_pre/token kubernetes_sd_configs:
  • role: node relabel_configs:
  • action: labelmap regex: __meta_kubernetes_nodelabel(.+)
  • target_label: address replacement: 126.201.27.100:443 (hapster IP)
  • source_labels: [meta_kubernetes_node_name] regex: (.+) target_label: metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics
• job_name: 'kubernetes-cadvisor' scheme: https tls_config: ca_file: /opt/prometheus/k8s_cert_pre/ca.crt insecure_skip_verify: true bearer_token_file: /opt/prometheus/k8s_cert_pre/token kubernetes_sd_configs:
  • role: node relabel_configs:
  • action: labelmap regex: __meta_kubernetes_nodelabel(.+)
  • target_label: address replacement: 126.201.27.100:443 (hapster IP)
  • source_labels: [meta_kubernetes_node_name] regex: (.+) target_label: metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
• job_name: 'kubernetes-kube-state' kubernetes_sd_configs:
  • role: pod relabel_configs:
  • action: labelmap regex: __meta_kubernetes_podlabel(.+)
  • source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name
  • source_labels: [__meta_kubernetes_pod_label_grafanak8sapp] regex: .true. action: keep
  • source_labels: ['meta_kubernetes_pod_label_daemon', 'meta_kubernetes_pod_node_name'] regex: 'node-exporter;(.*)' action: replace target_label: nodename

[jtlisi]

@bernapa0 You are missing the api_server config.

Excerpt from the documentation linked in my first comment:

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

Because you are not setting the api_server config, the Prometheus is running as if it is in the cluster, which it is not.

[prasad1204chinni]

I am trying to replicate the setup. I am adding below config to my external prometheus server.

• job_name: 'kubernetes-pods' scheme: https kubernetes_sd_configs:

  • role: endpoints api_server: 'url:443'

    bearer_token:

    tls_config:

    ca_file: /etc/prometheus/ca.cert

    insecure_skip_verify: true

    bearer_token_file: /etc/prometheus/bearer-token

  relabel_configs:

  • action: labelmap regex: __meta_kubernetes_podlabel(.+)
  • source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name

What iis the correct way of adding tls config or the bearer token.

If i use bearer token should i still give cert ? ALso could this be due to the default user being used ?

I get below error message.

level=error ts=2020-11-20T19:54:42.790Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:263: Failed to list *v1.Pod: the server rejected our request for an unknown reason (get pods)"