Closed kinoxyz1 closed 1 week ago
这是 kube-state-metrics pod的完整日志信息
I0815 06:18:30.462470 1 main.go:95] Using default resources
--
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.462728 1 main.go:107] Using all namespace
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.462739 1 main.go:128] metric allow-denylisting: Excluding the following lists that were on denylist:
Thu, Aug 15 2024 2:18:30 pm | W0815 06:18:30.462751 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.463335 1 main.go:234] Testing communication with server
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.467603 1 main.go:239] Running with Kubernetes cluster version: v1.21. git version: v1.21.14. git tree state: clean. commit: 0f77da5bd4809927e15d1658fb4aa8f13ad890a5. platform: linux/amd64
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.467613 1 main.go:241] Communication with server successful
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.467801 1 main.go:197] Starting metrics server: 0.0.0.0:8080
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.467803 1 metrics_handler.go:96] Autosharding disabled
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.467806 1 main.go:186] Starting kube-state-metrics self metrics server: 0.0.0.0:8081
Thu, Aug 15 2024 2:18:30 pm | I0815 06:18:30.468248 1 builder.go:164] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
Thu, Aug 15 2024 2:18:30 pm | W0815 06:18:30.469676 1 warnings.go:67] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Thu, Aug 15 2024 2:18:30 pm | W0815 06:18:30.470967 1 warnings.go:67] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Thu, Aug 15 2024 2:18:30 pm | W0815 06:18:30.473605 1 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Thu, Aug 15 2024 2:18:30 pm | W0815 06:18:30.474726 1 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Thu, Aug 15 2024 2:25:50 pm | W0815 06:25:50.475517 1 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Thu, Aug 15 2024 2:26:56 pm | W0815 06:26:56.471958 1 warnings.go:67] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Thu, Aug 15 2024 2:33:25 pm | W0815 06:33:25.472651 1 warnings.go:67] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Thu, Aug 15 2024 2:33:29 pm | W0815 06:33:29.476647 1 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Thu, Aug 15 2024 2:40:09 pm | W0815 06:40:09.473628 1 warnings.go:67] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Thu, Aug 15 2024 2:42:02 pm | W0815 06:42:02.477490 1 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Can you uninstall volcano and re-install volcano v1.9.0 and use installer/volcano-monitoring-latest.yaml to deploy monitoring in master branch to try once more?
Can you uninstall volcano and re-install volcano v1.9.0 and use installer/volcano-monitoring-latest.yaml to deploy monitoring in master branch to try once more?
Do I have to upgrade? Upgrading may not guarantee the operation of online services. Or rather, I just want to successfully deploy a monitoring system. Do you have any other solutions?
Please also paste the prometheus pod logs.
Please also paste the prometheus pod logs.
Okay, first of all, thank you for your reply. This is all the logs of Prometheus from startup to now:
ts=2024-08-15T08:08:21.559Z caller=main.go:589 level=info msg="No time or size retention was set so using the default time retention" duration=15d
--
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.559Z caller=main.go:633 level=info msg="Starting Prometheus Server" mode=server version="(version=2.53.2, branch=HEAD, revision=6e971a7dc905696d4bc4ffa150bf282fcfac5fa9)"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.559Z caller=main.go:638 level=info build_context="(go=go1.22.6, platform=linux/amd64, user=root@363b0aa99939, date=20240809-14:55:04, tags=netgo,builtinassets,stringlabels)"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.559Z caller=main.go:639 level=info host_details="(Linux 5.4.0-150-generic #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 prometheus-deployment-6468db6ddc-gdpgz (none))"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.559Z caller=main.go:640 level=info fd_limits="(soft=1048576, hard=1048576)"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.559Z caller=main.go:641 level=info vm_limits="(soft=unlimited, hard=unlimited)"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.561Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.561Z caller=main.go:1148 level=info msg="Starting TSDB ..."
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.562Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9090
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.562Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.563Z caller=head.go:626 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.563Z caller=head.go:713 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=803ns
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.563Z caller=head.go:721 level=info component=tsdb msg="Replaying WAL, this may take a while"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.563Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.563Z caller=head.go:830 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=16.607µs wal_replay_duration=500.625µs wbl_replay_duration=101ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=803ns total_replay_duration=530.767µs
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.565Z caller=main.go:1169 level=info fs_type=EXT4_SUPER_MAGIC
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.565Z caller=main.go:1172 level=info msg="TSDB started"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.565Z caller=main.go:1354 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=kubernetes.go:331 level=info component="discovery manager scrape" discovery=kubernetes config=kubernetes-cadvisor msg="Using pod service account via in-cluster config"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=kubernetes.go:331 level=info component="discovery manager scrape" discovery=kubernetes config=kubernetes-service-endpoints msg="Using pod service account via in-cluster config"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=kubernetes.go:331 level=info component="discovery manager scrape" discovery=kubernetes config=kubernetes-pods msg="Using pod service account via in-cluster config"
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=main.go:1391 level=info msg="updated GOGC" old=100 new=75
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=main.go:1402 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.294952ms db_storage=617ns remote_storage=798ns web_handler=174ns query_engine=387ns scrape=151.926µs scrape_sd=426.727µs notify=9.488µs notify_sd=3.756µs rules=237.831µs tracing=2.791µs
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=main.go:1133 level=info msg="Server is ready to receive web requests."
Thu, Aug 15 2024 4:08:21 pm | ts=2024-08-15T08:08:21.566Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."
Thu, Aug 15 2024 4:09:32 pm | ts=2024-08-15T08:09:32.499Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:10:37 pm | ts=2024-08-15T08:10:37.513Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:11:42 pm | ts=2024-08-15T08:11:42.479Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:12:47 pm | ts=2024-08-15T08:12:47.477Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:13:52 pm | ts=2024-08-15T08:13:52.485Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:14:57 pm | ts=2024-08-15T08:14:57.484Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:16:02 pm | ts=2024-08-15T08:16:02.490Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:17:07 pm | ts=2024-08-15T08:17:07.518Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:18:14 pm | ts=2024-08-15T08:18:14.485Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:19:17 pm | ts=2024-08-15T08:19:17.501Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:20:22 pm | ts=2024-08-15T08:20:22.481Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:21:27 pm | ts=2024-08-15T08:21:27.483Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:22:32 pm | ts=2024-08-15T08:22:32.569Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:23:37 pm | ts=2024-08-15T08:23:37.489Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:24:42 pm | ts=2024-08-15T08:24:42.477Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:25:47 pm | ts=2024-08-15T08:25:47.492Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:26:52 pm | ts=2024-08-15T08:26:52.481Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:27:57 pm | ts=2024-08-15T08:27:57.491Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:29:02 pm | ts=2024-08-15T08:29:02.485Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:30:07 pm | ts=2024-08-15T08:30:07.484Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:31:12 pm | ts=2024-08-15T08:31:12.483Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:32:17 pm | ts=2024-08-15T08:32:17.497Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:33:22 pm | ts=2024-08-15T08:33:22.577Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:34:27 pm | ts=2024-08-15T08:34:27.496Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:35:32 pm | ts=2024-08-15T08:35:32.493Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:36:37 pm | ts=2024-08-15T08:36:37.500Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:37:42 pm | ts=2024-08-15T08:37:42.486Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:38:47 pm | ts=2024-08-15T08:38:47.508Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:39:52 pm | ts=2024-08-15T08:39:52.476Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:40:57 pm | ts=2024-08-15T08:40:57.623Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:42:02 pm | ts=2024-08-15T08:42:02.490Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:43:07 pm | ts=2024-08-15T08:43:07.477Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:44:12 pm | ts=2024-08-15T08:44:12.899Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:45:18 pm | ts=2024-08-15T08:45:18.882Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:46:22 pm | ts=2024-08-15T08:46:22.487Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:47:29 pm | ts=2024-08-15T08:47:29.486Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:48:32 pm | ts=2024-08-15T08:48:32.487Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:49:37 pm | ts=2024-08-15T08:49:37.488Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:50:42 pm | ts=2024-08-15T08:50:42.484Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:51:47 pm | ts=2024-08-15T08:51:47.479Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:52:52 pm | ts=2024-08-15T08:52:52.475Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:53:57 pm | ts=2024-08-15T08:53:57.482Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:55:02 pm | ts=2024-08-15T08:55:02.485Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:56:07 pm | ts=2024-08-15T08:56:07.484Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:57:12 pm | ts=2024-08-15T08:57:12.479Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:58:17 pm | ts=2024-08-15T08:58:17.493Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 4:59:22 pm | ts=2024-08-15T08:59:22.500Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:00:27 pm | ts=2024-08-15T09:00:27.482Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:01:32 pm | ts=2024-08-15T09:01:32.484Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:02:37 pm | ts=2024-08-15T09:02:37.481Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:03:43 pm | ts=2024-08-15T09:03:43.838Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:04:47 pm | ts=2024-08-15T09:04:47.505Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:05:52 pm | ts=2024-08-15T09:05:52.507Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:06:57 pm | ts=2024-08-15T09:06:57.505Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:08:02 pm | ts=2024-08-15T09:08:02.485Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:09:07 pm | ts=2024-08-15T09:09:07.490Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:10:12 pm | ts=2024-08-15T09:10:12.488Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:11:17 pm | ts=2024-08-15T09:11:17.545Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:12:22 pm | ts=2024-08-15T09:12:22.491Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:13:27 pm | ts=2024-08-15T09:13:27.497Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:14:32 pm | ts=2024-08-15T09:14:32.483Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Thu, Aug 15 2024 5:15:38 pm | ts=2024-08-15T09:15:38.544Z caller=notifier.go:549 level=error component=notifier alertmanager=http://alertmanager.monitoring.svc:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://alertmanager.monitoring.svc:9093/api/v2/alerts\": dial tcp: lookup alertmanager.monitoring.svc on 10.64.0.10:53: no such host"
Maybe we also take a look at grafana's logs. : )
The problem has been solved, here are the steps to solve it:
kubectl apply -f installer/volcano-monitoring-latest.yaml
2.modify the metrics image version More detailed documentation reference: https://github.com/volcano-sh/kube-state-metrics
kubectl edit deploy prometheus-release-kube-state-metrics
...
image: volcanosh/kube-state-metrics:v2.0.0-beta
...
Here is the complete display:
The kube-state-metrics image version in installer/volcano-monitoring-latest.yaml is already docker.io/volcanosh/kube-state-metrics:v2.0.0-beta
: )
/close
@Monokaix: Closing this issue.
Please describe your problem in detail
版本信息
问题: Grafana 版本大部分指标都无数据,如下图
经检查发现是Prometheus中根本不存在相关指标。以下是几种我尝试过的部署方式,请帮我看看是否存在问题。
尝试一: 按照release-1.7 tag 中的 installer/volcano-monitoring-v1.7.0.yaml 执行后,以如下看板为例
kube_pod_volcano_container_resource_requests 指标在Prometheus中并不存在
尝试二: 按照 master branch 中的 installer/volcano-monitoring-latest.yaml 执行后,还是如上指标不存在的问题。
尝试三: 按照kube-state-metrics项目中的步骤,执行kubectl apply -f examples/standard 之后,还是如上指标不存在的问题。
Any other relevant information
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-15T14:17:29Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}$ cat /etc/os-release
NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.6 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic$ uname -a
Linux jz-desktop-08 5.4.0-150-generic #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux