Closed nwaistd closed 2 years ago
@nwaistd: The label(s) triage/support
cannot be applied, because the repository doesn't have them.
I'd had the same problem until I added the ingress resource (but I have baremetal). But now I have the same problem with TCP. TCP services don't provide metrics like nginx_ingress_controller_requests etc. Do you know what to do with that?
What happens if you try this https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/
/remove-kind support /triage needs-information
I use it. But the most part of panels with metrics are not available (No data). I can see only Average Memory Usage, Average CPU Usage, Controller Connections and Config Reloads.
Show screenshots.
You won't get TCP metrics i am almost sure but not 100%
@longwuyuan could you give me some links to relevant docs?
There is an nginx_ingress_controller_requests metric here, but it is not updated, because I do not use ingress resource. This appeared after the launch of ingress resource.
Prometheus metrics:
Dashboards:
@longwuyuan could you give me some links to relevant docs?
The link is already in the previous message. If you can install as per those docs and deploy monitoring as per those docs, maybe we will get some helpful info. But don't expect TCP traffic graphs in the dashboard you get from docs based install.
Who is your LoadBalancer provider ? Do they have monitoring for the LB they provide ?
We use metallb (https://metallb.universe.tf/configuration/) in layer2 mode. But it doesn't provide any useful metrics.
@nwaistd did you install as per the link I sent ?
@KozyarValeriy I don't think TCP metrics are exported but someone has to check and confirm. It will take a while. Hope I can check and update but very soon
@longwuyuan Oh, I mean about TCP metrics, why can't get TCP metrics and how to get them.
I set up ingress-controller(nlb) and prometheus in AWS EKS cluster. (k8s v1.19) as guided in the link. (added port 10254 in ingress-controller service and deployment)
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.100.8.94 ... 80:31999/TCP,443:32245/TCP,10254:32351/TCP 40m
ingress-nginx-controller-admission ClusterIP 10.100.104.162 <none> 443/TCP 40m
prometheus-server NodePort 10.100.227.128 <none>
9090:32371/TCP 37m
As you can see, all target labels are set to dropped.
I can get following metrics including 404 status metrics in a pod, though.
bash-5.1$ curl localhost:10254/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.198e-05
go_gc_duration_seconds{quantile="0.25"} 4.8626e-05
go_gc_duration_seconds{quantile="0.5"} 6.0267e-05
go_gc_duration_seconds{quantile="0.75"} 7.6392e-05
go_gc_duration_seconds{quantile="1"} 0.000169321
go_gc_duration_seconds_sum 0.001709751
go_gc_duration_seconds_count 26
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 90
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 7.54696e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 8.9655112e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.476707e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.023388e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 9.220351350273963e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.632656e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 7.54696e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.5861248e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.216e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 72698
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.5631872e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5077248e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6192428766532521e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.096086e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 138720
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 180224
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.877912e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 674061
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.998848e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.998848e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.5056128e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client
# TYPE nginx_ingress_controller_bytes_sent histogram
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100"} 0
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1e+06"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1e+07"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_bytes_sent_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 274
nginx_ingress_controller_bytes_sent_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_config_hash Running configuration hash actually running
# TYPE nginx_ingress_controller_config_hash gauge
nginx_ingress_controller_config_hash{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 9.155461334347864e+18
# HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful
# TYPE nginx_ingress_controller_config_last_reload_successful gauge
nginx_ingress_controller_config_last_reload_successful{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1
# HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge
nginx_ingress_controller_config_last_reload_successful_timestamp_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.619240952e+09
# HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress
# TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.5"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.9"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.99"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service=""} 0
nginx_ingress_controller_ingress_upstream_latency_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service=""} 1
# HELP nginx_ingress_controller_leader_election_status Gauge reporting status of the leader election, 0 indicates follower, 1 indicates leader. 'name' is the string used to identify the lease
# TYPE nginx_ingress_controller_leader_election_status gauge
nginx_ingress_controller_leader_election_status{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",name="ingress-controller-leader-nginx"} 1
# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting}
# TYPE nginx_ingress_controller_nginx_process_connections gauge
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="active"} 1
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="reading"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="waiting"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="writing"} 1
# HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled}
# TYPE nginx_ingress_controller_nginx_process_connections_total counter
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="accepted"} 426
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="handled"} 426
# HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds
# TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter
nginx_ingress_controller_nginx_process_cpu_seconds_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 0.9600000000000001
# HELP nginx_ingress_controller_nginx_process_num_procs number of processes
# TYPE nginx_ingress_controller_nginx_process_num_procs gauge
nginx_ingress_controller_nginx_process_num_procs{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 8
# HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge
nginx_ingress_controller_nginx_process_oldest_start_time_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.61924095e+09
# HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read
# TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter
nginx_ingress_controller_nginx_process_read_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 0
# HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests
# TYPE nginx_ingress_controller_nginx_process_requests_total counter
nginx_ingress_controller_nginx_process_requests_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 412
# HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge
nginx_ingress_controller_nginx_process_resident_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 2.01281536e+08
# HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge
nginx_ingress_controller_nginx_process_virtual_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.921888256e+09
# HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written
# TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter
nginx_ingress_controller_nginx_process_write_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 4096
# HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds
# TYPE nginx_ingress_controller_request_duration_seconds histogram
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_request_duration_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 0
nginx_ingress_controller_request_duration_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body)
# TYPE nginx_ingress_controller_request_size histogram
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="20"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="30"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="40"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="50"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="60"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="70"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="80"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="90"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_request_size_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 123
nginx_ingress_controller_request_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_requests The total number of client requests.
# TYPE nginx_ingress_controller_requests counter
nginx_ingress_controller_requests{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",status="404"} 1
# HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
# TYPE nginx_ingress_controller_response_duration_seconds histogram
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_response_duration_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 0
nginx_ingress_controller_response_duration_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body)
# TYPE nginx_ingress_controller_response_size histogram
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_response_size_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 274
nginx_ingress_controller_response_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire.\n An example to check if this certificate will expire in 10 days is: "nginx_ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))"
# TYPE nginx_ingress_controller_ssl_expire_time_seconds gauge
nginx_ingress_controller_ssl_expire_time_seconds{class="nginx",host="_",namespace="ingress-nginx"} 1.650776951e+09
# HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations
# TYPE nginx_ingress_controller_success counter
nginx_ingress_controller_success{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.14
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 37
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.9657472e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61924095054e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.58632448e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
And this is controller log.
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v0.45.0
Build: 7365e9eeb2f4961ef94e4ce5eb2b6e1bdb55ce5c
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.6
-------------------------------------------------------------------------------
I0424 05:47:31.877071 6 flags.go:208] "Watching for Ingress" class="nginx"
W0424 05:47:31.877142 6 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0424 05:47:31.877497 6 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0424 05:47:31.877646 6 main.go:241] "Creating API client" host="https://10.100.0.1:443"
I0424 05:47:31.884117 6 main.go:285] "Running in Kubernetes cluster" major="1" minor="19+" git="v1.19.8-eks-96780e" state="clean" commit="96780e1b30acbf0a52c38b6030d7853e575bcdf3" platform="linux/amd64"
I0424 05:47:32.244619 6 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0424 05:47:32.246124 6 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0424 05:47:32.251336 6 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0424 05:47:32.269713 6 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0424 05:47:32.294559 6 nginx.go:254] "Starting NGINX Ingress controller"
I0424 05:47:32.302767 6 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"aff039c8-6b4f-4d38-a4c3-ab40a79a7715", APIVersion:"v1", ResourceVersion:"953557", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0424 05:47:33.494923 6 nginx.go:296] "Starting NGINX process"
I0424 05:47:33.494968 6 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0424 05:47:33.495552 6 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0424 05:47:33.495729 6 controller.go:146] "Configuration changes detected, backend reload required"
I0424 05:47:33.501761 6 status.go:84] "New leader elected" identity="ingress-nginx-controller-6f5454cbfb-vc6hd"
I0424 05:47:33.569209 6 controller.go:163] "Backend successfully reloaded"
I0424 05:47:33.569280 6 controller.go:174] "Initial sync, sleeping for 1 second"
I0424 05:47:33.569334 6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5df4fbf9df-l8zst", UID:"8fd187a8-0149-4eed-89f5-1bdf89ab6075", APIVersion:"v1", ResourceVersion:"962870", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0424 05:48:34.984857 6 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0424 05:48:34.984875 6 status.go:84] "New leader elected" identity="ingress-nginx-controller-5df4fbf9df-l8zst"
@nwaistd did you install as per https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/ ?
@longwuyuan yes
@nwaistd so you installed as per docs and not only TCP metrics but other metrics that are supposed to be visualized in grafana, out of the box, are also not working ?
@longwuyuan yes
I installed with just these three commands.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/aws/deploy.yaml
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/
All targets are dropped. So can't import sources from prometheus
Can you come on slack and discuss this ?
@longwuyuan I opened a thread https://kubernetes.slack.com/archives/CANQGM8BA/p1619532483367100
Hi @longwuyuan,
I am also experiencing the same issue. I checked the slack thread but there was no progress on it.
I am using our own Prometheus instance and have added the additional scrape config (found here: https://github.com/kubernetes/ingress-nginx/blob/master/deploy/prometheus/prometheus.yaml) to prom and also set the pod scrape annotations for the controller pod(s). I am still only gathering a subset of the metrics that the provided grafana dashboard is showing:
curl -X GET 10.1.2.127:10254/metrics | wc -l
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12911 0 12911 0 0 504k 0 --:--:-- --:--:-- --:--:-- 504k
171
I am using the v0.47.0 version of the controller and v2.27.1 Prometheus server. The Nginx Controller was deployed using a helm chart. Any idea on how to get the additional metrics like upstream requests/responses?
Hi @nwaistd,
I did some digging and found a solution that works for us, I don't have a full understanding of everything so this may not be applicable for you or 100% correct but it seems the missing metrics are no longer present due to the commit made in the past as outlined in this issue: https://github.com/kubernetes/ingress-nginx/issues/3713. This commit made it so that ingress objects that do not specify a value in spec.rules.host field (which we are not doing currently in our development phase) will not have metrics counted in order to address valid requests vs DOS requests.
A workaround was implemented for this here in this pull request: https://github.com/kubernetes/ingress-nginx/pull/4139 and can be used by specifying --metrics-per-host=false
in the args when launching the controller.
@pchang388 Thank you so much! I'll test it and leave a comment soon.
I added host in ingress, and it worked.
...
spec:
rules:
- host: "<MY_HOST_NAME>"
http:
paths:
...
I can get various status metrics from localhost:10254/metrics
like this.
...
nginx_ingress_controller_response_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-78755949f6-mpn4t",host="<MY_HOST_NAME>",ingress="xxxxxx",method="POST",namespace="default",path="xxxxxx",service="xxxxxx",status="200"} 2
...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
@k8s-triage-robot: Closing this issue.
I deployed my ingress controller like this on AWS EKS
When I run
curl localhost:10254/metrics
in the ingress-controller-pod, it seems to show some part of all metrics. for example,So, most of dashboard are empty.
What am I missing?
please helpme
/triage support