search

Kong / kong-plugin-prometheus

Prometheus plugin for Kong - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo
Apache License 2.0
119 stars 57 forks source link

prometheus can not scrape kong metrics #37

Closed aeoluswing closed 5 years ago

aeoluswing commented 5 years ago

Environment : Docker Version: 18.03.1-ce Kong image : 1.0.0rc3 Grafana image : 5.4.2

I have exposed the kong metrics data by the kong-plugin-prometheus in Prometheus Exposition Format:

# curl -d - http://kong:8001/metrics
# HELP kong_bandwidth Total bandwidth in bytes consumed per service in Kong
# TYPE kong_bandwidth counter
kong_bandwidth{type="egress",service="attachment"} 120320
kong_bandwidth{type="egress",service="birthday"} 41875
kong_bandwidth{type="egress",service="buff"} 179299
kong_bandwidth{type="egress",service="mq"} 1113417
kong_bandwidth{type="egress",service="reqs"} 1335
kong_bandwidth{type="egress",service="ta"} 2109330
kong_bandwidth{type="egress",service="user"} 49883256
kong_bandwidth{type="ingress",service="attachment"} 1487485
kong_bandwidth{type="ingress",service="birthday"} 35550
kong_bandwidth{type="ingress",service="buff"} 436921
kong_bandwidth{type="ingress",service="mq"} 1659914
kong_bandwidth{type="ingress",service="reqs"} 4965
kong_bandwidth{type="ingress",service="ta"} 129358
kong_bandwidth{type="ingress",service="user"} 3735345
# HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable
# TYPE kong_datastore_reachable gauge
kong_datastore_reachable 1
# HELP kong_http_status HTTP status codes per service in Kong
# TYPE kong_http_status counter
kong_http_status{code="101",service="mq"} 3059
kong_http_status{code="200",service="attachment"} 164
kong_http_status{code="200",service="birthday"} 127
kong_http_status{code="200",service="buff"} 428
kong_http_status{code="200",service="mq"} 56
kong_http_status{code="200",service="reqs"} 4
kong_http_status{code="200",service="ta"} 430
kong_http_status{code="200",service="user"} 11437
kong_http_status{code="302",service="ta"} 1
kong_http_status{code="302",service="user"} 419
kong_http_status{code="304",service="user"} 6
kong_http_status{code="404",service="user"} 8
kong_http_status{code="499",service="birthday"} 1
kong_http_status{code="499",service="mq"} 1
kong_http_status{code="499",service="user"} 1
kong_http_status{code="502",service="mq"} 6
# HELP kong_latency Latency added by Kong, total request time and upstream latency for each service in Kong
# TYPE kong_latency histogram
kong_latency_bucket{type="kong",service="attachment",le="00001.0"} 127
kong_latency_bucket{type="kong",service="attachment",le="00002.0"} 151
kong_latency_bucket{type="kong",service="attachment",le="00005.0"} 154
kong_latency_bucket{type="kong",service="attachment",le="00007.0"} 159
kong_latency_bucket{type="kong",service="attachment",le="00010.0"} 162
kong_latency_bucket{type="kong",service="attachment",le="00015.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00020.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00025.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00030.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00040.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00050.0"} 163
kong_latency_bucket{type="kong",service="attachment",le="00060.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00070.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00080.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00090.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00100.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00200.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00300.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00400.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="00500.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="01000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="02000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="05000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="10000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="30000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="60000.0"} 164
kong_latency_bucket{type="kong",service="attachment",le="+Inf"} 164
kong_latency_bucket{type="kong",service="birthday",le="00001.0"} 99
kong_latency_bucket{type="kong",service="birthday",le="00002.0"} 119
kong_latency_bucket{type="kong",service="birthday",le="00005.0"} 122
kong_latency_bucket{type="kong",service="birthday",le="00007.0"} 126
kong_latency_bucket{type="kong",service="birthday",le="00010.0"} 127
kong_latency_bucket{type="kong",service="birthday",le="00015.0"} 127
kong_latency_bucket{type="kong",service="birthday",le="00020.0"} 127
kong_latency_bucket{type="kong",service="birthday",le="00025.0"} 127
kong_latency_bucket{type="kong",service="birthday",le="00030.0"} 127
kong_latency_bucket{type="kong",service="birthday",le="00040.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00050.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00060.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00070.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00080.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00090.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00100.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00200.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00300.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00400.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="00500.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="01000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="02000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="05000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="10000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="30000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="60000.0"} 128
kong_latency_bucket{type="kong",service="birthday",le="+Inf"} 128
kong_latency_bucket{type="kong",service="buff",le="00001.0"} 389
kong_latency_bucket{type="kong",service="buff",le="00002.0"} 418
kong_latency_bucket{type="kong",service="buff",le="00005.0"} 420
kong_latency_bucket{type="kong",service="buff",le="00007.0"} 424
kong_latency_bucket{type="kong",service="buff",le="00010.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00015.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00020.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00025.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00030.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00040.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00050.0"} 427
kong_latency_bucket{type="kong",service="buff",le="00060.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00070.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00080.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00090.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00100.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00200.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00300.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00400.0"} 428
kong_latency_bucket{type="kong",service="buff",le="00500.0"} 428
kong_latency_bucket{type="kong",service="buff",le="01000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="02000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="05000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="10000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="30000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="60000.0"} 428
kong_latency_bucket{type="kong",service="buff",le="+Inf"} 428
kong_latency_bucket{type="kong",service="mq",le="00001.0"} 2816
kong_latency_bucket{type="kong",service="mq",le="00002.0"} 2841
kong_latency_bucket{type="kong",service="mq",le="00005.0"} 2849
kong_latency_bucket{type="kong",service="mq",le="00007.0"} 2858
kong_latency_bucket{type="kong",service="mq",le="00010.0"} 2874
kong_latency_bucket{type="kong",service="mq",le="00015.0"} 2876
kong_latency_bucket{type="kong",service="mq",le="00020.0"} 2878
kong_latency_bucket{type="kong",service="mq",le="00025.0"} 2878
kong_latency_bucket{type="kong",service="mq",le="00030.0"} 2879
kong_latency_bucket{type="kong",service="mq",le="00040.0"} 2882
kong_latency_bucket{type="kong",service="mq",le="00050.0"} 2882
kong_latency_bucket{type="kong",service="mq",le="00060.0"} 2882
kong_latency_bucket{type="kong",service="mq",le="00070.0"} 2883
kong_latency_bucket{type="kong",service="mq",le="00080.0"} 2883
kong_latency_bucket{type="kong",service="mq",le="00090.0"} 2883
kong_latency_bucket{type="kong",service="mq",le="00100.0"} 2883
kong_latency_bucket{type="kong",service="mq",le="00200.0"} 2884
kong_latency_bucket{type="kong",service="mq",le="00300.0"} 2884
kong_latency_bucket{type="kong",service="mq",le="00400.0"} 2885
kong_latency_bucket{type="kong",service="mq",le="00500.0"} 2885
kong_latency_bucket{type="kong",service="mq",le="01000.0"} 2888
kong_latency_bucket{type="kong",service="mq",le="02000.0"} 2898
kong_latency_bucket{type="kong",service="mq",le="05000.0"} 2960
kong_latency_bucket{type="kong",service="mq",le="10000.0"} 3001
kong_latency_bucket{type="kong",service="mq",le="30000.0"} 3071
kong_latency_bucket{type="kong",service="mq",le="60000.0"} 3122
kong_latency_bucket{type="kong",service="mq",le="+Inf"} 3123
kong_latency_bucket{type="kong",service="reqs",le="00001.0"} 2
kong_latency_bucket{type="kong",service="reqs",le="00002.0"} 2
kong_latency_bucket{type="kong",service="reqs",le="00005.0"} 2
kong_latency_bucket{type="kong",service="reqs",le="00007.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00010.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00015.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00020.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00025.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00030.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00040.0"} 3
kong_latency_bucket{type="kong",service="reqs",le="00050.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00060.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00070.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00080.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00090.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00100.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00200.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00300.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00400.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="00500.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="01000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="02000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="05000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="10000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="30000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="60000.0"} 4
kong_latency_bucket{type="kong",service="reqs",le="+Inf"} 4
kong_latency_bucket{type="kong",service="ta",le="00001.0"} 331
kong_latency_bucket{type="kong",service="ta",le="00002.0"} 402
kong_latency_bucket{type="kong",service="ta",le="00005.0"} 416
kong_latency_bucket{type="kong",service="ta",le="00007.0"} 421
kong_latency_bucket{type="kong",service="ta",le="00010.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00015.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00020.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00025.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00030.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00040.0"} 430
kong_latency_bucket{type="kong",service="ta",le="00050.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00060.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00070.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00080.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00090.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00100.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00200.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00300.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00400.0"} 431
kong_latency_bucket{type="kong",service="ta",le="00500.0"} 431
kong_latency_bucket{type="kong",service="ta",le="01000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="02000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="05000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="10000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="30000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="60000.0"} 431
kong_latency_bucket{type="kong",service="ta",le="+Inf"} 431
kong_latency_bucket{type="kong",service="user",le="00001.0"} 11439
kong_latency_bucket{type="kong",service="user",le="00002.0"} 11680
kong_latency_bucket{type="kong",service="user",le="00005.0"} 11856
kong_latency_bucket{type="kong",service="user",le="00007.0"} 11869
kong_latency_bucket{type="kong",service="user",le="00010.0"} 11869
kong_latency_bucket{type="kong",service="user",le="00015.0"} 11869
kong_latency_bucket{type="kong",service="user",le="00020.0"} 11870
kong_latency_bucket{type="kong",service="user",le="00025.0"} 11870
kong_latency_bucket{type="kong",service="user",le="00030.0"} 11870
kong_latency_bucket{type="kong",service="user",le="00040.0"} 11870
kong_latency_bucket{type="kong",service="user",le="00050.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00060.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00070.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00080.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00090.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00100.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00200.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00300.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00400.0"} 11871
kong_latency_bucket{type="kong",service="user",le="00500.0"} 11871
kong_latency_bucket{type="kong",service="user",le="01000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="02000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="05000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="10000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="30000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="60000.0"} 11871
kong_latency_bucket{type="kong",service="user",le="+Inf"} 11871
kong_latency_bucket{type="request",service="attachment",le="00007.0"} 10
kong_latency_bucket{type="request",service="attachment",le="00010.0"} 53
kong_latency_bucket{type="request",service="attachment",le="00015.0"} 77
kong_latency_bucket{type="request",service="attachment",le="00020.0"} 83
kong_latency_bucket{type="request",service="attachment",le="00025.0"} 85
kong_latency_bucket{type="request",service="attachment",le="00030.0"} 85
kong_latency_bucket{type="request",service="attachment",le="00040.0"} 101
kong_latency_bucket{type="request",service="attachment",le="00050.0"} 121
kong_latency_bucket{type="request",service="attachment",le="00060.0"} 132
kong_latency_bucket{type="request",service="attachment",le="00070.0"} 140
kong_latency_bucket{type="request",service="attachment",le="00080.0"} 143
kong_latency_bucket{type="request",service="attachment",le="00090.0"} 144
kong_latency_bucket{type="request",service="attachment",le="00100.0"} 145
kong_latency_bucket{type="request",service="attachment",le="00200.0"} 151
kong_latency_bucket{type="request",service="attachment",le="00300.0"} 152
kong_latency_bucket{type="request",service="attachment",le="00400.0"} 154
kong_latency_bucket{type="request",service="attachment",le="00500.0"} 154
kong_latency_bucket{type="request",service="attachment",le="01000.0"} 162
kong_latency_bucket{type="request",service="attachment",le="02000.0"} 163
kong_latency_bucket{type="request",service="attachment",le="05000.0"} 164
kong_latency_bucket{type="request",service="attachment",le="10000.0"} 164
kong_latency_bucket{type="request",service="attachment",le="30000.0"} 164
kong_latency_bucket{type="request",service="attachment",le="60000.0"} 164
kong_latency_bucket{type="request",service="attachment",le="+Inf"} 164
kong_latency_bucket{type="request",service="birthday",le="00010.0"} 4
kong_latency_bucket{type="request",service="birthday",le="00015.0"} 10
kong_latency_bucket{type="request",service="birthday",le="00020.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00025.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00030.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00040.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00050.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00060.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00070.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00080.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00090.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00100.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00200.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00300.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00400.0"} 13
kong_latency_bucket{type="request",service="birthday",le="00500.0"} 14
kong_latency_bucket{type="request",service="birthday",le="01000.0"} 14
kong_latency_bucket{type="request",service="birthday",le="02000.0"} 16
kong_latency_bucket{type="request",service="birthday",le="05000.0"} 128
kong_latency_bucket{type="request",service="birthday",le="10000.0"} 128
kong_latency_bucket{type="request",service="birthday",le="30000.0"} 128
kong_latency_bucket{type="request",service="birthday",le="60000.0"} 128
kong_latency_bucket{type="request",service="birthday",le="+Inf"} 128
kong_latency_bucket{type="request",service="buff",le="00005.0"} 30
kong_latency_bucket{type="request",service="buff",le="00007.0"} 198
kong_latency_bucket{type="request",service="buff",le="00010.0"} 281
kong_latency_bucket{type="request",service="buff",le="00015.0"} 313
kong_latency_bucket{type="request",service="buff",le="00020.0"} 322
kong_latency_bucket{type="request",service="buff",le="00025.0"} 338
kong_latency_bucket{type="request",service="buff",le="00030.0"} 359
kong_latency_bucket{type="request",service="buff",le="00040.0"} 373
kong_latency_bucket{type="request",service="buff",le="00050.0"} 380
kong_latency_bucket{type="request",service="buff",le="00060.0"} 385
kong_latency_bucket{type="request",service="buff",le="00070.0"} 390
kong_latency_bucket{type="request",service="buff",le="00080.0"} 398
kong_latency_bucket{type="request",service="buff",le="00090.0"} 399
kong_latency_bucket{type="request",service="buff",le="00100.0"} 405
kong_latency_bucket{type="request",service="buff",le="00200.0"} 412
kong_latency_bucket{type="request",service="buff",le="00300.0"} 416
kong_latency_bucket{type="request",service="buff",le="00400.0"} 417
kong_latency_bucket{type="request",service="buff",le="00500.0"} 420
kong_latency_bucket{type="request",service="buff",le="01000.0"} 426
kong_latency_bucket{type="request",service="buff",le="02000.0"} 426
kong_latency_bucket{type="request",service="buff",le="05000.0"} 427
kong_latency_bucket{type="request",service="buff",le="10000.0"} 427
kong_latency_bucket{type="request",service="buff",le="30000.0"} 428
kong_latency_bucket{type="request",service="buff",le="60000.0"} 428
kong_latency_bucket{type="request",service="buff",le="+Inf"} 428
kong_latency_bucket{type="request",service="mq",le="00001.0"} 2
kong_latency_bucket{type="request",service="mq",le="00002.0"} 5
kong_latency_bucket{type="request",service="mq",le="00005.0"} 56
kong_latency_bucket{type="request",service="mq",le="00007.0"} 59
kong_latency_bucket{type="request",service="mq",le="00010.0"} 60
kong_latency_bucket{type="request",service="mq",le="00015.0"} 63
kong_latency_bucket{type="request",service="mq",le="00020.0"} 64
kong_latency_bucket{type="request",service="mq",le="00025.0"} 65
kong_latency_bucket{type="request",service="mq",le="00030.0"} 66
kong_latency_bucket{type="request",service="mq",le="00040.0"} 68
kong_latency_bucket{type="request",service="mq",le="00050.0"} 68
kong_latency_bucket{type="request",service="mq",le="00060.0"} 68
kong_latency_bucket{type="request",service="mq",le="00070.0"} 68
kong_latency_bucket{type="request",service="mq",le="00080.0"} 68
kong_latency_bucket{type="request",service="mq",le="00090.0"} 68
kong_latency_bucket{type="request",service="mq",le="00100.0"} 69
kong_latency_bucket{type="request",service="mq",le="00200.0"} 70
kong_latency_bucket{type="request",service="mq",le="00300.0"} 70
kong_latency_bucket{type="request",service="mq",le="00400.0"} 71
kong_latency_bucket{type="request",service="mq",le="00500.0"} 71
kong_latency_bucket{type="request",service="mq",le="01000.0"} 75
kong_latency_bucket{type="request",service="mq",le="02000.0"} 85
kong_latency_bucket{type="request",service="mq",le="05000.0"} 148
kong_latency_bucket{type="request",service="mq",le="10000.0"} 189
kong_latency_bucket{type="request",service="mq",le="30000.0"} 259
kong_latency_bucket{type="request",service="mq",le="60000.0"} 839
kong_latency_bucket{type="request",service="mq",le="+Inf"} 3123
kong_latency_bucket{type="request",service="reqs",le="00070.0"} 1
kong_latency_bucket{type="request",service="reqs",le="00080.0"} 1
kong_latency_bucket{type="request",service="reqs",le="00090.0"} 1
kong_latency_bucket{type="request",service="reqs",le="00100.0"} 1
kong_latency_bucket{type="request",service="reqs",le="00200.0"} 2
kong_latency_bucket{type="request",service="reqs",le="00300.0"} 2
kong_latency_bucket{type="request",service="reqs",le="00400.0"} 3
kong_latency_bucket{type="request",service="reqs",le="00500.0"} 3
kong_latency_bucket{type="request",service="reqs",le="01000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="02000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="05000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="10000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="30000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="60000.0"} 4
kong_latency_bucket{type="request",service="reqs",le="+Inf"} 4
kong_latency_bucket{type="request",service="ta",le="00007.0"} 21
kong_latency_bucket{type="request",service="ta",le="00010.0"} 81
kong_latency_bucket{type="request",service="ta",le="00015.0"} 123
kong_latency_bucket{type="request",service="ta",le="00020.0"} 135
kong_latency_bucket{type="request",service="ta",le="00025.0"} 147
kong_latency_bucket{type="request",service="ta",le="00030.0"} 169
kong_latency_bucket{type="request",service="ta",le="00040.0"} 219
kong_latency_bucket{type="request",service="ta",le="00050.0"} 232
kong_latency_bucket{type="request",service="ta",le="00060.0"} 235
kong_latency_bucket{type="request",service="ta",le="00070.0"} 245
kong_latency_bucket{type="request",service="ta",le="00080.0"} 267
kong_latency_bucket{type="request",service="ta",le="00090.0"} 294
kong_latency_bucket{type="request",service="ta",le="00100.0"} 321
kong_latency_bucket{type="request",service="ta",le="00200.0"} 423
kong_latency_bucket{type="request",service="ta",le="00300.0"} 424
kong_latency_bucket{type="request",service="ta",le="00400.0"} 427
kong_latency_bucket{type="request",service="ta",le="00500.0"} 429
kong_latency_bucket{type="request",service="ta",le="01000.0"} 431
kong_latency_bucket{type="request",service="ta",le="02000.0"} 431
kong_latency_bucket{type="request",service="ta",le="05000.0"} 431
kong_latency_bucket{type="request",service="ta",le="10000.0"} 431
kong_latency_bucket{type="request",service="ta",le="30000.0"} 431
kong_latency_bucket{type="request",service="ta",le="60000.0"} 431
kong_latency_bucket{type="request",service="ta",le="+Inf"} 431
kong_latency_bucket{type="request",service="user",le="00001.0"} 576
kong_latency_bucket{type="request",service="user",le="00002.0"} 5549
kong_latency_bucket{type="request",service="user",le="00005.0"} 10890
kong_latency_bucket{type="request",service="user",le="00007.0"} 11115
kong_latency_bucket{type="request",service="user",le="00010.0"} 11157
kong_latency_bucket{type="request",service="user",le="00015.0"} 11165
kong_latency_bucket{type="request",service="user",le="00020.0"} 11269
kong_latency_bucket{type="request",service="user",le="00025.0"} 11329
kong_latency_bucket{type="request",service="user",le="00030.0"} 11343
kong_latency_bucket{type="request",service="user",le="00040.0"} 11344
kong_latency_bucket{type="request",service="user",le="00050.0"} 11344
kong_latency_bucket{type="request",service="user",le="00060.0"} 11345
kong_latency_bucket{type="request",service="user",le="00070.0"} 11353
kong_latency_bucket{type="request",service="user",le="00080.0"} 11358
kong_latency_bucket{type="request",service="user",le="00090.0"} 11364
kong_latency_bucket{type="request",service="user",le="00100.0"} 11380
kong_latency_bucket{type="request",service="user",le="00200.0"} 11853
kong_latency_bucket{type="request",service="user",le="00300.0"} 11865
kong_latency_bucket{type="request",service="user",le="00400.0"} 11870
kong_latency_bucket{type="request",service="user",le="00500.0"} 11870
kong_latency_bucket{type="request",service="user",le="01000.0"} 11871
kong_latency_bucket{type="request",service="user",le="02000.0"} 11871
kong_latency_bucket{type="request",service="user",le="05000.0"} 11871
kong_latency_bucket{type="request",service="user",le="10000.0"} 11871
kong_latency_bucket{type="request",service="user",le="30000.0"} 11871
kong_latency_bucket{type="request",service="user",le="60000.0"} 11871
kong_latency_bucket{type="request",service="user",le="+Inf"} 11871
kong_latency_bucket{type="upstream",service="attachment",le="00007.0"} 22
kong_latency_bucket{type="upstream",service="attachment",le="00010.0"} 72
kong_latency_bucket{type="upstream",service="attachment",le="00015.0"} 82
kong_latency_bucket{type="upstream",service="attachment",le="00020.0"} 84
kong_latency_bucket{type="upstream",service="attachment",le="00025.0"} 85
kong_latency_bucket{type="upstream",service="attachment",le="00030.0"} 85
kong_latency_bucket{type="upstream",service="attachment",le="00040.0"} 104
kong_latency_bucket{type="upstream",service="attachment",le="00050.0"} 123
kong_latency_bucket{type="upstream",service="attachment",le="00060.0"} 133
kong_latency_bucket{type="upstream",service="attachment",le="00070.0"} 140
kong_latency_bucket{type="upstream",service="attachment",le="00080.0"} 145
kong_latency_bucket{type="upstream",service="attachment",le="00090.0"} 145
kong_latency_bucket{type="upstream",service="attachment",le="00100.0"} 146
kong_latency_bucket{type="upstream",service="attachment",le="00200.0"} 151
kong_latency_bucket{type="upstream",service="attachment",le="00300.0"} 152
kong_latency_bucket{type="upstream",service="attachment",le="00400.0"} 154
kong_latency_bucket{type="upstream",service="attachment",le="00500.0"} 154
kong_latency_bucket{type="upstream",service="attachment",le="01000.0"} 162
kong_latency_bucket{type="upstream",service="attachment",le="02000.0"} 163
kong_latency_bucket{type="upstream",service="attachment",le="05000.0"} 164
kong_latency_bucket{type="upstream",service="attachment",le="10000.0"} 164
kong_latency_bucket{type="upstream",service="attachment",le="30000.0"} 164
kong_latency_bucket{type="upstream",service="attachment",le="60000.0"} 164
kong_latency_bucket{type="upstream",service="attachment",le="+Inf"} 164
kong_latency_bucket{type="upstream",service="birthday",le="00007.0"} 1
kong_latency_bucket{type="upstream",service="birthday",le="00010.0"} 5
kong_latency_bucket{type="upstream",service="birthday",le="00015.0"} 11
kong_latency_bucket{type="upstream",service="birthday",le="00020.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00025.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00030.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00040.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00050.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00060.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00070.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00080.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00090.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00100.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00200.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00300.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00400.0"} 13
kong_latency_bucket{type="upstream",service="birthday",le="00500.0"} 14
kong_latency_bucket{type="upstream",service="birthday",le="01000.0"} 14
kong_latency_bucket{type="upstream",service="birthday",le="02000.0"} 15
kong_latency_bucket{type="upstream",service="birthday",le="05000.0"} 127
kong_latency_bucket{type="upstream",service="birthday",le="10000.0"} 127
kong_latency_bucket{type="upstream",service="birthday",le="30000.0"} 127
kong_latency_bucket{type="upstream",service="birthday",le="60000.0"} 127
kong_latency_bucket{type="upstream",service="birthday",le="+Inf"} 127
kong_latency_bucket{type="upstream",service="buff",le="00005.0"} 88
kong_latency_bucket{type="upstream",service="buff",le="00007.0"} 225
kong_latency_bucket{type="upstream",service="buff",le="00010.0"} 288
kong_latency_bucket{type="upstream",service="buff",le="00015.0"} 317
kong_latency_bucket{type="upstream",service="buff",le="00020.0"} 324
kong_latency_bucket{type="upstream",service="buff",le="00025.0"} 343
kong_latency_bucket{type="upstream",service="buff",le="00030.0"} 361
kong_latency_bucket{type="upstream",service="buff",le="00040.0"} 374
kong_latency_bucket{type="upstream",service="buff",le="00050.0"} 381
kong_latency_bucket{type="upstream",service="buff",le="00060.0"} 386
kong_latency_bucket{type="upstream",service="buff",le="00070.0"} 391
kong_latency_bucket{type="upstream",service="buff",le="00080.0"} 399
kong_latency_bucket{type="upstream",service="buff",le="00090.0"} 399
kong_latency_bucket{type="upstream",service="buff",le="00100.0"} 405
kong_latency_bucket{type="upstream",service="buff",le="00200.0"} 412
kong_latency_bucket{type="upstream",service="buff",le="00300.0"} 416
kong_latency_bucket{type="upstream",service="buff",le="00400.0"} 418
kong_latency_bucket{type="upstream",service="buff",le="00500.0"} 420
kong_latency_bucket{type="upstream",service="buff",le="01000.0"} 426
kong_latency_bucket{type="upstream",service="buff",le="02000.0"} 426
kong_latency_bucket{type="upstream",service="buff",le="05000.0"} 427
kong_latency_bucket{type="upstream",service="buff",le="10000.0"} 427
kong_latency_bucket{type="upstream",service="buff",le="30000.0"} 428
kong_latency_bucket{type="upstream",service="buff",le="60000.0"} 428
kong_latency_bucket{type="upstream",service="buff",le="+Inf"} 428
kong_latency_bucket{type="upstream",service="mq",le="00001.0"} 36
kong_latency_bucket{type="upstream",service="mq",le="00002.0"} 1269
kong_latency_bucket{type="upstream",service="mq",le="00005.0"} 3031
kong_latency_bucket{type="upstream",service="mq",le="00007.0"} 3092
kong_latency_bucket{type="upstream",service="mq",le="00010.0"} 3109
kong_latency_bucket{type="upstream",service="mq",le="00015.0"} 3115
kong_latency_bucket{type="upstream",service="mq",le="00020.0"} 3117
kong_latency_bucket{type="upstream",service="mq",le="00025.0"} 3118
kong_latency_bucket{type="upstream",service="mq",le="00030.0"} 3118
kong_latency_bucket{type="upstream",service="mq",le="00040.0"} 3118
kong_latency_bucket{type="upstream",service="mq",le="00050.0"} 3118
kong_latency_bucket{type="upstream",service="mq",le="00060.0"} 3119
kong_latency_bucket{type="upstream",service="mq",le="00070.0"} 3119
kong_latency_bucket{type="upstream",service="mq",le="00080.0"} 3119
kong_latency_bucket{type="upstream",service="mq",le="00090.0"} 3119
kong_latency_bucket{type="upstream",service="mq",le="00100.0"} 3119
kong_latency_bucket{type="upstream",service="mq",le="00200.0"} 3120
kong_latency_bucket{type="upstream",service="mq",le="00300.0"} 3120
kong_latency_bucket{type="upstream",service="mq",le="00400.0"} 3120
kong_latency_bucket{type="upstream",service="mq",le="00500.0"} 3120
kong_latency_bucket{type="upstream",service="mq",le="01000.0"} 3121
kong_latency_bucket{type="upstream",service="mq",le="02000.0"} 3121
kong_latency_bucket{type="upstream",service="mq",le="05000.0"} 3122
kong_latency_bucket{type="upstream",service="mq",le="10000.0"} 3122
kong_latency_bucket{type="upstream",service="mq",le="30000.0"} 3122
kong_latency_bucket{type="upstream",service="mq",le="60000.0"} 3122
kong_latency_bucket{type="upstream",service="mq",le="+Inf"} 3122
kong_latency_bucket{type="upstream",service="reqs",le="00070.0"} 1
kong_latency_bucket{type="upstream",service="reqs",le="00080.0"} 1
kong_latency_bucket{type="upstream",service="reqs",le="00090.0"} 1
kong_latency_bucket{type="upstream",service="reqs",le="00100.0"} 1
kong_latency_bucket{type="upstream",service="reqs",le="00200.0"} 2
kong_latency_bucket{type="upstream",service="reqs",le="00300.0"} 2
kong_latency_bucket{type="upstream",service="reqs",le="00400.0"} 3
kong_latency_bucket{type="upstream",service="reqs",le="00500.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="01000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="02000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="05000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="10000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="30000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="60000.0"} 4
kong_latency_bucket{type="upstream",service="reqs",le="+Inf"} 4
kong_latency_bucket{type="upstream",service="ta",le="00005.0"} 2
kong_latency_bucket{type="upstream",service="ta",le="00007.0"} 37
kong_latency_bucket{type="upstream",service="ta",le="00010.0"} 97
kong_latency_bucket{type="upstream",service="ta",le="00015.0"} 127
kong_latency_bucket{type="upstream",service="ta",le="00020.0"} 137
kong_latency_bucket{type="upstream",service="ta",le="00025.0"} 148
kong_latency_bucket{type="upstream",service="ta",le="00030.0"} 174
kong_latency_bucket{type="upstream",service="ta",le="00040.0"} 223
kong_latency_bucket{type="upstream",service="ta",le="00050.0"} 232
kong_latency_bucket{type="upstream",service="ta",le="00060.0"} 235
kong_latency_bucket{type="upstream",service="ta",le="00070.0"} 249
kong_latency_bucket{type="upstream",service="ta",le="00080.0"} 268
kong_latency_bucket{type="upstream",service="ta",le="00090.0"} 294
kong_latency_bucket{type="upstream",service="ta",le="00100.0"} 323
kong_latency_bucket{type="upstream",service="ta",le="00200.0"} 423
kong_latency_bucket{type="upstream",service="ta",le="00300.0"} 424
kong_latency_bucket{type="upstream",service="ta",le="00400.0"} 427
kong_latency_bucket{type="upstream",service="ta",le="00500.0"} 430
kong_latency_bucket{type="upstream",service="ta",le="01000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="02000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="05000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="10000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="30000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="60000.0"} 431
kong_latency_bucket{type="upstream",service="ta",le="+Inf"} 431
kong_latency_bucket{type="upstream",service="user",le="00001.0"} 1307
kong_latency_bucket{type="upstream",service="user",le="00002.0"} 7100
kong_latency_bucket{type="upstream",service="user",le="00005.0"} 10991
kong_latency_bucket{type="upstream",service="user",le="00007.0"} 11138
kong_latency_bucket{type="upstream",service="user",le="00010.0"} 11160
kong_latency_bucket{type="upstream",service="user",le="00015.0"} 11204
kong_latency_bucket{type="upstream",service="user",le="00020.0"} 11328
kong_latency_bucket{type="upstream",service="user",le="00025.0"} 11343
kong_latency_bucket{type="upstream",service="user",le="00030.0"} 11343
kong_latency_bucket{type="upstream",service="user",le="00040.0"} 11343
kong_latency_bucket{type="upstream",service="user",le="00050.0"} 11343
kong_latency_bucket{type="upstream",service="user",le="00060.0"} 11344
kong_latency_bucket{type="upstream",service="user",le="00070.0"} 11352
kong_latency_bucket{type="upstream",service="user",le="00080.0"} 11357
kong_latency_bucket{type="upstream",service="user",le="00090.0"} 11363
kong_latency_bucket{type="upstream",service="user",le="00100.0"} 11384
kong_latency_bucket{type="upstream",service="user",le="00200.0"} 11852
kong_latency_bucket{type="upstream",service="user",le="00300.0"} 11864
kong_latency_bucket{type="upstream",service="user",le="00400.0"} 11869
kong_latency_bucket{type="upstream",service="user",le="00500.0"} 11869
kong_latency_bucket{type="upstream",service="user",le="01000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="02000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="05000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="10000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="30000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="60000.0"} 11870
kong_latency_bucket{type="upstream",service="user",le="+Inf"} 11870
kong_latency_count{type="kong",service="attachment"} 164
kong_latency_count{type="kong",service="birthday"} 128
kong_latency_count{type="kong",service="buff"} 428
kong_latency_count{type="kong",service="mq"} 3123
kong_latency_count{type="kong",service="reqs"} 4
kong_latency_count{type="kong",service="ta"} 431
kong_latency_count{type="kong",service="user"} 11871
kong_latency_count{type="request",service="attachment"} 164
kong_latency_count{type="request",service="birthday"} 128
kong_latency_count{type="request",service="buff"} 428
kong_latency_count{type="request",service="mq"} 3123
kong_latency_count{type="request",service="reqs"} 4
kong_latency_count{type="request",service="ta"} 431
kong_latency_count{type="request",service="user"} 11871
kong_latency_count{type="upstream",service="attachment"} 164
kong_latency_count{type="upstream",service="birthday"} 127
kong_latency_count{type="upstream",service="buff"} 428
kong_latency_count{type="upstream",service="mq"} 3122
kong_latency_count{type="upstream",service="reqs"} 4
kong_latency_count{type="upstream",service="ta"} 431
kong_latency_count{type="upstream",service="user"} 11870
kong_latency_sum{type="kong",service="attachment"} 272
kong_latency_sum{type="kong",service="birthday"} 193
kong_latency_sum{type="kong",service="buff"} 380
kong_latency_sum{type="kong",service="mq"} 4278395
kong_latency_sum{type="kong",service="reqs"} 56
kong_latency_sum{type="kong",service="ta"} 575
kong_latency_sum{type="kong",service="user"} 3807
kong_latency_sum{type="request",service="attachment"} 14619
kong_latency_sum{type="request",service="birthday"} 327223
kong_latency_sum{type="request",service="buff"} 30901
kong_latency_sum{type="request",service="mq"} 173336147
kong_latency_sum{type="request",service="reqs"} 1114
kong_latency_sum{type="request",service="ta"} 28518
kong_latency_sum{type="request",service="user"} 95015
kong_latency_sum{type="upstream",service="attachment"} 14336
kong_latency_sum{type="upstream",service="birthday"} 325431
kong_latency_sum{type="upstream",service="buff"} 30500
kong_latency_sum{type="upstream",service="mq"} 11854
kong_latency_sum{type="upstream",service="reqs"} 1054
kong_latency_sum{type="upstream",service="ta"} 27897
kong_latency_sum{type="upstream",service="user"} 90499
# HELP kong_nginx_http_current_connections Number of HTTP connections
# TYPE kong_nginx_http_current_connections gauge
kong_nginx_http_current_connections{state="accepted"} 182786
kong_nginx_http_current_connections{state="active"} 8
kong_nginx_http_current_connections{state="handled"} 182786
kong_nginx_http_current_connections{state="reading"} 0
kong_nginx_http_current_connections{state="total"} 13467183
kong_nginx_http_current_connections{state="waiting"} 4
kong_nginx_http_current_connections{state="writing"} 4
# HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors
# TYPE kong_nginx_metric_errors_total counter
kong_nginx_metric_errors_total 0
{"message":"An unexpected error occurred"}

The issue is after scraped the /metrics endpoint just as issue #31 said,but prometheus got an 'valid start token' error:

prometheus_target_error

Kong grafana dashboard gains the metrics from DS_LOCAL datasource which i have changed to the prometheus datastore which i have mentioned before.

Here is the configuration of prometheus:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090','kong:8001']

I have tried service discovery and static config to scrape metrics but have not taken effect.

I'm not sure whether it's a prometheus plugin issue,but other metrics which exposed by exporters could be scraped by prometheus correctly.

Is there anything wrong i have tried?

hbagdi commented 5 years ago

This is likely due to the last line of the response returned for GET /metrics by Kong:

{"message":"An unexpected error occurred"}

This was a bug in rc3 but was fixed in the final release. Please use the 1.0.0 docker image.

Please close the issue if the problem is fixed with 1.0.0 for you. Thanks.

aeoluswing commented 5 years ago

prometheus_taget_success

@hbagdi After updating kong image version to 1.0.0,everything works well.As you said, the cause of the problem is the last line of the response in rc3 version.Great appreciation for your suggestion.