search

krakend / krakend-otel

KrakenD component for OpenTelemetry
Apache License 2.0
8 stars 10 forks source link

Prometheus metrics being generated for opentelemetry skip_paths #25

Closed bjordan closed 2 months ago

bjordan commented 3 months ago

Environment info:

Describe the bug Prometheus metrics are being generated for opentelemetry skip_paths. If no skip_paths are defined and I hit /__health metrics are generated. If I explicitely add a list of skip_paths as below, the same is being observed. The requests produce the following, a status of 200 and route of 404 Not Found when requesting /metrics: http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="5"}

Your configuration file: I've removed all endpoint for simplicity, but running using the following config produces the same issue.

{
    "version": 3,
    "name": "gw-architecture",
    "port": 3000,
    "timeout": "10s",
    "endpoints": [],
    "extra_config": {
        "telemetry/logging": {
            "level": "DEBUG",
            "syslog": false,
            "stdout": true
        },
        "telemetry/opentelemetry": {
            "metric_reporting_period": 30,
            "skip_paths": [
                "/__health",
                "/__debug/",
                "/__echo/",
                "/__stats/"
            ],
            "exporters": {
                "prometheus": [
                    {
                        "disable_metrics": false,
                        "port": 9091,
                        "name": "gw_architecture",
                        "process_metrics": false,
                        "go_metrics": false
                    }
                ]
            },
            "layers": {
                "proxy": {
                    "disable_metrics": true
                }
            }
        },
        "router": {
            "disable_access_log": false,
            "logger_skip_paths": [
                "/__health"
            ]
        }
    },
    "debug_endpoint": true,
    "echo_endpoint": true
}

Commands used How did you start the software?

docker run --rm -it -p "3000:3000" -p "9091:9091" -v $PWD:/etc/krakend/ devopsfaith/krakend run -c krakend.json

Expected behavior I would not expect to see any metrics for skip_paths.

Logs

docker run --rm -it -p "3000:3000" -p "9091:9091" -v $PWD:/etc/krakend/ devopsfaith/krakend run -c krakend.json
Parsing configuration file: krakend.json
 2024/05/15 - 05:09:23.786 ▶ DEBUG [SERVICE: telemetry/logging] Improved logging started.
 2024/05/15 - 05:09:23.786 ▶ INFO Starting KrakenD v2.6.2
 2024/05/15 - 05:09:23.787 ▶ INFO Working directory is /etc/krakend
 2024/05/15 - 05:09:23.787 ▶ DEBUG [SERVICE: Gin] Debug enabled
 2024/05/15 - 05:09:23.787 ▶ INFO Starting the KrakenD instance
 2024/05/15 - 05:09:23.787 ▶ INFO [SERVICE: Gin] Listening on port: 3000
 2024/05/15 - 05:09:28.791 ▶ DEBUG [SERVICE: Telemetry] Registering usage stats for Cluster ID LJKR8A/p0p72i0Thk2SP9e07T9mngGs8ebHFN1IJkMw=

Additional context Metrics output

curl -v localhost:9091/metrics                                                                          

# HELP http_server_duration 

# TYPE http_server_duration histogram

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.01"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.02"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.05"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.075"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.1"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.125"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.15"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.175"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.2"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.25"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.3"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.35"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.5"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="0.75"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="1"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="1.5"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="2"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="3.5"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="5"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="10"} 5

http_server_duration_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="+Inf"} 5

http_server_duration_sum{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version=""} 0.000357677

http_server_duration_count{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version=""} 5

# HELP http_server_response_size 

# TYPE http_server_response_size histogram

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="128"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="256"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="512"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="1024"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="4096"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="8192"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="16384"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="32768"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="65536"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="262144"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="524288"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="1.048576e+06"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="4.194304e+06"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="1.6777216e+07"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="6.7108864e+07"} 5

http_server_response_size_bucket{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version="",le="+Inf"} 5

http_server_response_size_sum{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version=""} 455

http_server_response_size_count{http_request_method="GET",http_response_status_code="200",http_route="404 Not Found",otel_scope_name="io.krakend.krakend-otel",otel_scope_version=""} 5

# HELP otel_scope_info Instrumentation Scope metadata

# TYPE otel_scope_info gauge

otel_scope_info{otel_scope_name="io.krakend.krakend-otel",otel_scope_version=""} 1

# HELP target_info Target metadata

# TYPE target_info gauge

target_info{service_name="gw-architecture",service_version="2.6.2"} 1
paulsmir commented 3 months ago

Same here, 2.6.2 deployed with helm chart.

alombarte commented 2 months ago

@bjordan, @paulsmir I am closing this task as the the version v2.6.3 released on 2024-05-16 addressed this.

Let us know otherwise