unsafeSsl is being ignored in prometheus scaler for versions 2.13+

Report

We are using Prometheus Scaler to point to a sigv4 proxy service in order to integrate Keda with Amazon managed Prometheus (example : https://aws.amazon.com/blogs/mt/proactive-autoscaling-kubernetes-workloads-keda-metrics-ingested-into-aws-amp/) and everything was working fine so far (specifying unsafeSsl does work fine). However, starting Keda 2.13, this doesn't work anymore and we get the following error :

2024-06-07T13:54:30Z ERROR prometheus_scaler error executing prometheus query {"type": "ScaledObject", "namespace": "otel-gateway", "name": "uoc-hpa", "error": "Get \"https://\": tls: failed to verify certificate: x509: certificate signed by unknown authority"} github.com/kedacore/keda/v2/pkg/scalers.(prometheusScaler).GetMetricsAndActivity /workspace/pkg/scalers/prometheus_scaler.go:391 github.com/kedacore/keda/v2/pkg/scaling/cache.(ScalersCache).GetMetricsAndActivityForScaler /workspace/pkg/scaling/cache/scalers_cache.go:140 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics.func1 /workspace/pkg/scaling/scale_handler.go:526

When we describe the scaledobject:

Events: Type Reason Age From Message

Normal KEDAScalersStarted 14m keda-operator Started scalers watch Normal ScaledObjectReady 14m keda-operator ScaledObject is ready for scaling Warning KEDAScalerFailed 14m keda-operator Get "https://": tls: failed to verify certificate: x509: certificate signed by unknown authority Warning KEDAScalerFailed 13m keda-operator Get "https://": tls: failed to verify certificate: x509: certificate signed by unknown authority

Expected Behavior

When using 2.12 or 2.11 keda versions, unsafeSsl works successfully with prometheus scaler.

Actual Behavior

Upgraded Keda to 2.13, deployed prometheus scaledobject and we specified unsafeSsl , we get the following:

tls: failed to verify certificate: x509: certificate signed by unknown authority

Steps to Reproduce the Problem

Deploy Keda 2.13 version
Deploy Prometheus scaledobjet with unsafeSsl

Logs from KEDA operator

2024-06-07T13:54:15Z    DEBUG   scale_handler   Getting metrics from trigger    {"scaledObject.Namespace": "otel-gateway", "scaledObject.Name": "uoc-hpa", "trigger": "prometheusScaler", "metricName": "s0-prometheus", "metrics": [], "scalerError": "Get \"https://<awssigv4-URL>\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
2024-06-07T13:54:15Z    INFO    fallback        Suppressing error, falling back to fallback.replicas    {"scaledObject.Namespace": "otel-gateway", "scaledObject.Name": "uoc-hpa", "suppressedError": "Get \"https://<awssigv4-URL>\": tls: failed to verify certificate: x509: certificate signed by unknown authority", "fallback.replicas": 5}
2024-06-07T13:54:15Z    DEBUG   grpc_server     Providing metrics       {"scaledObjectName": "uoc-hpa", "scaledObjectNamespace": "otel-gateway", "metrics": "&ExternalMetricValueList{ListMeta:{   <nil>},Items:[]ExternalMetricValue{ExternalMetricValue{MetricName:s0-prometheus,MetricLabels:map[string]string{},Timestamp:2024-06-07 13:54:15.769598946 +0000 UTC m=+118.751492577,WindowSeconds:nil,Value:{{450000 -3} {<nil>}  DecimalSI},},},}"}
2024-06-07T13:54:30Z    ERROR   prometheus_scaler       error executing prometheus query        {"type": "ScaledObject", "namespace": "otel-gateway", "name": "uoc-hpa", "error": "Get \"https://<awssigv4-URL>\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
github.com/kedacore/keda/v2/pkg/scalers.(*prometheusScaler).GetMetricsAndActivity
        /workspace/pkg/scalers/prometheus_scaler.go:391
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler
        /workspace/pkg/scaling/cache/scalers_cache.go:130
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScaledObjectMetrics.func1
        /workspace/pkg/scaling/scale_handler.go:526

KEDA Version

2.13.1

Kubernetes Version

1.28

Platform

Amazon Web Services

Scaler Details

Prometheus

Anything else?

No response

kedacore / keda