tomkerkhove / promitor

Bringing Azure Monitor metrics where you need them.
https://promitor.io
MIT License
249 stars 91 forks source link

Unable to scrape metrics from Azure monitor #2021

Closed tomkerkhove closed 2 years ago

tomkerkhove commented 2 years ago

Discussed in https://github.com/tomkerkhove/promitor/discussions/2011

Originally posted by **anoop2503** April 21, 2022 Hi, I could manage to set up the promitor application in kubernetes environment. The application is up and running without any issues. However, it does not scrape metrics from Azure monitor. **Here is the configuration for setting up the promitor scraper agent (without resourceDiscovery).** ``` --- apiVersion: v1 kind: ConfigMap metadata: name: promitor-agent-scraper namespace: monitoring labels: app: promitor-agent-scraper data: runtime.yaml: |- server: httpPort: "5000" authentication: mode: UserAssignedManagedIdentity identityId: metrics-identity metricSinks: prometheusScrapingEndpoint: metricUnavailableValue: "NaN" enableMetricTimestamps: "true" baseUriPath: "/metrics" labels: transformation: "None" metricsConfiguration: absolutePath: /config/metrics-declaration.yaml telemetry: applicationInsights: isEnabled: "false" containerLogs: isEnabled: "true" verbosity: "Debug" defaultVerbosity: "Debug" metrics-declaration.yaml: |- version: v1 azureMetadata: tenantId: xxxx-xxxxx-xxxxx-xxxx-xxxxx subscriptionId: xxxx-xxx-xxx-xxx-xxxx resourceGroupName: xxxxxxxxx cloud: Global metricDefaults: aggregation: interval: 00:02:00 scraping: schedule: "*/2 * * * *" metrics: - name: azure_load_balancer_traffic_bytes description: "Average amount of bytes sent through an Azure Load Balancer" resourceType: LoadBalancer scraping: schedule: "*/2 * * * *" azureMetricConfiguration: metricName: ByteCount aggregation: type: Average interval: 00:05:00 resources: # Optional, required when no resource discovery is configured - loadBalancerName: bda-azr-dev-wu2-zonal-agent-pool-internal --- apiVersion: apps/v1 kind: Deployment metadata: name: promitor-agent-scraper namespace: monitoring labels: aadpodidbinding: metrics-identity app: promitor-agent-scraper spec: replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 1 selector: matchLabels: app: promitor-agent-scraper template: metadata: labels: app: promitor-agent-scraper aadpodidbinding: metrics-identity spec: containers: - name: promitor-agent-scraper image: "ghcr.io/tomkerkhove/promitor-agent-scraper:latest" imagePullPolicy: Always ports: - name: http containerPort: 5000 protocol: TCP volumeMounts: - name: config-volume-promitor-agent-scraper mountPath: /config/ - name: tmp mountPath: /tmp/ imagePullSecrets: - name: registry-credentials volumes: - name: config-volume-promitor-agent-scraper configMap: name: promitor-agent-scraper - emptyDir: {} name: tmp ``` **I can see there is some authentication failure logs in the container logs though. Here is the container logs:** ``` ██████╗ ██████╗ ██████╗ ███╗ ███╗██╗████████╗ ██████╗ ██████╗ ██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗ ██████╔╝██████╔╝██║ ██║██╔████╔██║██║ ██║ ██║ ██║██████╔╝ ██╔═══╝ ██╔══██╗██║ ██║██║╚██╔╝██║██║ ██║ ██║ ██║██╔══██╗ ██║ ██║ ██║╚██████╔╝██║ ╚═╝ ██║██║ ██║ ╚██████╔╝██║ ██║ ╚═╝ ╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝ [14:06:55 INF] Booting up Promitor v2.6.1 running .NET .NET 6.0.3 - Thank you for using Promitor! [14:06:55 INF] Running .NET 6.0.3 on Linux (alpine.3.14-x64 | Linux 5.10.61-flatcar #1 SMP Tue Aug 31 15:41:58 -00 2021). [14:06:55 INF] Using configuration folder '/config/' [14:06:56 INF] Starting validation of Promitor setup [14:06:56 INF] Scrape configuration found at '/config/metrics-declaration.yaml' ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name │ Outcome │ Details ┃ ┠───────────────────────────────────┼─────────┼────────────────────────────────┨ ┃ Metrics Declaration Path │ Success │ Everything is well-configured. ┃ ┃ Azure Authentication │ Success │ Everything is well-configured. ┃ ┃ Metrics Declaration │ Success │ Everything is well-configured. ┃ ┃ Resource Discovery │ Success │ Everything is well-configured. ┃ ┃ StatsD Metric Sink │ Success │ Everything is well-configured. ┃ ┃ Prometheus Scraping Endpoint │ Success │ Everything is well-configured. ┃ ┃ Metric Sink │ │ ┃ ┃ Atlassian Statuspage Metric Sink │ Success │ Everything is well-configured. ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ Validation [14:06:56 INF] Promitor configuration is valid, we are good to go. [14:06:56 INF] Here's an overview of what was configured: ╭───────────────────┬───────────────────┬───────────────────┬──────────────────╮ │ Tenant Id │ Subscription Id │ Resource Group │ Azure Cloud │ │ │ │ Name (Default) │ │ ├───────────────────┼───────────────────┼───────────────────┼──────────────────┤ │ xxxx-xxxx-xxxxxxx │ xxxxxxxx-xxxx-xxx │ xxxxxxxxxxxxxxxxx │ Azure Global │ │ x-xxxx-xxxxxxxxxx │ x-xxxx-xxxxxxxxxx │ xx │ Cloud │ │ xx │ xx │ │ │ ╰───────────────────┴───────────────────┴───────────────────┴──────────────────╯ Azure Metadata ╭─────────────┬─────────────┬────────┬─────────────┬─────────────┬─────────────╮ │ Metric Name │ Resource │ Labels │ Azure │ Resources │ Resource │ │ │ Type │ │ Monitor │ To Scrape │ Discovery │ │ │ │ │ Metric │ │ Groups To │ │ │ │ │ │ │ Scrape │ ├─────────────┼─────────────┼────────┼─────────────┼─────────────┼─────────────┤ │ azure_load_ │ Load │ None │ ByteCount │ xxx-azr-dev │ None │ │ balancer_tr │ Balancer │ │ │ -xxx-zonal- │ │ │ affic_bytes │ │ │ │ agent-pool- │ │ │ │ │ │ │ internal │ │ ╰─────────────┴─────────────┴────────┴─────────────┴─────────────┴─────────────╯ Configured Metrics [14:06:57 INF] [Job][f7926029-4df3-43de-b973-baf17deefa69-k8sdevwu2clusterbda-azure_load_balancer_traffic_bytes-xxx-azr-dev-xxx-zonal-agent-pool-internal-fd6a9af8-799f-4136-b69a-8dd648655f61] is running under time zone: UTC [14:06:57 INF] Scraping Azure Monitor - 04/21/2022 14:06:57 +00:00 [14:06:57 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:06:57 INF] Adding Prometheus sink to expose on /metrics [14:06:57 INF] Adding Prometheus sink to expose on /metrics [14:06:57 INF] Now listening on: http://[::]:5000 [14:06:57 INF] Application started. Press Ctrl+C to shut down. [14:06:57 INF] Hosting environment: Production [14:06:57 INF] Content root path: /app [14:08:00 INF] Scraping Azure Monitor - 04/21/2022 14:08:00 +00:00 [14:08:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:10:00 INF] Scraping Azure Monitor - 04/21/2022 14:10:00 +00:00 [14:10:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:12:00 INF] Scraping Azure Monitor - 04/21/2022 14:12:00 +00:00 [14:12:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:14:00 INF] Scraping Azure Monitor - 04/21/2022 14:14:00 +00:00 [14:14:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:16:00 INF] Scraping Azure Monitor - 04/21/2022 14:16:00 +00:00 [14:16:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:18:00 INF] Scraping Azure Monitor - 04/21/2022 14:18:00 +00:00 [14:18:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:20:00 INF] Scraping Azure Monitor - 04/21/2022 14:20:00 +00:00 [14:20:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:22:00 INF] Scraping Azure Monitor - 04/21/2022 14:22:00 +00:00 [14:22:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:24:00 INF] Scraping Azure Monitor - 04/21/2022 14:24:00 +00:00 [14:24:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:26:00 INF] Scraping Azure Monitor - 04/21/2022 14:26:00 +00:00 [14:26:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:28:00 INF] Scraping Azure Monitor - 04/21/2022 14:28:00 +00:00 [14:28:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:30:00 INF] Scraping Azure Monitor - 04/21/2022 14:30:00 +00:00 [14:30:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:32:00 INF] Scraping Azure Monitor - 04/21/2022 14:32:00 +00:00 [14:32:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:34:00 INF] Scraping Azure Monitor - 04/21/2022 14:34:00 +00:00 [14:34:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:36:00 INF] Scraping Azure Monitor - 04/21/2022 14:36:00 +00:00 [14:36:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:38:00 INF] Scraping Azure Monitor - 04/21/2022 14:38:00 +00:00 [14:38:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:40:00 INF] Scraping Azure Monitor - 04/21/2022 14:40:00 +00:00 [14:40:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:42:00 INF] Scraping Azure Monitor - 04/21/2022 14:42:00 +00:00 [14:42:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:44:00 INF] Scraping Azure Monitor - 04/21/2022 14:44:00 +00:00 [14:44:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:46:00 INF] Scraping Azure Monitor - 04/21/2022 14:46:00 +00:00 [14:46:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:48:00 INF] Scraping Azure Monitor - 04/21/2022 14:48:00 +00:00 [14:48:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer [14:48:26 FTL] Failed to scrape resource for metric 'azure_load_balancer_traffic_bytes' Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSIMaxRetryReachedException: MSI: Failed to acquire tokens after retrying %20 times at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.RetrieveTokenFromIMDSWithRetryAsync(String resource, CancellationToken cancellationToken) at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetTokenFromIMDSEndpointAsync(String resource, CancellationToken cancellationToken) at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderForVirtualMachineAsync(String resource, CancellationToken cancellationToken) at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderAsync(CancellationToken cancellationToken) at Microsoft.Rest.TokenCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken) at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken) at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken) at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken) at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetMetricDefinitionsAsync(String resourceId) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 124 at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 80 at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 54 at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 78 ``` **Here are the metrics what promitor is exposing now:** ``` # HELP promitor_runtime_dotnet_collection_count_total GC collection count # TYPE promitor_runtime_dotnet_collection_count_total counter promitor_runtime_dotnet_collection_count_total{generation="0"} 2 promitor_runtime_dotnet_collection_count_total{generation="1"} 1 promitor_runtime_dotnet_collection_count_total{generation="2"} 0 # HELP promitor_runtime_dotnet_totalmemory Total known allocated memory # TYPE promitor_runtime_dotnet_totalmemory gauge promitor_runtime_dotnet_totalmemory 10389832 # HELP promitor_runtime_process_cpu_seconds_total Total user and system CPU time spent in seconds # TYPE promitor_runtime_process_cpu_seconds_total counter promitor_runtime_process_cpu_seconds_total 10.1 # HELP promitor_runtime_process_virtual_bytes Process virtual memory size # TYPE promitor_runtime_process_virtual_bytes gauge promitor_runtime_process_virtual_bytes 23826468864 # HELP promitor_runtime_process_working_set Process working set # TYPE promitor_runtime_process_working_set gauge promitor_runtime_process_working_set 195641344 # HELP promitor_runtime_process_private_bytes Process private memory size # TYPE promitor_runtime_process_private_bytes gauge promitor_runtime_process_private_bytes 174456832 # HELP promitor_runtime_process_num_threads Total number of threads # TYPE promitor_runtime_process_num_threads gauge promitor_runtime_process_num_threads 28 # HELP promitor_runtime_process_processid Process ID # TYPE promitor_runtime_process_processid gauge promitor_runtime_process_processid 1 # HELP promitor_runtime_process_start_time_seconds Start time of the process since unix epoch in seconds # TYPE promitor_runtime_process_start_time_seconds gauge promitor_runtime_process_start_time_seconds 1650550015 # HELP promitor_runtime_http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path # TYPE promitor_runtime_http_request_duration_seconds histogram promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.005"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.01"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.025"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.05"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.075"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.1"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.25"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.75"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="1"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="2.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="7.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="10"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="+Inf"} 1 promitor_runtime_http_request_duration_seconds_sum{status_code="200",method="GET",path="/metrics"} 0.040377444 promitor_runtime_http_request_duration_seconds_count{status_code="200",method="GET",path="/metrics"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.005"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.01"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.025"} 0 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.05"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.075"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.1"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.25"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.75"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="1"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="2.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="7.5"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="10"} 1 promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="+Inf"} 1 promitor_runtime_http_request_duration_seconds_sum{status_code="404",method="GET",path="/favicon.ico"} 0.030159532 promitor_runtime_http_request_duration_seconds_count{status_code="404",method="GET",path="/favicon.ico"} 1 # HELP promitor_scrape_error Provides an indication that the scraping of the resource has failed # TYPE promitor_scrape_error gauge promitor_scrape_error{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 1 1650552506423 # HELP promitor_scrape_success Provides an indication that the scraping of the resource was successful # TYPE promitor_scrape_success gauge promitor_scrape_success{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 0 1650552506422 ``` What I understood from the container log is, the UserAssignedManagedIdentity authentication is failing when scraping metrics from azure monitor. But, I tried the same UserAssignedManagedIdentity as aadpodidbinding with [azure_metrics_exporter](https://github.com/RobustPerception/azure_metrics_exporter) and it returns the metrics without any issues. So, wondering if I have done any mistakes in the configurations side? For authentication.identityId in runtime.yaml, I am using the same value what I am using for aadpodidbinding in deployment label section. I believe that is the correct configuration. Please help. Thanks,
Mimetis commented 2 years ago

Here are the checks I would do to investigate this issue (following the docs http://docs.promitor.io/walkthrough/use-promitor-with-managed-identity)

Be sure you have installed your AKS cluster using the managed identity option. Query the identity from the cluster:

echo "Retrieving cluster identity ID"
export aks_mi_identity="$(az aks show -g ${RG_NAME} -n ${CLUSTER_NAME} --query identityProfile.kubeletidentity.clientId -otsv)"

If I remember well, you may have the managed identity item also in the AKS resource group (called MC_....)

If you have created another managed identity (to respect the separation of concerns), be sure this identity has the "Monitor Reader" role assigned on your subscription / resource group

az role assignment list --assignee $AD_POD_IDENTITY_CLIENT_ID -g $RG_NAME | jq -r '.[].roleDefinitionName'

If you are using the system assigned identity, you can make the same check

From your cluster, check that your AAD Pod identity is correctly configured

kubectl get azureidentity
kubectl get azureidentitybinding

If you have correctly bind the AAD Pod Identity to your Managed Identity (System assigne or User Assigned) you should be able to use it. As an example, you can spin up a Azure CLI using this identity:

kubectl run azure-cli -it --image=mcr.microsoft.com/azure-cli --labels=aadpodidbinding=$AD_POD_IDENTITY_NAME /bin/bash

az login -i --debug

If you CLI is launching correctly that means your AAD pod Identity is configured correctly and ready to use.

Once you are sure everything is correctly configured on your AKS cluster, you can configure Promitor

azureAuthentication:
  mode: SystemAssignedManagedIdentity
  identity:
    binding: <aad-pod-identity-name>  
anoop2503 commented 2 years ago

@Mimetis Thank you so much for clear instructions.

I think the issue was with authentication mode. Based on the steps mentioned in your comment I tried the command "az login -i --debug" and found that the identity which I was using SystemAssignedManagedIdentity and not UserAssignedManagedIdentity. The application started scraping metrics from Azure monitor without any issues after changing the azureAuthentication mode to "SystemAssignedManagedIdentity".

Thanks for your help.

tomkerkhove commented 2 years ago

Great to hear that helped. Would it help if this is a troubleshooting guide on docs.promitor.io @anoop2503?

Thank you @Mimetis!