search

tomkerkhove / promitor

Bringing Azure Monitor metrics where you need them.
https://promitor.io
MIT License
248 stars 91 forks source link

Failed to scrape resource for metric -> System.NullReferenceException #2263

Open andreibogdan opened 1 year ago

andreibogdan commented 1 year ago

Report

Hi, I am using promitor-agent-scraper:2.8 with the following config metric declaration config:

podLabels:
  name: promitor-agent-scraper
azureMetadata:
  tenantId: XXXX
  subscriptionId: XXXXX
  resourceGroupName: XXX
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "55 * * ? * *"
metrics:
  - name: azure_service_bus_queue_successful_requests
    description: "The number of successful requests"
    resourceType: ServiceBusNamespace
    azureMetricConfiguration:
      metricName: SuccessfulRequests
      aggregation:
        type: Total
    resources:
      - namespace: XXXXX
        queueName: xxxx-queue
        resourceGroupName: xxxxxx

When running using docker the metric is scraped but when running in a K8s cluster with Promitor scraper deployed via helm(latest version) I got the following error. On K8s namespace, I deployed only the scrapper.

Expected Behavior

Metrics to show up

Actual Behavior

When running using docker the metric is scraped but when running in a K8s cluster with Promitor scraper deployed via helm(latest version) I got the following error. On K8s namespace, I deployed only the scrapper.

Steps to Reproduce the Problem

  1. deploy on azure k8s using helm

  3. ...

Component

Scraper

Version

2.8.0

Configuration

Configuration:

podLabels:
  name: promitor-agent-scraper
azureMetadata:
  tenantId: xxxxx
  subscriptionId: xxxxx
  resourceGroupName: xxxxxx
metricDefaults:
  aggregation:
    interval: 00:05:00
  scraping:
    schedule: "55 * * ? * *"
metrics:
  - name: azure_service_bus_queue_successful_requests
    description: "The number of successful requests"
    resourceType: ServiceBusNamespace
    azureMetricConfiguration:
      metricName: SuccessfulRequests
      aggregation:
        type: Total
    resources:
      - namespace: xxx-stage
        queueName: xxx-queue

Logs

[15:37:52 INF] Booting up Promitor v2.8.0 running .NET .NET 6.0.9 - Thank you for using Promitor!
[15:37:52 INF] Running .NET 6.0.9 on Linux (alpine.3.16-x64 | Linux 5.15.43-flatcar #1 SMP Tue May 31 00:28:53 -00 2022).
[15:37:52 INF] Using configuration folder '/config/'
[15:37:53 INF] The following metric sinks were configured and are being enabled:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ Metric Sink                  │ Details        ┃
┠──────────────────────────────┼────────────────┨
┃ Prometheus Scraping Endpoint │ Url: /metrics. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
             Configured Metric Sinks
[15:37:53 INF] Starting validation of Promitor setup
[15:37:53 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                              │ Outcome │ Details                        ┃
┠───────────────────────────────────┼─────────┼────────────────────────────────┨
┃ Metrics Declaration Path          │ Success │ Everything is well-configured. ┃
┃ Azure Authentication              │ Success │ Everything is well-configured. ┃
┃ Metrics Declaration               │ Success │ Everything is well-configured. ┃
┃ Resource Discovery                │ Success │ Everything is well-configured. ┃
┃ Atlassian Statuspage Metric Sink  │ Success │ Everything is well-configured. ┃
┃ OpenTelemetry Collector Metric    │ Success │ Everything is well-configured. ┃
┃ Sink                              │         │                                ┃
┃ Prometheus Scraping Endpoint      │ Success │ Everything is well-configured. ┃
┃ Metric Sink                       │         │                                ┃
┃ StatsD Metric Sink                │ Success │ Everything is well-configured. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                   Validation
[15:37:53 INF] Promitor configuration is valid, we are good to go.
[15:37:53 INF] Here's an overview of what was configured:
╭───────────────────┬───────────────────┬───────────────────┬──────────────────╮
│ Tenant Id         │ Subscription Id   │ Resource Group    │ Azure Cloud      │
│                   │                   │ Name (Default)    │                  │
├───────────────────┼───────────────────┼───────────────────┼──────────────────┤
│ fxxx │ xxxxx │ xxxxx │ Azure Global     │
│ xxxx │ xxxxx │ e                 │ Cloud            │
│ e1                │ 87                │                   │                  │
╰───────────────────┴───────────────────┴───────────────────┴──────────────────╯
                                 Azure Metadata
╭─────────────┬─────────────┬────────┬─────────────┬─────────────┬─────────────╮
│ Metric Name │ Resource    │ Labels │ Azure       │ Resources   │ Resource    │
│             │ Type        │        │ Monitor     │ To Scrape   │ Discovery   │
│             │             │        │ Metric      │             │ Groups To   │
│             │             │        │             │             │ Scrape      │
├─────────────┼─────────────┼────────┼─────────────┼─────────────┼─────────────┤
│ azure_servi │ Service Bus │ None   │ SuccessfulR │ xxxx │ None        │
│ ce_bus_queu │ Namespace   │        │ equests     │ xxx     │             │
│ e_successfu │             │        │             │             │             │
│ l_requests  │             │        │             │             │             │
╰─────────────┴─────────────┴────────┴─────────────┴─────────────┴─────────────╯
                               Configured Metrics
[15:37:53 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:53 INF] Adding Prometheus sink to expose on /metrics
[15:37:53 INF] Adding Prometheus sink to expose on /metrics
[15:37:53 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:53 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:37:54 INF] Now listening on: http://[::]:8081
[15:37:54 INF] Application started. Press Ctrl+C to shut down.
[15:37:54 INF] Hosting environment: Production
[15:37:54 INF] Content root path: /app
[15:37:54 INF] [Job][Scraper-1Metrics-At_55_seconds_past_the_minute-0d2019a1316a4b87a5349e97d3d74bc2] is running under time zone: UTC
[15:37:54 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:37:55 INF] Scraping azure_service_bus_queue_successful_requests for resource type ServiceBusNamespace.
[15:38:00 INF] Request starting HTTP/1.1 GET http://172.18.191.168:8081/api/v1/health?includeDependencies=false - -
[15:38:01 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.Scraper.Controllers.v1.HealthController (Promitor.Agents.Scraper).
[15:38:55 FTL] Failed to scrape resource for metric 'azure_service_bus_queue_successful_requests'
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Http.AdalHttpClient.GetResponseAsync[T]()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.DiscoverAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.GetMetadataEntryAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Instance.Authenticator.UpdateFromTemplateAsync(RequestContext requestContext)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.PreRunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.RunAsync()
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource, ClientKey clientKey)
   at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(String resource, ClientCredential clientCredential)
   at Microsoft.Rest.Azure.Authentication.Internal.MemoryApplicationAuthenticationProvider.AuthenticateAsync(String clientId, String audience, AuthenticationContext context)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, IApplicationAuthenticationProvider authenticationProvider, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, ClientCredential credential, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, String secret, ActiveDirectoryServiceSettings settings, TokenCache cache)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetMetricDefinitionsAsync(String resourceId) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 128
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 84
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 54
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 78

Platform

Microsoft Azure

Contact Details

andrei.bogdan.mail@gmail.com