search

DataDog / integrations-core

Core integrations of the Datadog Agent
BSD 3-Clause "New" or "Revised" License
932 stars 1.4k forks source link

HTTP basic auth is not possible for integrations that require empty string values for username or password #12390

Closed samg-monster closed 2 years ago

samg-monster commented 2 years ago

Output of the info page

===============
Agent (v7.36.1)
===============

  Status date: 2022-06-22 04:22:06.629 UTC (1655871726629)
  Agent start: 2022-06-22 03:24:29.871 UTC (1655868269871)
  Pid: 380
  Go Version: go1.17.6
  Python Version: 3.8.11
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: debug

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 87.36ms
    System time: 2022-06-22 04:22:06.629 UTC (1655871726629)

  Host Info
  =========
    bootTime: 2022-06-22 00:38:28 UTC (1655858308000)
    hostId: 4a7f4fd8-0000-0000-99f8-bfb7eb8e0c53
    kernelArch: x86_64
    kernelVersion: 5.10.104-linuxkit
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 21.10
    procs: 169
    uptime: 2h46m8s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    hostname: docker-desktop
    socket-fqdn: 18ffe09f11c2
    socket-hostname: 18ffe09f11c2
    hostname provider: container
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
      azure: azure_hostname_style is set to 'os'
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: GCE metadata API error: Get "http://169.254.169.254/computeMetadata/v1/instance/hostname": dial tcp 169.254.169.254:80: i/o timeout (Client.Timeout exceeded while awaiting headers)

  Metadata
  ========
    agent_version: 7.36.1
    config_apm_dd_url:
    config_dd_url:
    config_logs_dd_url:
    config_logs_socks5_proxy_address:
    config_no_proxy: []
    config_process_dd_url:
    config_proxy_http:
    config_proxy_https:
    config_site:
    feature_apm_enabled: true
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: false
    feature_networks_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: false
    feature_processes_container_enabled: true
    flavor: agent
    hostname_source: container
    install_method_installer_version: docker
    install_method_tool: docker
    install_method_tool_version: docker

=========
Collector
=========

  Running Checks
  ==============

    container
    ---------
      Instance ID: container [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/container.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 31, Total: 7,130
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 6ms
      Last Execution Date : 2022-06-22 04:21:54 UTC (1655871714000)
      Last Successful Execution Date : 2022-06-22 04:21:54 UTC (1655871714000)

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 9, Total: 2,063
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-06-22 04:22:01 UTC (1655871721000)
      Last Successful Execution Date : 2022-06-22 04:22:01 UTC (1655871721000)

    disk (4.7.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 296, Total: 68,080
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 52ms
      Last Execution Date : 2022-06-22 04:21:53 UTC (1655871713000)
      Last Successful Execution Date : 2022-06-22 04:21:53 UTC (1655871713000)

    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 11, Total: 2,530
      Events: Last Run: 1, Total: 2
      Service Checks: Last Run: 1, Total: 230
      Average Execution Time : 19ms
      Last Execution Date : 2022-06-22 04:22:00 UTC (1655871720000)
      Last Successful Execution Date : 2022-06-22 04:22:00 UTC (1655871720000)

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/file_handle.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 5, Total: 1,150
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-06-22 04:21:52 UTC (1655871712000)
      Last Successful Execution Date : 2022-06-22 04:21:52 UTC (1655871712000)

    io
    --
      Instance ID: io [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/io.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 28, Total: 6,422
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 1ms
      Last Execution Date : 2022-06-22 04:21:59 UTC (1655871719000)
      Last Successful Execution Date : 2022-06-22 04:21:59 UTC (1655871719000)

    load
    ----
      Instance ID: load [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/load.d/conf.yaml.default
      Total Runs: 231
      Metric Samples: Last Run: 6, Total: 1,386
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-06-22 04:22:06 UTC (1655871726000)
      Last Successful Execution Date : 2022-06-22 04:22:06 UTC (1655871726000)

    memory
    ------
      Instance ID: memory [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/memory.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 20, Total: 4,600
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-06-22 04:21:58 UTC (1655871718000)
      Last Successful Execution Date : 2022-06-22 04:21:58 UTC (1655871718000)

    network (2.7.0)
    ---------------
      Instance ID: network:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/network.d/conf.yaml.default
      Total Runs: 231
      Metric Samples: Last Run: 118, Total: 27,258
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 10ms
      Last Execution Date : 2022-06-22 04:22:05 UTC (1655871725000)
      Last Successful Execution Date : 2022-06-22 04:22:05 UTC (1655871725000)

    ntp
    ---
      Instance ID: ntp:d884b5186b651429 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/ntp.d/conf.yaml.default
      Total Runs: 4
      Metric Samples: Last Run: 1, Total: 4
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 4
      Average Execution Time : 662ms
      Last Execution Date : 2022-06-22 04:09:34 UTC (1655870974000)
      Last Successful Execution Date : 2022-06-22 04:09:34 UTC (1655870974000)

    sonarqube (2.0.1)
    -----------------
      Instance ID: sonarqube:9fef5d0883332e3b [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/sonarqube.d/conf.yaml
      Total Runs: 231
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 231
      Average Execution Time : 1.046s
      Last Execution Date : 2022-06-22 04:22:03 UTC (1655871723000)
      Last Successful Execution Date : 2022-06-22 04:22:03 UTC (1655871723000)

    uptime
    ------
      Instance ID: uptime [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
      Total Runs: 230
      Metric Samples: Last Run: 1, Total: 230
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2022-06-22 04:21:57 UTC (1655871717000)
      Last Successful Execution Date : 2022-06-22 04:21:57 UTC (1655871717000)

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Ingress: 0
    Job: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 487
    Successes By Endpoint:
      check_run_v1: 230
      intake: 21
      metadata_v1: 6
      series_v1: 230

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with de44a: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - de44a

==========
Logs Agent
==========

  Logs Agent is not running

=============
Process Agent
=============

  Version: 7.36.1
  Status date: 2022-06-22 04:22:11.184 UTC (1655871731184)
  Process Agent Start: 2022-06-22 03:24:29.97 UTC (1655868269970)
  Pid: 378
  Go Version: go1.17.6
  Build arch: amd64
  Log Level: debug
  Enabled Checks: [container rtcontainer process_discovery]
  Allocated Memory: 15,468,088 bytes
  Hostname: docker-desktop

  =================
  Process Endpoints
  =================
    https://process.datadoghq.com - API Key ending with:
        - de44a

  =========
  Collector
  =========
    Last collection time: 2022-06-22 04:22:10
    Docker socket: /var/run/docker.sock
    Number of processes: 0
    Number of containers: 1
    Process Queue length: 0
    RTProcess Queue length: 0
    Pod Queue length: 0
    Process Bytes enqueued: 0
    RTProcess Bytes enqueued: 0
    Pod Bytes enqueued: 0

=========
APM Agent
=========
  Status: Running
  Pid: 376
  Uptime: 3465 seconds
  Mem alloc: 9,705,864 bytes
  Hostname: docker-desktop
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.com

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 130,527
  Dogstatsd Metric Sample: 37,852
  Event: 3
  Events Flushed: 3
  Number Of Flushes: 230
  Series Flushed: 126,562
  Service Check: 3,002
  Service Checks Flushed: 3,224
=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 37,851
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 5,716,236
  Udp Packet Reading Errors: 0
  Udp Packets: 21,982
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0
  Unterminated Metric Errors: 0

=============
Autodiscovery
=============
  Enabled Features
  ================
    docker

Additional environment details (Operating System, Cloud provider, etc): Docker agent running on MacOS

Steps to reproduce the issue: This issue can be reproduced using the Datadog Sonarqube integration as an example.

  1. Validate that you can retrieve metrics from the target Sonarqube server using curl:
curl -I -u <user token>: https://<sonarqube server>/api/metrics/search

The result should be an HTTP 200 status code to indicate that the authentication was successful. Note that the server expects an empty string for the password value and will fail to authenticate otherwise.

  1. Configure a Datadog agent to collect metrics from the Sonarqube server via the web API by creating the file sonarqube.d/conf.yaml in the agent's conf.d directory with the following contents: 
    init_config:
is_jmx: false
collect_default_metrics: true
instances:
# Web API instance
- is_jmx: false
web_endpoint: <sonarqube server URL>
log_requests: true
auth_type: basic
username: "<sonarqube user token>"
password: ""
components:
  my-project:
    tag: my-project
  2. Start the agent with DD_LOG_LEVEL=debug set and check the agent logs in /var/log/datadog/agent.log. You will see HTTP 401 errors as the agent cannot authenticate to the Sonarqube server. Example: 
    2022-06-22 03:24:37 UTC | CORE | DEBUG | (pkg/collector/python/datadog_agent.go:128 in LogMessage) | - | (connectionpool.py:456) | https://sonarqube.organisation.com:443 "GET /api/metrics/search?p=1 HTTP/1.1" 401 0

    Describe the results you received: The agent retrieves no metrics from the Sonarqube server as it cannot authenticate.

Describe the results you expected: The agent should successfully authenticate and retrieve metrics.

Additional information you deem important (e.g. issue happens only occasionally):

The issue is caused by this line of code: https://github.com/DataDog/integrations-core/blob/981926c042d380b6811df07cca2cb2bc56967976/datadog_checks_base/datadog_checks/base/utils/http.py#L630

If an empty string is provided as the password this conditional evaluates to false and prevents the authentication from progressing further. The solution is to change the line to 

if config['username'] is not None and config['password'] is not None:
fanny-jiang commented 2 years ago

Hi @samg-monster, thanks for opening this issue! I'm taking a look into this and will follow up here.

alopezz commented 2 years ago

The fix for this has been merged (#12437) and will be included in a future agent release. Thanks for reporting!