search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.89k stars 1.21k forks source link

Help: Trace-Apm not working (Linux(Debian)/Docker) but logging does #3221

Open jacklekk opened 5 years ago

jacklekk commented 5 years ago

What have I done incorrectly?

So I have made an application (.Net Core v2.2) API. I have a simple controller that returns an OK() 200 http status code.

I have made a DockerFile as so:

FROM microsoft/dotnet:2.2-aspnetcore-runtime-bionic AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM microsoft/dotnet:2.2-sdk-bionic AS build
WORKDIR /src
COPY ["appname/appname.csproj", "appname/"]
RUN dotnet restore -r ubuntu.18.04-x64 "appname/appname.csproj"
COPY . .
RUN dotnet build -r ubuntu.18.04-x64 -c Release -o /app "appname/appname.csproj"

FROM build AS publish
RUN dotnet publish -c Release -o /app "appname/appname.csproj"

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json
ENTRYPOINT ["dotnet", "appname.dll"]

I have installed the DataDog Agent on the host according to the documentation.

I had to add a docker_daemon.yaml file as the folder did not have one:

init_config:

instances:
    - url: "unix://var/run/docker.sock"
      new_tag_names: true

Not sure if it is right or I even need it.

I have altered the datadog.yaml file, which now contains:

api_key: xxxxxxxxx
log_level: DEBUG
logs_enabled: true
listeners:
  - name: docker
config_providers:
  - name: docker
    polling: true
logs_config:
  container_collect_all: true
apm_config:
    enabled: true

I have then made sure of the DataDog status:

===============
Agent (v6.10.2)
===============

  Status date: 2019-03-28 13:28:22.808502 UTC
  Pid: 16813
  Python Version: 2.7.15
  Logs:
  Check Runners: 4
  Log Level: DEBUG

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 353µs
    System UTC time: 2019-03-28 13:28:22.808502 UTC

  Host Info
  =========
    bootTime: 2019-03-20 14:11:31.000000 UTC
    kernelVersion: 4.18.0-1013-azure
    os: linux
    platform: ubuntu
    platformFamily: debian
    platformVersion: 18.04
    procs: 206
    uptime: 191h8m3s
    virtualizationRole: host
    virtualizationSystem: kvm

  Hostnames
  =========
    host_aliases: [010ad375-75b5-43ce-939a-5a745ebfc01b]
    hostname: dev-vm4
    socket-fqdn: dev-vm4
    socket-hostname: dev-vm4
    hostname provider: os
    unused hostname providers:
      aws: not retrieving hostname from AWS: the host is not an ECS instance, and other providers already retrieve non-default hostnames
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://xxx.xxx.xxx.xxx/computeMetadata/v1/instance/hostname

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Total Runs: 35
      Metric Samples: Last Run: 6, Total: 204
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    disk (2.1.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [OK]
      Total Runs: 35
      Metric Samples: Last Run: 106, Total: 3,710
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 77ms

    docker
    ------
      Instance ID: docker [OK]
      Total Runs: 36
      Metric Samples: Last Run: 22, Total: 792
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 36
      Average Execution Time : 10ms

    file_handle
    -----------
      Instance ID: file_handle [OK]
      Total Runs: 35
      Metric Samples: Last Run: 5, Total: 175
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    io
    --
      Instance ID: io [OK]
      Total Runs: 35
      Metric Samples: Last Run: 104, Total: 3,568
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    load
    ----
      Instance ID: load [OK]
      Total Runs: 35
      Metric Samples: Last Run: 6, Total: 210
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    memory
    ------
      Instance ID: memory [OK]
      Total Runs: 35
      Metric Samples: Last Run: 17, Total: 595
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

    network (1.9.0)
    ---------------
      Instance ID: network:2a218184ebe03606 [OK]
      Total Runs: 35
      Metric Samples: Last Run: 33, Total: 1,155
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 3ms

    ntp
    ---
      Instance ID: ntp:b4579e02d1981c12 [OK]
      Total Runs: 35
      Metric Samples: Last Run: 1, Total: 35
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 35
      Average Execution Time : 0s

    uptime
    ------
      Instance ID: uptime [OK]
      Total Runs: 35
      Metric Samples: Last Run: 1, Total: 35
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 35
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 3
    Metadata: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 73
    TimeseriesV1: 35

  API Keys status
  ===============
    API key ending with xxxxxx: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - xxxxxx

==========
Logs Agent
==========

  container_collect_all
  ---------------------
    Type: docker
    Status: OK
    Inputs: 25301d3d7855426a2baba3a47b184afd2e58548014b5b14e98d89f3782ad2b46

=========
Aggregator
=========
  Checks Metric Sample: 11,112
  Dogstatsd Metric Sample: 506
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 35
  Series Flushed: 9,779
  Service Check: 388
  Service Checks Flushed: 420

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 506
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Packet Reading Errors: 0
  Udp Packets: 507
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

I have installed the following Nuget Package to the application: <PackageReference Include="Datadog.Trace.ClrProfiler.Managed" Version="0.8.1-beta" />

Which I then ran the command on the Linux environment:

curl -LO https://github.com/DataDog/dd-trace-dotnet/releases/download/v0.8.1-beta/datadog-dotnet-apm_0.8.1_amd64.deb
sudo dpkg -i ./datadog-dotnet-apm_0.8.1_amd64.deb

I then run the docker container with a simple command: docker run -p 4004:80 -d --name apptest

then I check the docker container environment:

 "Env": [
                 "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ASPNETCORE_URLS=http://+:80",
                "DOTNET_RUNNING_IN_CONTAINER=true",
                "ASPNETCORE_VERSION=2.2.3",
                "CORECLR_ENABLE_PROFILING=1",
                "CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}",
                "CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so",
                "DD_INTEGRATIONS=/opt/datadog/integrations.json"
            ],

I then execute the endpoint by processing this command:

curl -i -H "Accept: application/json" http://localhost:4001/api/v1/Test

This action after 1 minute, will occur in the logs are of DataDog.

No APM data...

The trace-agent logs say this:

2019-03-28 13:18:56 INFO (trace.go:60) - Trace writer initializing with config: {MaxSpansPerPayload:1000 FlushPeriod:5s UpdateInfoPeriod:1m0s SenderConfig:{MaxAge:20m0s MaxQueuedBytes:67108864 MaxQueuedPayloads:-1 ExponentialBackoff:{MaxDuration:2m0s GrowthBase:2 Base:200ms}}}
2019-03-28 13:18:56 INFO (stats.go:47) - Stats writer initializing with config: {MaxEntriesPerPayload:12000 UpdateInfoPeriod:1m0s SenderConfig:{MaxAge:20m0s MaxQueuedBytes:67108864 MaxQueuedPayloads:-1 ExponentialBackoff:{MaxDuration:2m0s GrowthBase:2 Base:200ms}}}
2019-03-28 13:18:56 INFO (service.go:37) - Service writer initializing with config: {FlushPeriod:5s UpdateInfoPeriod:1m0s SenderConfig:{MaxAge:20m0s MaxQueuedBytes:67108864 MaxQueuedPayloads:-1 ExponentialBackoff:{MaxDuration:2m0s GrowthBase:2 Base:200ms}}}
2019-03-28 13:18:56 INFO (run.go:136) - trace-agent running on host dev-vm4
2019-03-28 13:18:56 INFO (api.go:144) - listening for traces at http://localhost:8126
2019-03-28 13:18:56 DEBUG (service.go:69) - starting service writer
2019-03-28 13:18:56 DEBUG (concentrator.go:84) - starting concentrator
2019-03-28 13:18:56 DEBUG (trace.go:123) - starting trace writer
2019-03-28 13:18:56 DEBUG (stats.go:79) - starting stats writer
2019-03-28 13:19:01 DEBUG (trace.go:130) - Flushing current traces
2019-03-28 13:19:06 DEBUG (trace.go:130) - Flushing current traces
2019-03-28 13:19:06 INFO (api.go:341) - no data received
2019-03-28 13:19:06 DEBUG (sampler.go:112) - *sampler.ScoreEngine: flushed 0 sampled traces out of 0
2019-03-28 13:19:06 DEBUG (sampler.go:118) - *sampler.ScoreEngine: inTPS: 0.000000, outTPS: 0.000000, maxTPS: 10.000000, offset: 1.000000, slope: 3.000000, cardinality: 0
2019-03-28 13:19:06 DEBUG (sampler.go:112) - *sampler.ScoreEngine: flushed 0 sampled traces out of 0
2019-03-28 13:19:06 DEBUG (sampler.go:118) - *sampler.ScoreEngine: inTPS: 0.000000, outTPS: 0.000000, maxTPS: 10.000000, offset: 1.000000, slope: 3.000000, cardinality: 0
2019-03-28 13:19:06 DEBUG (sampler.go:112) - *sampler.PriorityEngine: flushed 0 sampled traces out of 0
2019-03-28 13:19:06 DEBUG (sampler.go:118) - *sampler.PriorityEngine: inTPS: 0.000000, outTPS: 0.000000, maxTPS: 10.000000, offset: 1.000000, slope: 3.000000, cardinality: 0
2019-03-28 13:19:11 DEBUG (trace.go:130) - Flushing current traces

And a bit further down: 2019-03-28 13:21:34 INFO (service.go:63) - total number of tracked services: 0

Describe what happened: Only logging occurs correctly.

Describe what you expected: To see entries on the dashboard for APM as well as the logging.

Steps to reproduce the issue: Linux (Debian) Docker .Net Core v2.2.3 Api application with a simple controller for returning a 200 Ok Status

Additional environment details (Operating System, Cloud provider, etc):

ghost commented 4 years ago

I'm running into the same issue. I've used the same dd-agent with a node project and traces work fine. @jacklekk Have you gotten this to work?