DataDog / dd-trace-rb

Datadog Tracing Ruby Client
https://docs.datadoghq.com/tracing/
Other
311 stars 375 forks source link

SemanticLogger integration not correlating logs with server/environment #2356

Open Drowze opened 2 years ago

Drowze commented 2 years ago

Current behaviour SemanticLogger integration is not correlating logs with server/environment (configured via DD_SERVICE/DD_ENV)

Expected behaviour Given the quote below, I'd expect rails_semantic_logger to just work with Datadog and correlate the logs with their traces and correct env/service (i.e. the ones supplied by the DD_ENV and DD_SERVICE environment variables)

For Rails applications using the default logger (ActiveSupport::TaggedLogging), lograge, or semantic_logger, trace correlation injection is automatically configured. _from: https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/ruby/_

Steps to reproduce

Environment

Drowze commented 2 years ago

Some further testing, this time NOT using any Ruby process. With only a datadog agent with the following logs config:

logs:
  - type: tcp
    port: 10518
    service: log-echo
    source: ruby
  1. A JSON log with dd on the root object.

    echo '{
      "dd": {
        "trace_id": "754516775358317257",
        "env": "rafael-testing",
        "service": "auth-user",
        "version":42
      },
      "ddsource": "ruby-overwrite",
      "message": "HI DAWG"
    }' | jq -c | nc localhost 10518

    trace reported ✅ environment reported ✅ service reported (overwritten) ✅ version reported ✅ source: ignored (not overwritten) :x:

  2. A JSON log with dd inside a named_tags object (this is how SemanticLogger does it by default)

    echo '{
      "named_tags": {
        "dd": {
          "trace_id": "754516775358317257",
          "env": "rafael-testing",
          "service": "auth-user",
          "version":42
        },
        "ddsource": "ruby-overwrite"
      },
      "message": "HI DAWG"
    }' | jq -c | nc localhost 10518

    trace reported ✅ environment ignored :x: service ignored(not overwritten) :x: version ignored :x: source ignored (not overwritten) :x:

  3. A JSON log with dd inside an object with a name different than named_tags

    echo '{
      "weirdly_named": {
        "dd": {
          "trace_id": "754516775358317257",
          "env": "rafael-testing",
          "service": "auth-user",
          "version":42
        },
        "ddsource": "ruby-overwrite"
      },
      "message": "HI DAWG"
    }' | jq -c | nc localhost 10518

    trace ignored :x: environment ignored :x: service ignored(not overwritten) :x: version ignored :x: source ignored (not overwritten) :x:

  4. A formatted plain string

    echo '[dd.env=rafael-testing dd.service=auth-user dd.version=42 dd.source=ruby-overwrite dd.trace_id=754516775358317257] HI DAWG' | nc localhost 10518

    trace reported ✅ environment reported ✅ service reported (overwritten) ✅ version reported ✅ source: ignored (not overwritten) :x:


Seeing the above results, some points:

TonyCTHsu commented 2 years ago

👋 @Drowze Thanks for reporting, I will take a look at this issue.

TonyCTHsu commented 2 years ago

👋 @Drowze , I want to get back to you about this issue.

I am still communicating with our internal backend team to figure out the culprit that is causing the correlation issue with semantic logger.

So far, with semantic logger format, only correlation for trace_id and span_id in the named_tags are implemented. service and source from agent configuration also seems to be implemented.

I will keep you posted with my research and make changes for tracer if necessary. 😉

TonyCTHsu commented 1 year ago

@Drowze Sorry I have not made too much progress on this.

Could you file for a ticket about this issue? This would go through layers of triage from our support engineers(and to track this), would eventually find the owner of this.

https://help.datadoghq.com/hc/en-us/requests/new

You could simply referred to this Github issue, and tagged me for suggesting doing so.

silasb commented 7 months ago

Any updates around this?

tylerwillingham commented 1 month ago

We're noticing an issue in our setup that may be similar but possibly distinct.

We're in a containerized environment and the SemanticLogger.host attribute is picking up our pod name and propagating that up as the host value in our logs. (i.e. a JSON log starts with { host: 'pod-name-example' ....

This has caused us correlation headaches where we now have significant deltas in how logs from a single pod are reported because the majority of logs are application logs via rails_semantic_logger including the invalid hostname.

It looks like datadog stream-logs has the proper host information associated at the agent-level, but the log's value is overriding it. To get around this, I need a custom formatter that looks like:

module SemanticLogger
  module Formatters
    class HostlessJson < SemanticLogger::Formatters::Raw
      def initialize(time_format: :iso_8601, time_key: :timestamp, **args)
        super(time_format: time_format, time_key: time_key, **args)
      end

      # NOTE: `#call` is where all of the changes live. This is basically a carbon copy of SemanticLogger::Formatters::Json
      #  formatter but with the `h.delete(:host)` call.
      def call(log, logger)
        log_hash = super(log, logger).tap do |h|
          h.delete :host
        end.to_json
      end
    end
  end
end

I think this could equally be seen as a Datadog or a SemanticLogger issue since SemanticLogger.host is implemented in such a way that it's always non-nil, but the Datadog instrumentation is implemented via a patch, rather than a custom appender but the impact this has on us is greater on the Datadog side, as a customer


Edit: It turns out SemanticLogger has an API for this behavior if you specify a custom formatter.

# config/environments/production.rb
log_formatter = SemanticLogger::Formatters::Json.new(
  log_host: false # ✨✨✨
)
config.semantic_logger.add_appender(io: STDOUT, formatter: log_formatter)
januszm commented 2 weeks ago

I see that this issue was opened 2 years ago, does it mean semantic_logger integration is still broken or will we see trace_id and span_id in logs and proper correlation with default semantic_logger setup and dd c.tracing.log_injection = true ?

marcotc commented 2 weeks ago

👋 Sorry for the delay here. I tried a very simple local app outputting SemanticLogger log lines, and I can't get the any sort of rich parsing from the Logs product:

# I get "No attributes have been extracted from the log message." for
SemanticLogger.add_appender(io: log_output) 

# I don't get the log line at all for
SemanticLogger.add_appender(io: log_output, formatter: :json)

# For reference, my structured message looks like this in memory:
{"host":"my-host","application":"Semantic Logger","timestamp":"2024-11-08T19:43:49.177537Z","level":"info","level_index":2,"pid":1234,"thread":"132","named_tags":{"dd":{"env":"production","service":"MyService","version":"1.2.3","trace_id":"1039570011634864780","span_id":"1469129669426488971"},"ddsource":"ruby"},"name":"TestClass","message":"My Message!"}

I'll try to setup a basic Rails app from scratch, then add SemanticLogger, to ensure that I'm using a realistic setup, instead of manually using the SemanticLogger API which is not a common application configuration.

tylerwillingham commented 2 weeks ago

I use SemanticLogger at work with datadog 2.x (we're running on 2.5 in prod currently).

rails_semantic_logger + datadog gems work as expected.

In config/environments/production.rb there are a couple of additional lines to log to STDOUT via JSON and you can use config.rails_semantic_logger.format = :json in config/environments/development.rbto getlogs/development.log` printing in JSON under RAILS_ENV=development.

Trace correlation works out of the box for us with the current SemanticLogger integration BUT I wanted to post that bit about host correlation because it was a gotcha for us which caused DD logs to not correlate to their host infra properly until I dropped the host value from the payload.

I was able to refactor it to be more simple by using:

config.semantic_logger.add_appender(
  io: STDOUT,
  formatter: SemanticLogger::Formatters::Json.new(log_host: false)
)
marcotc commented 1 week ago

Thank you, @tylerwillingham.

I'll play around with the log_host option and document the incompatibility in a clear way (maybe emitting a warning for it).