instana / ruby-sensor

💎 Ruby Distributed Tracing & Metrics Sensor for Instana
https://www.instana.com/
MIT License
26 stars 25 forks source link

Trace is listed as Unspecified when additional HTTP headers are present in the request. #160

Closed Adithya-copart closed 5 years ago

Adithya-copart commented 5 years ago

Adding a header to the HTTP request is altering how the trace is displayed in the dashboard. This is using a Sinatra application.

Whenever a correlationid is part of the HTTP request headers, the trace endpoint name appears as "Unspecified".

This is the visible difference between a unspecified and a proper trace json.

## Unspecified trace

   "data" : {
      "http" : {
        "0" : {
          "method" : "GET",
          "host" : "example.com",
          "url" : "/test",
          "status" : 200
        },
        "correlationid" : "testing_instana"
      }
    }

## Properly reported trace
    "data" : {
      "http" : {
        "rawUrl" : "/test",
        "path" : "/test",
        "method" : "GET",
        "host" : "example.com",
        "status" : 200
      }
    }
pglombardo commented 5 years ago

Hi @Adithya-copart thanks for reporting. At first glance I can't imagine how we would get a string key of "0" since the gem internally prefers symbols and calls to_sym on most strings. I wonder if this could be something specific to JRuby?

At least for MRI Ruby, we have tests to validate the generated Keys/Values and they are all green. :-/

This will need some testing.

Separately, I just released 1.10.5 which fixes how custom headers are reported so they properly show up in span details:

Instana Ruby custom headers 2019-09-20 at 11 55 42 AM

pglombardo commented 5 years ago

Running the Rack tests under Jruby also pass:

Screen Shot 2019-09-20 at 12 39 12 PM

Any idea on how I could reproduce what you're getting?

Adithya-copart commented 5 years ago

The method, host, url and status keys are being added in `rack instrumentation.

The JSON is downloaded from the dashboard and the key rawUrl seems to be added by the agent. I went ahead and verified if the kvs value varied with the presence of a new header but it did not change.

https://github.com/instana/ruby-sensor/blob/449ef63224de891acff7237bc2361a8e09a57870/lib/instana/instrumentation/rack.rb#L78

I don't think this is related to JRuby.

I will try to deploy a sample application and also enable instana debugging to see if I can find something more concrete.

pglombardo commented 5 years ago

Ok for information: the trace downloaded from the dashboard reflects post processing and will add in a bunch more stuff. If you want to access the raw spans as they're being reported, you can log spans to a file from the Instana host agent: https://docs.instana.io/quick_start/agent_configuration/#logging-metrics-or-traces-to-file

That is helpful to get the raw traces before they are processed and enriched by the backend.

Adithya-copart commented 5 years ago

I tried version 1.10.6 and this is the JSON from the dashboard. The issue seems to be resolved. I will re-open this issue and will add the raw spans if anything changes.

    "data" : {
      "http" : {
        "rawUrl" : "/foo",
        "path" : "/foo",
        "method" : "GET",
        "headerSearch" : [ "correlationid=testing_instana" ],
        "host" : "example.com",
        "header" : {
          "correlationid" : "testing_instana"
        },
        "status" : 200
      }
    }