skylightio / skylight-ruby

Skylight agent for Ruby
https://www.skylight.io/
Other
314 stars 76 forks source link

failed to instrument span; msg=native Trace#start_span failed #48

Closed airblade closed 7 years ago

airblade commented 9 years ago

I'm using skylight-ruby v0.6.1 and Rails 3.2. My skylight.log is full of:

[SKYLIGHT] [0.6.1] native Trace#start_span failed
[SKYLIGHT] [0.6.1] native Trace#start_span failed
[SKYLIGHT] [0.6.1] native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed
[SKYLIGHT] [0.6.1] failed to instrument span; msg=native Trace#start_span failed

My application code looks like:

class Foo
  def trace(name, &block)
    Skylight.instrument(title: name) do
      yield
    end
  end

  def bar
    trace('do something') { do_something }
  end
end

What am I doing wrong?

jeandenis-k commented 8 years ago

Same problem here, using Rails 3.2 as well (3.2.22 to be precise).

airblade commented 8 years ago

I still have this problem. I've been on Skylight 0.10.2 for a while.

wagenet commented 8 years ago

This error happens when we've hit the limit for how many spans we instrument in a single request. The current limit is 1024, but we've talked about doubling it. I suspect this will happen soon. That said, we should definitely handle the failure case more elegantly. It's on our radar and we hope to devote some time to it soon.

thilonel commented 7 years ago

[SKYLIGHT] [1.0.0] failed to instrument span; msg=native Trace#start_span failed

Still around. Please fix it! :)

hojberg commented 7 years ago

@wagenet anyway to configure this at least? To up the max num spans

faizalzakaria commented 7 years ago

Do we have plan to fix this ?

wagenet commented 7 years ago

Unfortunately, we've been delayed on resolving this. I'll try to reprioritize.

wagenet commented 7 years ago

I just released version 1.4.0-beta which doubles the limit. Let me know if that helps at all. Depending on just how high your number of spans are it may not resolve it for you, but it's worth a try.

lookageek commented 7 years ago

Hitting the 2048 limit too. Ruby 2.3.1 and Rails 5.0 on Ubuntu 16.04

ERROR:skylight::c_api: failed to start tracing span; err="Exceeded maximum number of trace spans; MAX=2048"
[SKYLIGHT] internal error: failed to start tracing span

Can we configure this number? If yes how to even come up with a number that is specific to each app?

wagenet commented 7 years ago

@lookageek it's a number we've set on our end to avoid overloading our systems with a badly behaved endpoint. We'll probably up it again in the future but we need to make sure we can handle any extra load. What sort of work are you doing in the endpoint? I assume a complex loop?

piotrze commented 7 years ago

@wagenet We're hitting this limit too(app is quite big), does it mean when we hit this error the request will not be stored in skylight?

AndrewRayCode commented 6 years ago

Seeing this error too, what is going on? What are the downstream effects of getting this error in our app code?

wagenet commented 6 years ago

@AndrewRayCode no downstream effects for your app's operation. However, this may cause some data to not show up in your Skylight UI. Can you reach out to us via the in-app messenger or support@skylight.io?

@piotrze can you also reach out to our support if you're still having trouble?

AndrewRayCode commented 6 years ago

Does this mean that skylight won't report on significantly complex Ruby traces (which are the ones we probably want skylight on the most?)

wagenet commented 6 years ago

@AndrewRayCode we can best serve you via our standard support channels.