search

instana / ruby-sensor

💎 Ruby Distributed Tracing & Metrics Sensor for Instana
https://www.instana.com/
MIT License
26 stars 25 forks source link

[Bug]: HostAgentActivationObserver chokes on unexpected HostAgent response and stops retrying #267

Closed breunigs closed 2 years ago

breunigs commented 2 years ago

Problem Description

This stack trace:

App 120 output: I, [2022-03-25T11:47:29.709278 #120]  INFO -- : Proces `77` has forked into 120. Running post fork hook.
App 120 output: E, [2022-03-25T11:47:44.598462 #120] ERROR -- : no implicit conversion of String into Integer
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_activation_observer.rb:38:in `[]'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_activation_observer.rb:38:in `block in update'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_activation_observer.rb:83:in `try_forever_with_backoff'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_activation_observer.rb:30:in `update'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102:in `block in notify_to'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100:in `each'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100:in `notify_to'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64:in `notify_observers'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/atom.rb:183:in `compare_and_set'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/atom.rb:165:in `block in swap'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/atom.rb:160:in `loop'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/atom.rb:160:in `swap'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_reporting_observer.rb:68:in `block in report_traces'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/tracing/processor.rb:82:in `each'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/tracing/processor.rb:82:in `each_slice'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/tracing/processor.rb:82:in `send'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_reporting_observer.rb:63:in `report_traces'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_reporting_observer.rb:38:in `report_to_backend'
App 120 output: /app/gems/gems/instana-1.209.0.pre1/lib/instana/backend/host_agent_reporting_observer.rb:22:in `block in initialize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19:in `execute'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:312:in `execute_task'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19:in `execute'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169:in `safe_execute'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285:in `process_task'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165:in `block (2 levels) in process_tasks'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:363:in `run_task'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `block (3 levels) in create_worker'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `loop'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `block (2 levels) in create_worker'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `catch'
App 120 output: /app/gems/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block in create_worker'
[ E 2022-03-25 11:48:59.8438 20/T17 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app: A timeout occurred while spawning an application process.

Tracing unsurprisingly doesn't work when this occurs, since this is not a scenario for which try_forever_with_backoff retries.

Looking at the agent, it has an error for the pod in question, but also unrelated errors that make the agent itself look quite unhealthy:

2022-03-25T10:47:29.758+00:00 | INFO  | instana-http-client-thread-1     | nProcessUtilImpl | com.instana.agent - 1.1.626 | Request to lookup remote process PID for address: 10.142.10.146:36366
2022-03-25T10:47:29.760+00:00 | INFO  | instana-http-client-thread-1     | nProcessUtilImpl | com.instana.agent - 1.1.626 | Remote process socket opened by address '10.142.10.146' which appears not to be on this host
2022-03-25T10:47:29.876+00:00 | INFO  | instana-http-client-thread-3     | nProcessUtilImpl | com.instana.agent - 1.1.626 | Request to lookup remote process PID for address: 10.142.10.146:36374
2022-03-25T10:47:29.877+00:00 | INFO  | instana-http-client-thread-3     | nProcessUtilImpl | com.instana.agent - 1.1.626 | Remote process socket opened by address '10.142.10.146' which appears not to be on this host
2022-03-25T10:47:30.142+00:00 | ERROR | instana-http-client-thread-3     | Ruby             | com.instana.discovery-ruby - 1.6.3 | Could not find the process RubyProcess{pid='250', pidFromParentNS='false', name='Passenger', arguments=[RubyApp:, /app, (preview)], cpuSetFileContent='/kubepods/burstable/poded5266f9-0c79-4d75-ac70-1e7477bcc437/0ff6aa49bbfa26e919011f17f855e571874724c1c4332c63d43347d2f19748b8', fd='9', inode='socket:[2449397783]'}.
2022-03-25T10:47:44.418+00:00 | INFO  | nstana-sensor-scheduler-thread-4 | turesManagerImpl | com.instana.agent-bootstrap - 1.2.16 | Installed instana-ruby-trace-sensor
2022-03-25T10:47:44.549+00:00 | INFO  | d3a-2711-42d4-a45c-a71d6617e174) | RubyTrace        | com.instana.sensor-ruby-trace - 1.2.1 | Activated Sensor
2022-03-25T10:47:44.646+00:00 | INFO  | ca2-3496-47c9-bea6-01077d7ddc48) | Process          | com.instana.sensor-process - 1.1.45 | Activated Sensor for PID 2426753
2022-03-25T10:47:44.721+00:00 | ERROR | instana-http-client-thread-1     | Ruby             | com.instana.discovery-ruby - 1.6.3 | Could not find the process RubyProcess{pid='120', pidFromParentNS='false', name='Passenger', arguments=[RubyApp:, /app, (preview)], cpuSetFileContent='/kubepods/burstable/poded5266f9-0c79-4d75-ac70-1e7477bcc437/0ff6aa49bbfa26e919011f17f855e571874724c1c4332c63d43347d2f19748b8', fd='23', inode='socket:[2449446067]'}.
2022-03-25T10:47:44.742+00:00 | INFO  | 89a-b9f6-477a-b6f3-445a14e0ffc0) | Java             | com.instana.sensor-java - 1.2.113 | Activated Sensor for JVM 2435270
2022-03-25T10:47:44.771+00:00 | INFO  | nstana.plugin.java.2435270attach | Java             | com.instana.sensor-java - 1.2.113 | Attaching to JVM with PID 2435270
2022-03-25T10:47:44.777+00:00 | INFO  | nstana.plugin.java.2435270attach | aagentLoaderImpl | com.instana.agent - 1.1.626 | Performing initial attach to JVM with PID 2435270
2022-03-25T10:47:44.796+00:00 | INFO  | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | Attaching to JVM 8 in container 57a074a187efee32e821ceb1a7e91a78b140b8271e22839fe67f6394879f6ac9, which is 2435270 on host.
2022-03-25T10:47:44.816+00:00 | INFO  | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | Executing: ctr --timeout 60s -n k8s.io task exec --exec-id instanaAttach1648205264815 57a074a187efee32e821ceb1a7e91a78b140b8271e22839fe67f6394879f6ac9 /opt/jdk1.8.0_201/bin/java -Xms16m -Xmx16m -Xint -XX:+UseSerialGC -XX:+PerfDisableSharedMem -Duser.dir=/ -cp /tmp/attach-tools-1.10.3.jar:/tmp/javaagent-loader-1.3.38.jar com.instana.agent.loader.AgentLoaderAttach agent=/tmp/javaagent-loader-1.3.38.jar verify=true pid=8 host=10.12.132.30 port=42699 boot=/com.instana.agent.artifact/com.instana/instrumentation-boot/1.1.26/jar shared=/com.instana.agent.artifact/com.instana/instrumentation-shared/1.0.135/jar hostpid=2435270
2022-03-25T10:47:44.862+00:00 | INFO  | e9b-a2fb-4093-ab72-b5c3f7929250) | Process          | com.instana.sensor-process - 1.1.45 | Activated Sensor for PID 2426753
2022-03-25T10:47:44.937+00:00 | INFO  | f11-b99b-4e75-ac77-cdc94e2caa45) | Process          | com.instana.sensor-process - 1.1.45 | Activated Sensor for PID 2426753
2022-03-25T10:47:46.738+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | Failed attaching loader to JVM ContainerizedVirtualMachineImpl [pid=2435270, inContainerPid=8, containerFileSystem=/proc/2435270/rootparentPid=2435270, attachType=HOTSPOT_TOOLS, hasAttachFile=false, commandLine=org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint --configDir /opt/flink/conf --executionMode cluster, vmArgs=-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml, name=Java HotSpot(TM) 64-Bit Server VM, vendor=Oracle Corporation, version=1.8.0_201, build=25.201-b09, heapCapacity=-1, process=ContainerdProcessImpl [pid=2435270, parentPid=<lazy>, inContainerPid=8, containerId=57a074a187efee32e821ceb1a7e91a78b140b8271e22839fe67f6394879f6ac9, namespace=k8s.io, name=<some java options here that I excluded because these might be internal and are not relevant for this ticket> cluster], userName=<lazy>, groupName=<lazy>]] with code 52: Instana Agent is unable to connect to the attach socket of the target JVM, please try restarting the JVM
2022-03-25T10:47:46.739+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | java.lang.IllegalStateException: Target VM did not respond: 8
2022-03-25T10:47:46.740+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at net.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:243)
2022-03-25T10:47:46.740+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at net.bytebuddy.agent.VirtualMachine$ForHotSpot.attach(VirtualMachine.java:256)
2022-03-25T10:47:46.740+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at net.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$Factory$ForSocketFile.connect(VirtualMachine.java:541)
2022-03-25T10:47:46.741+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-03-25T10:47:46.741+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-03-25T10:47:46.742+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-03-25T10:47:46.743+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at com.instana.agent.loader.AgentLoaderAttach.attach(AgentLoaderAttach.java:364)
2022-03-25T10:47:46.743+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at java.lang.reflect.Method.invoke(Method.java:498)
2022-03-25T10:47:46.744+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at com.instana.agent.loader.AgentLoaderAttach.parseArgsAndRun(AgentLoaderAttach.java:90)
2022-03-25T10:47:46.744+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at com.instana.agent.loader.AgentLoaderAttach.run(AgentLoaderAttach.java:124)
2022-03-25T10:47:46.745+00:00 | ERROR | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 |     at com.instana.agent.loader.AgentLoaderAttach.main(AgentLoaderAttach.java:73)
2022-03-25T10:47:46.745+00:00 | INFO  | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | Cannot use in container attach for JVM 8 in container 57a074a187efee32e821ceb1a7e91a78b140b8271e22839fe67f6394879f6ac9, which is 2435270 on host, due to previous failure. May attempt recovery via direct attach.
2022-03-25T10:47:46.848+00:00 | INFO  | nstana.plugin.java.2435270attach | AttachHelperImpl | com.instana.agent - 1.1.626 | Agent user {Uid=0, Gid=0} and target JVM user {Uid=9999, Gid=9999} not identical.

10.142.10.146 is the IP of the Kubernetes pod where the initial stack trace is from.

The setup for that app is PhusionPassenger on nginx with smart forking, no Instana AutoTrace™, but this manual integration:

# Gemfile
…
gem 'instana', ">= 1.209.0.pre1"
# config/warmup.rb
PhusionPassenger.on_event(:starting_worker_process) do
  ::Instana.agent.after_fork unless ENV.key?('INSTANA_DISABLE')

  client = Rack::MockRequest.new(Rails.application)
  client.get('warmup')
end

INSTANA_AGENT_HOST gets set through k8s downward API.

Other pods with the same setup work fine, only the ones going to the misbehaving agent were experiencing the issue.

I would expect that the sensor retries indefinitely, even if the agent sends garbage. Killing the instana-agent pod made me unable to reproduce the issue so far, but the Ruby app pod would stay broken of course.

Minimal, Complete, Verifiable, Example

Your terms of conditions disallow me from reverse engineering the Instana Host Agent, so I can't provide one. Unfortunately debug logging was not enabled for the ruby-sensor at the time, so I can't tell you what value was set in discovery:

          discovery = discovery_response.json
          @logger.debug("Discovery complete (`#{discovery}`). Waiting for agent.")
          wait_for_backend(discovery['pid'])

My best guess is that it was an integer or some such:

[2] pry(main)> discovery=42
=> 42
[3] pry(main)> discovery['pid']
TypeError: no implicit conversion of String into Integer

Gemfile.lock

# please note that I had to redact some parts
GEM
  remote: REDACTED
  specs:
    actioncable (7.0.2.3)
      actionpack (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
    actionmailbox (7.0.2.3)
      actionpack (= 7.0.2.3)
      activejob (= 7.0.2.3)
      activerecord (= 7.0.2.3)
      activestorage (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      mail (>= 2.7.1)
      net-imap
      net-pop
      net-smtp
    actionmailer (7.0.2.3)
      actionpack (= 7.0.2.3)
      actionview (= 7.0.2.3)
      activejob (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      mail (~> 2.5, >= 2.5.4)
      net-imap
      net-pop
      net-smtp
      rails-dom-testing (~> 2.0)
    actionpack (7.0.2.3)
      actionview (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      rack (~> 2.0, >= 2.2.0)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.2.0)
    actiontext (7.0.2.3)
      actionpack (= 7.0.2.3)
      activerecord (= 7.0.2.3)
      activestorage (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      globalid (>= 0.6.0)
      nokogiri (>= 1.8.5)
    actionview (7.0.2.3)
      activesupport (= 7.0.2.3)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.1, >= 1.2.0)
    activejob (7.0.2.3)
      activesupport (= 7.0.2.3)
      globalid (>= 0.3.6)
    activemodel (7.0.2.3)
      activesupport (= 7.0.2.3)
    activerecord (7.0.2.3)
      activemodel (= 7.0.2.3)
      activesupport (= 7.0.2.3)
    activestorage (7.0.2.3)
      actionpack (= 7.0.2.3)
      activejob (= 7.0.2.3)
      activerecord (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      marcel (~> 1.0)
      mini_mime (>= 1.1.0)
    activesupport (7.0.2.3)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 1.6, < 2)
      minitest (>= 5.1)
      tzinfo (~> 2.0)
    builder (3.2.4)
    coderay (1.1.3)
    concurrent-ruby (1.1.9)
    crass (1.0.6)
    diff-lcs (1.5.0)
    digest (3.1.0)
    docile (1.4.0)
    dotenv (2.7.6)
    dotenv-rails (2.7.6)
      dotenv (= 2.7.6)
      railties (>= 3.2)
    erubi (1.10.0)
    ffi (1.15.5)
    ffi-rzmq (2.0.7)
      ffi-rzmq-core (>= 1.0.7)
    ffi-rzmq-core (1.0.7)
      ffi
    globalid (1.0.0)
      activesupport (>= 5.0)
    i18n (1.10.0)
      concurrent-ruby (~> 1.0)
    instana (1.209.0.pre1)
      concurrent-ruby (>= 1.1)
      oj (>= 3.0.11)
      sys-proctable (>= 1.2.2)
    io-wait (0.2.1)
    logjam_agent (0.33.0)
      activesupport
      ffi-rzmq (>= 2.0.4)
      ffi-rzmq-core (>= 1.0.5)
      REDACTED (>= 0.12.2)
    loofah (2.15.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    marcel (1.0.2)
    method_source (1.0.0)
    mini_mime (1.1.2)
    mini_portile2 (2.8.0)
    minitest (5.15.0)
    net-imap (0.2.3)
      digest
      net-protocol
      strscan
    net-pop (0.1.1)
      digest
      net-protocol
      timeout
    net-protocol (0.1.2)
      io-wait
      timeout
    net-smtp (0.3.1)
      digest
      net-protocol
      timeout
    nio4r (2.5.8)
    nokogiri (1.13.3)
      mini_portile2 (~> 2.8.0)
      racc (~> 1.4)
    oj (3.13.11)
    pry (0.14.1)
      coderay (~> 1.1)
      method_source (~> 1.0)
    racc (1.6.0)
    rack (2.2.3)
    rack-test (1.1.0)
      rack (>= 1.0, < 3)
    rails (7.0.2.3)
      actioncable (= 7.0.2.3)
      actionmailbox (= 7.0.2.3)
      actionmailer (= 7.0.2.3)
      actionpack (= 7.0.2.3)
      actiontext (= 7.0.2.3)
      actionview (= 7.0.2.3)
      activejob (= 7.0.2.3)
      activemodel (= 7.0.2.3)
      activerecord (= 7.0.2.3)
      activestorage (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      bundler (>= 1.15.0)
      railties (= 7.0.2.3)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.4.2)
      loofah (~> 2.3)
    railties (7.0.2.3)
      actionpack (= 7.0.2.3)
      activesupport (= 7.0.2.3)
      method_source
      rake (>= 12.2)
      thor (~> 1.0)
      zeitwerk (~> 2.5)
    rake (13.0.6)
    rspec-core (3.11.0)
      rspec-support (~> 3.11.0)
    rspec-expectations (3.11.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.11.0)
    rspec-mocks (3.11.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.11.0)
    rspec-rails (5.1.1)
      actionpack (>= 5.2)
      activesupport (>= 5.2)
      railties (>= 5.2)
      rspec-core (~> 3.10)
      rspec-expectations (~> 3.10)
      rspec-mocks (~> 3.10)
      rspec-support (~> 3.10)
    rspec-support (3.11.0)
    simplecov (0.21.2)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-html (0.12.3)
    simplecov_json_formatter (0.1.4)
    snappy (0.2.0)
    strscan (3.0.1)
    sys-proctable (1.2.6)
      ffi
    thor (1.2.1)
    thread_variables (0.2.0)
    REDACTED (0.12.5)
      activesupport (>= 5.2.0)
      thread_variables
    timeout (0.2.0)
    tzinfo (2.0.4)
      concurrent-ruby (~> 1.0)
    websocket-driver (0.7.5)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)
    REDACTED (1.5.3)
      activesupport (>= 5.2)
      ffi-rzmq (~> 2.0.7)
      ffi-rzmq-core (~> 1.0.7)
      logjam_agent (>= 0.32.2)
      railties (>= 5.2)
      snappy (~> 0.2.0)
      REDACTED (>= 0.12.3)
    zeitwerk (2.5.4)

PLATFORMS
  ruby

DEPENDENCIES
  dotenv-rails
  instana (>= 1.209.0.pre1)
  pry
  rails (~> 7.0)
  rspec-rails
  simplecov
  REDACTED

BUNDLED WITH
   2.3.4

Ruby Version

ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [x86_64-linux]
hmadison commented 2 years ago

Hi @breunigs,

I've opened #269 to resolve this issue. It should be out later today. If this issue persists, can you open a ticket with support@instana.com directly? When I've seen this issue come up for other customers, its generally related to something at the agent deployment layer.

Thanks, Hunter