Markdown test 2 - Githubissues

kaylareopelle commented 2 years ago

This is a dupe of an issue the newrelic-ruby-agent received from a customer, all info is for testing purposes only

Description

When running a short-lived process, the newrelic agent shutdown often adds 3 seconds to the total runtime.

Without newrelic:

$ time bin/rails runner "puts 'hi'"
hi

real    0m1.591s
user    0m0.907s
sys 0m0.399s

with newrelic:

$ time bin/rails runner "puts 'hi'"
hi

real    0m5.338s
user    0m0.991s
sys 0m0.558s

Expected Behavior

NewRelic doesn't stall for 3 seconds while trying to shut down the agent thread.

The problem occurs here - https://github.com/newrelic/newrelic-ruby-agent/blob/2516b9b8721c5dd4cb28a198bb726e4e992ca1ee/lib/new_relic/agent/agent/start_worker_thread.rb#L43-L51 - if @event_loop is still nil because we're still waiting for the connect in deferred_work to complete. (On my machine, connect takes around 1.1 seconds)

Not sure of a clean fix that completely avoids race conditions... maybe stop_event_loop needs to loop a few times?

      def stop_event_loop
        30.times do
          @event_loop.stop if @event_loop
          # Wait the end of the event loop thread.
          if @worker_thread
            if @worker_thread.join(0.1)
              return
            end
          end
        end

        ::NewRelic::Agent.logger.debug "Event loop thread did not stop within 3 seconds"
      end

pretty gross though, there must be a better way.

Impact

Low, those 3 seconds aren't going to kill me. It's just irritating when I run a script from terminal, see it finish, but then have to sit there waiting like an idiot for newrelic to fail to kill its worker thread.