eigengo / monitor

Library for monitoring the Typesafe stack-based applications.
Apache License 2.0
127 stars 24 forks source link

System initialization error when trying to send InitializeLogger message to the default logger #91

Open vonnagy opened 11 years ago

vonnagy commented 11 years ago

It appears as though the DefaultLogger does not reply to InitializeLogger on line 200 of the LoggingBus.

import akka.pattern.ask

val response = try Await.result(actor ? InitializeLogger(this), timeout.duration) catch {
      case _: TimeoutException ⇒
        publish(Warning(logName, this.getClass, "Logger " + name + " did not respond      within " + timeout + " to InitializeLogger(bus)"))
        "[TIMEOUT]"
    }

This issue is due to the actor system 'statsd' being created in the class AkkaIOStatsdCounterInterface. The issue only occurs when system agents are being monitored e.g. includeSystemAgents: true.

Here is the scenario:

As an experiment to prove this out, I first changed the creation of the 'statsd' system to lazy instantiation see #90. I then wrapped the logic in theCounterInterface methods in a Future. This disconnected the logic from the running thread which, in turn, allowed all messages to be received properly.

The test above is not necessarily a good long-term solution, but does point out the issue and possible ideas to its resolution.

janm399 commented 11 years ago

Do you think we can close it now? I think we'll soon come to a time when we'll all need to have a hangout and discuss any refactorings we may want to apply.

vonnagy commented 11 years ago

This is still an issue when includeSystemAgents is turned on and someone is using AkkaIOStatsdCounterInterface. As we know it is a threading issue because the application's actor system is starting it's logger at the same time the 'statsd' is being doing the same. Setting the logger-startup-timeout has no effect as the issue stems from a threading conflict.

This would be a candidate for some further discussion on a possible refactor as I don't believe the use of Future as outlined above is the right direction. Perhaps and instance of the CounterInterface is created per actor system and the 'statsd' actor is created within that system.