search

PennState / jms-tools

Apache License 2.0
3 stars 4 forks source link

Failures in sub-type MessageProcessor constructor calls cause open loop retries #10

Open smoyer64 opened 6 years ago

smoyer64 commented 6 years ago

When the MessageHandler attempts to spawn a new MessageProcessor sub-type, if there's an error in the constructor the MessageHandler attempts to build another without rate limiting. There are two types of errors that classes implementing MessageProcessor should differentiate between:

  1. Errors due to misconfiguration shouldn't include retries as they'll never succeed. In this case the MessageHandler should itself exit with an error.
  2. Errors due to temporary loss of connectivity or other transient problems may eventually resolve. in this case, the MessageHandler should "gracefully" retry. Exponential backoff would be nice but even a timer to limit the rate at which the MessageHandler tries to instantiate a MessageProcessor would work.

An example of the open loop behavior is as follows:

20:00:56.732 [] [main] INFO  edu.psu.activemq.MessageHandler - Processed: 0
20:00:56.732 [] [main] TRACE edu.psu.activemq.MessageHandler - Checking thresholds, count = 178 threshold = 10
20:00:56.732 [] [main] TRACE edu.psu.activemq.MessageHandler - Constructing a new Message Processor
20:00:56.732 [] [main] ERROR edu.psu.activemq.MessageHandler - Error in message handler
java.lang.InstantiationException: null
        at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at edu.psu.activemq.MessageHandler.buildNewMessageProcessor(MessageHandler.java:246)
        at edu.psu.activemq.MessageHandler.startMonitor(MessageHandler.java:198)
        at edu.psu.activemq.MessageHandler.init(MessageHandler.java:96)
        at edu.psu.activemq.MessageHandlerFactory.build(MessageHandlerFactory.java:66)
        at edu.psu.service.academic.o365.Main.start(Main.java:29)
        at edu.psu.service.academic.o365.Main.main(Main.java:14)
20:00:56.747 [] [main] INFO  edu.psu.activemq.MessageHandler - Processed: 0
20:00:56.747 [] [main] TRACE edu.psu.activemq.MessageHandler - Checking thresholds, count = 178 threshold = 10
20:00:56.747 [] [main] TRACE edu.psu.activemq.MessageHandler - Constructing a new Message Processor
20:00:56.747 [] [main] ERROR edu.psu.activemq.MessageHandler - Error in message handler
java.lang.InstantiationException: null
        at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at edu.psu.activemq.MessageHandler.buildNewMessageProcessor(MessageHandler.java:246)
        at edu.psu.activemq.MessageHandler.startMonitor(MessageHandler.java:198)
        at edu.psu.activemq.MessageHandler.init(MessageHandler.java:96)
        at edu.psu.activemq.MessageHandlerFactory.build(MessageHandlerFactory.java:66)
        at edu.psu.service.academic.o365.Main.start(Main.java:29)
        at edu.psu.service.academic.o365.Main.main(Main.java:14)
smoyer64 commented 6 years ago

Shawn has proposed that:

  1. When RuntimeException is thrown, sub-type MessageProcessor should log the specifics of the error and the MessageHandler should exit .
  2. When a checked exception is thrown, the sub-type MessageProcessor should log the specifics of the temporary problem and retry.