I've been doing some work with event processing using a reactor. The reactor's job is to react to an event and then call a downstream system (marketplace) to trigger some behaviour. I need to ensure the call is successfully made and as such raise an error in the reactor if the downstream system is down or receives an unexpected response. The built in retry functionality inside ESPRunner will work for this purpose.
https://github.cm/envato/event_sourcery/blob/master/lib/event_sourcery/event_processing/esp_runner.rb#L38
Upon testing my reactor out in development and specifically how it behaves when the downstream system is not available. I found that instead of continually retrying and receiving the same error, I would eventually begin to receive a timeout from Sequel trying to obtain a database connection.
Processor author_enroller died with timeout: 5.0, elapsed: 5.001521. /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/sequel-4.42.1/lib/sequel/connection_pool/threaded.rb:264:in `raise_pool_timeout'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/sequel-4.42.1/lib/sequel/connection_pool/threaded.rb:167:in `block in acquire'
I initially thought this was purely an issue with not enough database connections being available, but it is caused by additional poll waiter threads being created when the ESPRunner retries and tells the event processor to subscribe to the event store again which creates another Poll Waiter and Subscription. This is fine, however the previous Thread still remains and sits there holding an idle db connection listening to new events. This will repeat until all db connections have been consumed and then the Sequel timeout will occur.
Not sure if there is a better way to handle this and I am a bit of a noob when it comes to Threading.
I've been doing some work with event processing using a reactor. The reactor's job is to react to an event and then call a downstream system (marketplace) to trigger some behaviour. I need to ensure the call is successfully made and as such raise an error in the reactor if the downstream system is down or receives an unexpected response. The built in retry functionality inside ESPRunner will work for this purpose. https://github.cm/envato/event_sourcery/blob/master/lib/event_sourcery/event_processing/esp_runner.rb#L38
Upon testing my reactor out in development and specifically how it behaves when the downstream system is not available. I found that instead of continually retrying and receiving the same error, I would eventually begin to receive a timeout from Sequel trying to obtain a database connection.
I initially thought this was purely an issue with not enough database connections being available, but it is caused by additional poll waiter threads being created when the ESPRunner retries and tells the event processor to subscribe to the event store again which creates another Poll Waiter and Subscription. This is fine, however the previous Thread still remains and sits there holding an idle db connection listening to new events. This will repeat until all db connections have been consumed and then the Sequel timeout will occur.
Not sure if there is a better way to handle this and I am a bit of a noob when it comes to Threading.