search

flipkart-incubator / databuilderframework

A data driven execution engine
33 stars 29 forks source link

EnhancementRequest - ReactiveExecutor #10

Closed gokulvanan closed 5 years ago

gokulvanan commented 8 years ago

As our application of Databuilder primarily was used in Orchestrating downstream service calls we noticed while using Multithreaded executor, A lot of threads needed to be created for Databuilder as dowstream service in their on thread pools where blocking and builder was running out of threads in timed_wait. The situation worsens when builder starts blocking controller threads.

The idea here is to have databuilder threads recyclable and reusable, such that they could leverage IO hand off to respective httpClient pools.

DataBuilderExecutor would need to implement and Observable kinda of interface returning data when possible. Also Internal to builder each process method invoke is blocking. This should also be reactive.

santanusinha commented 8 years ago

Isn't hystrix chaining a better fit for this than using DataBuilders?

gokulvanan commented 8 years ago

Hystrix chaining would work but we will end up building a fixed tree of chains connecting Observable output of one to next hystrix command for an orchestration of service calls.

By this we will loose out all the benefits of having a modularity and configurable graph being built in Databuilder plus the traceability of finding out where something broke.

santanusinha commented 8 years ago

Ok .. understood.

This might need bigger design changes in the system. Let's discuss the design once before you get into the implementation.

Upset the established order, and everything becomes chaos. I'm an agent of chaos. Oh and you know the thing about chaos, it's fair.

On Wed, Oct 14, 2015 at 11:57 AM, gokulvanan notifications@github.com wrote:

Hystrix chaining would work but we will end up building a fixed tree of chains connecting Observable output of one to next hystrix command for an orchestration of service calls.

But this loose out all the benefits of having a modularity and configurable graph being built in Databuilder plus the traceability of finding out where something broke.

— Reply to this email directly or view it on GitHub https://github.com/flipkart-incubator/databuilderframework/issues/10#issuecomment-147950710 .

santanusinha commented 5 years ago

Closing this as no movement.