Knots with circuit breaker and Hystix dashboard

[tomaszmichalak]

The problem The timeout during a Knot processing means that something bad happened in Knot's logic - it can be time-consuming operation or services that do not respond in time.

Solution Fragment processing is an example of a producer-consumer problem, where Knot Engine is the message producer and Knots are consumers. So our problem arises when the producer sends messages faster than the consumer can consume (consume = respond). From the other hand, our consumers (Knots) are event loop handlers and delegate all blocking operations. They are very efficient.

That situation can lead to cascade failures in which Knot.x is able to make so many calls to the specific service that it is unable to return to a stable state.

The solution is a circuit breaker that temporary bans Knots providing fallback response.

The Hystrix dashboard can be used to monitor the state of circuit breakers.

Other options They can be modeled also as a stream of fragments events that are emitted via Knot Engine (Observable) and use the backpressure mechanism. This approach

References With https://github.com/Knotx/knotx-knot-engine/issues/3 we would be able to define timeouts and other options connected with Knot invocations.

[tomaszmichalak]

This is the proposal solution:

Overview

Knot

A function that converts one FragmentEvent into another FragmentEvent and provides a transition. The transition is the named graph edge, which makes the processing graph oriented.

Configuration

Operations

An operation connects business logic defined in Knots with configurable invocation behaviour. It works like an around advice in Aspect Oriented Programming allowing to apply such features like a circuit breaker or caching mechanism in a configurable and extendable manner. In addition, a Knot invocation is also configurable, so it does not matter if Knot listens on Vert.x Event Bus or is available in the classpath.

Note: Please note that operations can be stateful (request scope caching, circuit breaker). All operations are created via Factories once and are cached.

Note: The Knots interface can be converted to the java.util.function.Function<FragmentEventContext, Single<FragmentEventResult>> function. It allows you to implement your logic inside factories like, such as the inplaceBody factory defined in the examples below.

Knots parameters

    proxyOptions {
      # creates proxy that communicates via Event Bus
      factory = eb
      options {
        address = knotx.knot.databridge
      }
      configuration {
        books {
          params.path = "/api/v2/books"
        }
      }
    }

Fragments

Fragments section defines how a particular fragment should be evaluated. The definition is in the form of graph. A graph node defines the business logic (operation) to be applied to the fragment and transitions (graph edges) allowing to react to various operation results.

The default transitions are:

• next when operation ends successfully
• error when operation fails

Use cases

Page with one dynamic component

A configured operations looks like:

operations {
  # operation name
  getUserData {
    # operation factory name, operation comes with specific behaviour
    factory = circuitBreaker
    # operation options
    options {
      circuitBreakerOptions {
        timeout = 1000
      }
    }
    # business logic invocation options
    proxyOptions {
      # creates proxy that communicates via Event Bus
      factory = eb
      options {
        address = knotx.knot.user
      }
    }

  }
  handleUserDataError {
    proxyOptions {
      factory = inplaceBody
      options {
        # replace fragment body with custom markup
        body = "<div><p>The service is not available, please try again later</p></div>"
      }
    }
  }
  exchangeBusinessLogicWithFallback {
    factory = default
    proxyOptions {
      factory = classpath
      options {
        # delivers alternatice data 
        class = com.myproject.user.AlternativeUserDataServiceKnot
      }
    }
  }
  applyDataToBodyTemplate {
    proxyOptions {
      factory = eb
      options {
        address = knotx.knot.te
        deliveryOptions {
          sendTimeout = 1000
        }
      }
    }
  }
  hideFragment {
    proxyOptions {
      factory = inplaceBody
      options {
        # replace fragment body with empty markup
        body = ""
      }
    }
  }
}

Then we are able to define a named fragment processing graph:

fragments {
  userBox {
    operation = getUserData
    onTransition {
      # generic      
      next {
        operation = applyDataToBodyTemplate
        onTransition {
          error {
            operation = hideFragment
          }
        }
      }
      error {
        operation = handleUserDataError
      }
      # circuit breaker transition
      fallback {
        operation = exchangeBusinessLogicWithFallback
        onTransition {
          next {
            operation = applyDataToBodyTemplate
            onTransition {
              error {
                operation = handleUserDataError
              }
            }
          } 
        }
      }
    }
  }
}

And finally, the userBox fragment definition is used in the fragment:

<html>
  <knotx:snippet fragment="userBox">
  </knotx:snippet>
</html>

Product details page

Product details pages display data about a particular product. Those pages are fully managed by business so all items such as product name, product description, product details can not be provided as a single Knot.x fragment. This means that we would have many fragments using the same data.

This use case shows how we can provide request scope cache for all fragments.

Let's define operations first:

operations {
  getProductData {
    # here is a cache implementation
    factory = requestScopeCacheWithCircuitBreaker
    proxyOptions {
      factory = eb
      options {
        address = knotx.knot.product
      }
    }
    options {
      circuitBreakerOptions {
        timeout = 1000
      }
      cacheOptions {
        payload = REGEXP
      }
    }
  }
  applyDataToBodyTemplate {
    # SOME CONFIGURATION HERE
  }
  handleNoProductName {
    # SOME CONFIGURATION HERE
  }
  handleNoProductDetails {
    # SOME CONFIGURATION HERE
  }
  getAlternativeData {
    # SOME CONFIGURATION HERE
  }
  handleWithFallback {
    # SOME CONFIGURATION HERE
  }
}

Now we can define two fragment processing graphs (productName and productDetails):

fragments {
  productName {
    operation = getProductData
    onTransition {
      # generic
      next {
        operation = applyDataToBodyTemplate
      }
      error {
        operation = getAlternativeData
        onTransition {
          next {
            operation = applyDataToBodyTemplate
          }
          error {
            operation = handleNoProductName
          }
        }
      }
      # custom transition returned by Knot
      noProduct {
        operation = handleNoProductName
      }
      # circuit breaker behaviour
      fallback {
        operation = handleWithFallback
        onTransition {
          next {
            operation = applyDataToBodyTemplate
          }
        }
      }
    }
  }
  productDetails {
    operation = getProductData
    onTransition {
      # generic
      next {
        operation = applyDataToBodyTemplate
      }
      error {
        operation = getAlternativeData
        onTransition {
          next {
            operation = applyDataToBodyTemplate
          }
        }
      }
      # custom transition returned by Knot
      noProduct {
        operation = handleNoProductDetails
      }
      # circuit breaker behaviour
      fallback {
        operation = handleWithFallback
        onTransition {
          next {
            operation = applyDataToBodyTemplate
          }
        }
      }
    }
  }
}

And finally, fragments markup:

<html>
  <knotx:snippet fragment="productName">
  </knotx:snippet>
  some static content
  <knotx:snippet fragment="productDetails">
  </knotx:snippet>
</html>

[tomaszmichalak]

This should be placed in the documentation. In addition, the Knot.x HTTP Server parameter should be specified.