Support async backend services [RENAMED]

[chadmott]

Scenario 1: When a request comes in over REST, we return a 202 no content, and the RIG responds asynchronously over Kafka. This works great for async events when a compatible consumer is available (like a reactive UI). We use this pattern when we have to communicate with a legacy system(s) over kafka.

Scenario 2: The RIG will also reply RESTfully (200) to microservices that can respond in that way. In our experience, these services have no downstream dependency.

Senario 3: Feature Request I want to do scenario 1, but I don't have a modern consumer. I have instead a 3rd party API consumer who is expecting RESTful responses without polling.

Today, solving scenario 3 is tricky:

• microservice gets request, and then registers a listener to kafka for the response service is now stateful
• microservice must be picky about the kafka partition it publishes/consumes b/c if there are mulitple, there is no guarantee that the response the MS is looking for service is now fragile
• I must duplicate this logic for any microservice that needs similar functionality

I see this as a natural fit for the RIG b/c it alraeady manages state, and can be the one place to hold these requests. We should be able to configure the timeout.

I would envision that you have some kind of correlation ID that is propagated throughout the system to make this work:

Request (with header indicating you are expecting a REStful response) -> RIG assigns ID -> systems propagate ID around the system -> publish to Kafka -> RIG sees correlation ID and responds to request with the RIG payload

[kevinbader]

I agree that this could be a useful addition, as it decouples the client from how the request is processed "internally".

My current approach is to allow a response_from field in the proxy endpoint configuration, e.g.:

{
  id: "myendpoint",
  type: "http",
  not_secured: true,
  method: "GET",
  path: "/myendpoint",
  response_from: "kafka"
}

With the following test cases:

• Given response_from is not set, the http response is forwarded as-is.
• Given response_from is set to a Kafka topic, the http response is taken from a Kafka topic instead of forwarding the backend's original response.

[kevinbader]

Closed by #93 and via #77