kaskada-ai / kaskada

Modern, open-source event-processing
https://kaskada.io/
Apache License 2.0
348 stars 15 forks source link

Implement HTTP request/response source #839

Open kerinin opened 10 months ago

kerinin commented 10 months ago

Idea is something along the lines of the code block shown below. Things to consider:

  1. Support both "read-only sources" (things which always respond 200 and are used in defining other responses)
  2. Support "read/write sources" (things which compute responses using kaskada)

It's OK if this only works with the new partitioned execution code paths (eg., using the Source / Destination traits).

# this produces `{ request_id: i64, data: <schema> }`
source = kd.sources.Http(
  schema = ...,
  # Optional response. This allows for requests that append data.
  # If not specified, then a later `run_and_respond` is necessary.
  respond_with = 200,
)

output = kd.record({
  request_id: source.request_id,
  x_plus_y = source.data.x + source.data.y,
)}

source.run_and_respond(
  # Use Kaskada to define responses.
  respond_with = output,
  # Report a warning if the output is not available
  # for a request within this timelimit.
  # this may happen if the request gets buffered in
  # a shift, etc.
  timelimit = 1s
)