Trigger function execution from Kafka Event

[lucasjellema]

Functions can ideally be triggered by more than just HTTP requests. Especially Events (and even more especially Kafka Events on selected Topics) would be a desirable trigger for function execution (just as Azure Functions for example supports).

I understand I can create a Kafka Client, have it consume events and in turn make an HTTP invocation to a serverless function. (a very un-serverless approach).

I would like for the Fn platform to offer more support for this - provide a connector between a Kafka Topic and Function invocation.

This connector is configured with:

• name of Kafka topic
• connect details to Kafka Server
• optionally name of Kafka consumer group
• possibly filter condition on message
• routing to function that should be triggered by the event
• possibly a limit on the number of concurrent events to be handled per partition (only if set to 1 can we guarantee that events are picked up in order delivered)
• setting: is event handled as soon as function is invoked or (preferably) only when function returns a success status (and is the event regarded as unprocessed if the function does not return a timely success status)

I think we can initially assume that the event contains a JSON message payload. at a later stage, we probably want to support different types of payload as well (AVRO, XML, plain text). In the configuration of the connector, we should then specify how the connector is to take the event's payload and pass it to the function.

When the Kafka Topic makes an event available to the connector-client, the payload is taken from the event and passed into the configured function. When the function completes successfully, the event is considered processed for this connector.

Note: Multiple connectors can be created for the same event topic - each triggering a different function.

[and1truong]

This very cool features, but can you guys consider NATS-streaming, it's a CNCF member.