🦎 A multi-protocol edge & service proxy. Seamlessly interface web apps, IoT clients, & microservices to Apache Kafka® via declaratively defined, stateless APIs.
kafka-http binding handles mapping of kafka messages on a topic to a specific url endpoint
[ ] inject a consistent idempotency-key http header based on request hash if not present
[ ] inject http :scheme, :authority, and :path headers if not present
derived from url message key
[ ] requires tls to trust server certificates for all webhook subscribers, possibly just trustcacerts: true
[ ] requires tcp to support outbound connections to dynamic host name and port number
inject target address in stream metadata, much like dynamically discovered kafka broker hostnames
[ ] http client request security
request signing
injected authorization header via guard
[ ] retry logic for failed webhook requests
should pause delivery for that endpoint to maintain webhook event ordering
should not pause delivery for other endpoints, they should proceed in parallel
behaves as if multiple consumers over shared messages topic, filtered by consumer to give a virtual topic
each virtual topic is a stream of messages with the same key, from the same partition
consumer marks progress on the virtual topic, manage ack gaps to prevent redelivery of ack'd virtual topic messages
discover virtual topics for system topic consumer group member
adjusts dynamically during consumer group rebalance
consumer id implicitly derived from underlying consumer group id and message key
[ ] capture latest webhook response in correlated reply topic using same key as request message
used to verify behavior during setup
includes attempt number as kafka message header
give up when maximum retry attempts exceeded
[ ] expire messages using time-based retention policy on requests topic to prevent retries, (say) after 24h
Note: monitor via http metrics at http client binding, export via prometheus, etc - may require high cardinality dynamic metrics.
Note: resolution of micro service events to filtered webhook subscribers is applied before publishing subscriber url targeted messages to the kafka topic.
Note: receiving inbound webhooks is handled by http-kafka binding, may need minor additional work to support full inbound url as produced message key, or to restrict set of headers included with request message payload.
Note: broadcast scenarios where one message is sent to multiple webhook endpoints requires a late join of request with different urls, which is anticipated to have occurred before zilla observes the webhook request in the kafka requests topic.
This is a brilliant feature. It will be a real game-changer. It will allow to turn legacy products with REST APIs only into the modern event-driven architecture. Looking forward to it.
Kafka Webhooks mapping
kafka-http
binding handles mapping of kafka messages on a topic to a specificurl
endpointidempotency-key
http header based on request hash if not present:scheme
,:authority
, and:path
headers if not presenturl
message keytls
to trust server certificates for all webhook subscribers, possibly justtrustcacerts: true
tcp
to support outbound connections to dynamic host name and port numberkey
, from the same partition24h
Note: monitor via http metrics at http client binding, export via prometheus, etc - may require high cardinality dynamic metrics.
Note: resolution of micro service events to filtered webhook subscribers is applied before publishing subscriber
url
targeted messages to the kafka topic.Note: receiving inbound webhooks is handled by
http-kafka
binding, may need minor additional work to support full inboundurl
as produced message key, or to restrict set of headers included with request message payload.Note: broadcast scenarios where one message is sent to multiple webhook endpoints requires a late join of request with different urls, which is anticipated to have occurred before zilla observes the webhook request in the kafka requests topic.