help request: Pubsub with Kafka ready for production ?

[arnauddeman]

Description

Hello, I would like to know the status of PubSub with Kafka implementation in Apisix. Is it an experimental feature or is it ready for production ? The underlying Kafka Client lua-resty-kafka is still flagged as experimental.

My second question is about the implementation of the pubsub pattern in Apisix : the new messages are not pushed to the client only the messages published before the PubSubQuery are. Is this the expected behaviour ?

Environment

APISIX docker version, image apache/apisix:3.6.0-debian

[shreemaan-abhishek]

Push will not occur (due to the complexity of Nginx workers and some long connections). The existing mode is req/resp, where the client tells the gateway from which offset position to start sending subsequent messages.

The input parameters of fetch message is topic/partition/offset, which will send the latest message starting from offset (if there are too many messages and exceed the maximum byte limit, the number of messages that meet the response body size will be returned).

After the client receives the response, it can traverse it to retrieve all messages, and in the last message, you will get the timestamp and offset of this message. The client can use them to retrieve newer messages in the next fetch request.

[arnauddeman]

Thanks for your answer, I am in the context of reactive programming, so I think that will not feet my needs. I also need to be able to filter the Kafka messages. Another experiment around this topic seems to be more successful : Apisix <------------------> API (Spring Kafka) <----------------> Kafka (Websocket
and OIDC plugin)

[juzhiyuan]

Hi @arnauddeman, I have checked with a few of the maintainers.

Pub/Sub is still experimental, requiring the client to use WebSocket rather than HTTP or Kafka Protocol.

I have also checked the issues you raised at https://github.com/apache/apisix/issues/created_by/arnauddeman, but I would like to understand your requirements and scenarios. Can you share the details? (what kind of problems do you want to resolve? e.g., proxy Kafka behind APISIX? or configuring OIDC with Kafka?).

This can help us better understand your needs and may provide a suggestion.

[arnauddeman]

Hi @juzhiyuan, Thanks for your answer. The objective is to implement a notification system and the first idea was to proxy Kafka behind Apisix with the OIDC plugin. This doesn't seem to be the good approach because we need the messages to be pushed and filtered.

As you suggested, I have tested the use of a web socket route, connected to a locally developed API which uses a Kafka client. The associated APISIX route uses the OIDC plugin and the local API can retrieve the token to determine the user and filter the Kafka messages.

[juzhiyuan]

Hi @arnauddeman,

I have checked with @bzp2010 and @moonming; we need some time to discuss this. Will keep you updated.