Some questions about connecting to kafka clients and performance issues using lua-resty-kafka

[PaperTiger7]

Hello, I have some questions when using lua-resty-lua. I wanted to avoid having to re-connect to kafka client every time a request came into Lua code, so I used something like singleton mode to make a kafka client connection and then use the same connection to send messages to kafka. I'd like to ask if this is necessary? Here's connection code.

KafkaUtils = {}
local mt = { __index = KafkaUtils }
-- 基础类方法 new
function KafkaUtils.new()
    local broker_list = {
        {host = "172.X.X.1", port = 9092},
        {host = "172.X.X.2", port = 9092},
        {host = "172.X.X.3", port = 9092}
    }
    kafkaProducer = producer:new(broker_list, { producer_type = "async", refresh_interval = 10000, required_acks = 0})
    return setmetatable({kafkaProducer = kafkaProducer}, mt)
end
--- 仿单例模式
function KafkaUtils:Instance()
    if self.instance == nil then
        self.instance = self:new()
    end
    return self.instance
end
--- 生产者发送消息
function KafkaUtils:sendMsg(topic, message)
    local kafkaProducer = self.kafkaProducer;
    local ok, err = kafkaProducer:send(topic, nil, message);
    if not ok then
        ngx.log(ngx.ERR, 'kafka send err:', err)
    end
end
return KafkaUtils;

Here's code for sending massage.

--- 发送消息
KafkaUtils:Instance():sendMsg(topic, massage);

I currently have 12 servers using openrsty for message production and 3 servers for kafka clusters and 9 partitions per topic. After testing, when the concurrency is around 4000 connections, only 500 messages can be sent per second, and each message is only 30 characters long. I would like to ask how to configure to increase the number of messages sent? Looking forward to your reply.

[doujiang24]

only 500 messages can be sent per second

this is too low, it does not make sense. over 1k~10k message per second per nginx worker should be resonable.

does the openresty and kafka server are distribution in different regions? they have very high network latency?

[PaperTiger7]

only 500 messages can be sent per second

this is too low, it does not make sense. over 1k~10k message per second per nginx worker should be resonable.

does the openresty and kafka server are distribution in different regions? they have very high network latency?

Thank you for your reply. the openresty and kafka server are distribution in same regions.