optimize the consumption performance of vector- consumer_group

[tietouwaa]

Vector Version

0.17.3

Vector Configuration File

[sources.kafka]
type = "kafka"
bootstrap_servers = "xxxx:9092,xxxx:9092,xxxx:9092"
group_id = "ngxallacctest1"
librdkafka_options."client.id" = "ngxallacc0"
auto_offset_reset = "latest"
#key_field = "message_key"
topics = [ "nginxAccessLogTest" ]
#topics = [ "^(elk-).*" ]
decoding.codec = "json"
librdkafka_options."fetch.min.bytes" = "5"
librdkafka_options."auto.commit.interval.ms" = "10000"
librdkafka_options."queued.max.messages.kbytes" = "524288"
fetch_wait_max_ms = 200
librdkafka_options."partition.assignment.strategy" = "roundrobin"
#==================transfer=============================#
#[transforms.parsing]
#type = "remap"
#inputs = [ "kafka" ] 
#source = """
#. = parse_json!(string!(.message))
#del(.offset)
#del(.partition)
#del(.user_id)
#del(.xff)
#del(.source_type)
#del(.topic)
#.web = del(.http_host)
#"""

#[transforms.to_json]
#type = "filter"
#type = "json_parser"
#inputs = [ "kafka" ]
#condition = '.status != 200'
#==================sinks=============================#
[sinks.logserver]
type = "file"
inputs = [ "kafka" ]
compression = "none"
path = "/dev/null"
encoding.codec = "ndjson"

Debug Output

1、

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
ngxallacctest1 nginxAccessLogTest 9 61481153458 61481300809 147351
ngxallacctest1 nginxAccessLogTest 22 61481798068 61481950971 152903

2、 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
ngxallacctest1 nginxAccessLogTest 9 61481251339 61481600321 348982 ngxallacctest1 nginxAccessLogTest 22 61481878802 61482251540 372738

Expected Behavior

1、Use vector to consume data from Kafka and output it locally, but the consumer performance of vector is very poor, and there is a large backlog of messages in the queue 2、I tried to start 8 processes and assign 16 threads to each, there is still a large backlog of messages, and the CPU usage of the processes is not high 3、Machine: Vector: Tencent Cloud CVM 16c32g

Actual Behavior

topic - configuration

How should I optimize the consumption performance of vector- consumer_group，thx~

[fuchsnj]

Could you try 2 things to help narrow down the issue here? I'm not sure what the performance of the file sink is writing to /dev/null, could you try the blackhole sink?

Also, have you tried the kafka console consumer, or any other kafka consumer to compare it with vector? This would help eliminate issues with the kafka cluster itself / your setup.

[tietouwaa]

1. The designated sink is blackhole I use [/usr/bin/vector --config kafkatofile1.toml> /tmp/vector1.log 2>&1 &] to start multiple processes （1）Start a process, consumption speed and backlog comparison (2)Start two process, Consumption backlog comparison (3)Comparison of the number of transmitted bytes 【 blackhole vs file】 sinks config：

(4)disk io (5)cpu info

Finally, I don't understand why there is a large backlog in the queue. The CPU usage of the vector process is not high.

And a process specifies 16 threads, if only one process is started, the CPU usage rate is about 200%; if two are started, the usage rate is 200% in total

Looking forward to your reply