promtail as kafka consumer to loki

[ly123-liu]

loki version：2.6.1 promtail version ： main release

we test promtail used as kafka consumer to loki, shows one promtail only to 4k/s speed（while use logstash can be 1w/s）， promtail just can't impove speed anymore ，is my configuration not correct?

monitoring image：

promtail configuration： server: http_listen_address: 0.0.0.0 http_listen_port: 9080 grpc_server_max_recv_msg_size: 104857600 grpc_server_max_send_msg_size: 104857600

positions: filename: /promtail/positions.yaml

clients:

• url: http://loki:3100/loki/api/v1/push batchsize: 1048576 batchwait: 1s # default backoff_config: min_period: 2s # default 500 max_period: 1m # default 5m max_retries: 100 # default 10

scrape_configs:

• job_name: log-kafka kafka: brokers:
  • log-kafka-d001:9092
  • log-kafka-e001:9092
  • log-kafka-e002:9092 topics:
  • ^log-tt group_id: infra-log-loki-consumer use_incoming_timestamp: true authentication: type: ssl tls_config: ca_file: /etc/promtail/certs/ca.crt cert_file: /etc/promtail/certs/ying99_ylog.crt key_file: /etc/promtail/certs/ying99_ylog.key insecure_skip_verify: true labels: job: kafka pipeline_stages:
  • json: expressions: timestamp: timestamp cluster: fields.cluster namespace: fields.namespace container: fields.container podname: fields.podname podip: fields.podip nodename: fields.nodename log_topic: fields.log_topic log_type: fields.log_type
  • labels: timestamp: cluster: namespace: nodename: container: podname: podip: log_topic: log_type: relabel_configs:
  • action: replace source_labels:
    • __meta_kafka_topic target_label: topic

      - action: replace

      source_labels:

      - __meta_kafka_partition

      target_label: partition

      - action: replace

      source_labels:

      - __meta_kafka_group_id

      target_label: group

limits_config: readline_rate: 4000 readline_burst: 8000 readline_rate_enabled: true readline_rate_drop: false max_streams: 0

[liguozhong]

limits_config:
  readline_rate: 4000
  readline_burst: 8000
  readline_rate_enabled: true
  readline_rate_drop: false

->

limits_config:
  readline_rate: 10000
  readline_burst: 20000
  readline_rate_enabled: true
  readline_rate_drop: false

[liguozhong]

join slack channel https://slack.grafana.com/

[ly123-liu]

sorry for so long , thanks, tests above those config helps, but still show some problems:

when promtail consumer one topic, promtail speed can to 19k/s

while promtail consumer many many topic , speed can only to 12k/s， when i scale promtail to above 12 or 24 pod , sped can only to 13w/s（while promtail consume specific kafka topic can be 30w/s）, seems increase prometail can not up speed， why is that？