Kafka logger not working

[gaoxingliang]

Description

I try to use kafka logger to record all the requests and response and replace the clickhouse-logger(it's working) after I add it in the apisix dashboard, it shows nothing messages about kafka. and when I triggered a request and found the data is never sent to kafka.

at same time, I created another plugin response-rewrite and found it's working.

The apisix conf shows it's enabled:

plugins:    # plugin list
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - file-logger
  - grpc-transcode
  - grpc-web
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
  - ext-plugin-post-resp
  - server-info
  - elasticsearch-logger
  - clickhouse-logger
  - forward-auth
stream_plugins:
  - mqtt-proxy
  - ip-restriction
  - limit-conn

kafka logger conf:

{
  "_meta": {
    "disable": false
  },
  "batch_max_size": 1,
  "brokers": [
    {
      "host": "kafka.devops",
      "port": 9092
    }
  ],
  "disable": false,
  "kafka_topic": "apisix-log-test",
  "name": "kafka logger"
}

It seems this plugin is DISAPPEARED.

Environment

• APISIX version (run apisix version): 3.3.0
• APISIX dashboard: 3.0.0
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): 3.5.0
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

Are there any errors in error.log?

[gaoxingliang]

@shreemaan-abhishek I don;t see any error in the apisix error log. I created a new version cluster with 3.6.0. and it worked. and I also found the apisix occasionally got error of topic not found (even it's there in apisix 3.6.0):

2023/10/19 05:56:00 [error] 51#51: *1062684 [lua] batch-processor.lua:95: Batch Processor[apisix-kafkalogger] failed to process entries: failed to send data to Kafka topic: not found topic, brokers: null, context: ngx.timer, client: 10.233.113.0, server: 0.0.0.0:9080
2023/10/19 05:56:00 [error] 51#51: *1062684 [lua] batch-processor.lua:104: Batch Processor[apisix-kafkalogger] exceeded the max_retry_count[1] dropping the entries, context: ngx.timer, client: 10.233.113.0, server: 0.0.0.0:9080

[Revolyssup]

@gaoxingliang This might be an issue with lua-resty-kafka when FQDN is used instead of IP. Relevant issues: https://github.com/doujiang24/lua-resty-kafka/issues/6 https://github.com/doujiang24/lua-resty-kafka/issues/5

[gaoxingliang]

This issue persist. The apisix version is 3.6.0. and the conf:

{
  "_meta": {
    "disable": false
  },
  "brokers": [
    {
      "host": "192.168.102.227",
      "port": 39091
    }
  ],
  "include_req_body": true,
  "kafka_topic": "apisix-log-test",
  "name": "apisix-kafkalogger"
}

and the error log:

 2023/11/21 09:02:25 [warn] 55#55: *1345942 [lua] plugins.lua:54: query parameter "all" will be deprecated soon., client: 10.233.113.84, server: , request: "GET /apisix/admin/plugins?all=true HTTP/1.1", host: "apisix-admin.apisix.svc.cluster.local:9180"

 2023/11/21 09:02:28 [error] 57#57: *1350184 [lua] batch-processor.lua:95: Batch Processor[apisix-kafkalogger] failed to process entries: failed to send data to Kafka topic: not found broker, brokers: null, context: ngx.timer, client: 10.233.107.0, server: 0.0.0.0:9080

 2023/11/21 09:02:28 [error] 57#57: *1350184 [lua] batch-processor.lua:104: Batch Processor[apisix-kafkalogger] exceeded the max_retry_count[1] dropping the entries, context: ngx.timer, client: 10.233.107.0, server: 0.0.0.0:9080

I don't know why it said the not found broker. I telnet the kafka and it's connected.

I found the kafka logger plugin code error msg use the deprecated broker_list. so I also try with the conf:

  "broker_list": {
    "192.168.102.227": 39091
  },

It also says the broker list is null. I think the problem is: the conf get from etcd or somewhere is NULL. the source code is here: https://github.com/apache/apisix/blob/master/apisix/plugins/kafka-logger.lua#L203

[monkeyDluffy6017]

@gaoxingliang have you solved your problem?

[gaoxingliang]

@monkeyDluffy6017 no. It's unstable. I have to do the log manually.

[kayx23]

Please try the examples in this doc to see if anything helps with your case: https://docs.api7.ai/hub/kafka-logger

They were tested to be working and the doc is more current.