SahilKang / cl-rdkafka

Common Lisp library for Kafka
Other
42 stars 7 forks source link

Provide a restart when producer's send reaches max queue #59

Open SahilKang opened 4 years ago

SahilKang commented 4 years ago

Something like this signals a partition-error from the producer's send method:

(ql:quickload '(cl-rdkafka babel))

(let ((producer (make-instance
                 'kf:producer
                 :conf '("bootstrap.servers" "127.0.0.1:9092")
                 :serde #'babel:string-to-octets)))
  (loop
     repeat 500000
     do (kf:send producer "test-topic" "test-message"))
  (kf:flush producer))

The signaled condition looks something like:

Encountered error CL-RDKAFKA/LOW-LEVEL:RD-KAFKA-RESP-ERR--QUEUE-FULL for `test-topic:-1`: "Local: Queue full"

This is one of the expected errors that may occur from the rd_kafka_produce function and a restart should be provided in this case to recover.