search

jcustenborder / kafka-connect-solr

Kafka Connect connector for writing to Solr.
Apache License 2.0
44 stars 28 forks source link

Retry Attempt without data loss when sink side solr is down during runtime #43

Open rabeeshcharles opened 3 years ago

rabeeshcharles commented 3 years ago

Need to retry ( without any data loss) based on count from connector config if solr server is down during runtime.

In my case, Its working perfectly whenboth connector and solr are in running state [Active]. But while only solr server is down, there is no retry process until my data passed to the solr leads to data loss..

Sample Connnector config

curl -X POST -H "Content-Type: application/json" --data '{ "name": "t1", "config": { "tasks.max": "1", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "key.converter.schemas.enable": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "connector.class": "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector", "topics": "TRAN", "solr.queue.size": "100", "solr.commit.within": "10", "solr.url": "http://192.168.2.221:27052/solr/TRAN", "errors.retry.delay.max.ms":"5000", "errors.retry.timeout":"600000", "errors.tolerance":"all", "errors.log.enable":"true", "errors.log.include.messages":"false", "errors.deadletterqueue.topic.name":"DEAD_TRAN", "errors.deadletterqueue.topic.replication.factor":"1", "retry.backoff.ms":"1000", "reconnect.backoff.ms":"5000", "reconnect.backoff.max.ms":"600000" } }' http://localhost:8083/connectors

Error Information

image

rabeeshcharles commented 3 years ago

Can Someone help me on this?...