Facing issues while sending Databricks Lineage (Spline) to kafka

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark

https://absaoss.github.io/spline/

Apache License 2.0

175 stars 90 forks source link

Facing issues while sending Databricks Lineage (Spline) to kafka #766

Open B09306 opened 7 months ago

B09306 commented 7 months ago

Hi Team, we have enabled spline on databricks cluster and we are able to connect from databricks to kafka using SSL certificate but spline is not able to connect to kafka that's why it is not able to send lineage to kafka & got below error .

SparkLineageInitializer: Spline successfully initialized. Spark Lineage tracking is ENABLED. 23/11/13 11:07:49 WARN NetworkClient: [Producer clientId=producer-1] Bootstrap broker xxxxxxxxxxxxx:9092 (id: -1 rack: null) disconnected

could anyone please look into this and help us.

cerveada commented 7 months ago

You have to configure the Kafka producer the same way you provide bootstrap servers you can provide all other Kafka properties. Just be sure to use the right prefix.

For example for bootstrap.servers and ssl.key.password

spline.lineageDispatcher.kafka.producer.bootstrap.servers=localhost:9092
spline.lineageDispatcher.kafka.producer.ssl.key.password=xxx
//and so on

https://docs.confluent.io/platform/7.5/installation/configuration/producer-configs.html