Propagate pulsar client factory class name from driver spark context to RDD - Githubissues

streamnative / pulsar-spark

Spark Connector to read and write with Pulsar

Apache License 2.0

113 stars 50 forks source link

Propagate pulsar client factory class name from driver spark context to RDD #162

Closed chaoqin-li1123 closed 1 year ago

chaoqin-li1123 commented 1 year ago

Motivation

Previously we used the SparkConf in SparkEnv to get the pulsar client factory class in executor, which is not configurable in driver side.

Modifications

Propagate the pulsarClientFactoryClassName from driver to executor tasks.

Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)

[ ] This change is a trivial rework / code cleanup without any test coverage.
[x] This change is already covered by existing tests, such as:
[ ] This change added tests and can be verified as follows:

Documentation

Check the box below.

Need to update docs?

[ ] doc-required
[x] no-need-doc
[ ] doc