search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
176 stars 92 forks source link

Update MongoSpark connector #668

Open mohdmsl opened 1 year ago

mohdmsl commented 1 year ago

the mongo spark version is very old 

<dependency>
    <groupId>org.mongodb.spark</groupId>
    <artifactId>mongo-spark-connector_${scala.binary.version}</artifactId>
    <version>2.4.1</version>
</dependency>

Please update to lates version which is 10.1.1

I am getting error if I use latest version of mongo-spark-connector. but with older versions its working fine.

So please update the mongo-spark-connector

wajda commented 1 year ago

Spline agent has to support both Spline 2.11 and 2.12, while mongo-spark-connector ver 10.1.1 only exists for Scala 2.12+ It's possible to solve with reflection, but it's not a quick change. Since upgrading Mongodb connector isn't a priority for the team it'll not be done soon. We appreciate PRs though!