Support Spark 3.0/Scala 2.12

[vungle-xinming-wang]

As the pre-built version of Spark 3.0 is based on Scala 2.12, if we want to upgrade Spark cluster to 3.0 and run salesforce DataFrame, we need spark-salesforce support Spark 3.0 and Scala 2.12.

[jackwang2]

@samuel-pt @k4jiang Appreciate if you can give any feedback.

[jackwang2]

@samuel-pt @k4jiang Could you give @vungle-xinming-wang access to push a branch, we'd like to create PR to make spark-salesforce support Scala 2.12/Spark 3.0.1?

[mkosovec]

+1 here. We're upgrading to Databricks 7.0+ and this is a requirement.

Let me know if you need any help developing the PR.

[jackwang2]

Hi @samuel-pt @k4jiang @phatak-dev @nhamiltontt @msathiyarajan @mengxr @dlively1,

I know some of you may no longer maintain the tool for now, but it's an open-source project and used by many companies in their data pipelines, and they want to upgrade it to support business growth.

We appreciate your efforts to build this tool very much and it did help others a lot. And now, let's make it better, please allow us to contribute our code.

If you got a chance to look at the notification, please kindly reply to us, thank you.

[phatak-dev]

@jackwang2 I don't have repo access from long time now. @samuel-pt or @springml should be able to help you.

[jackwang2]

@phatak-dev We tried but not able to reach @samuel-pt and @springml, appreciate it if you can help us contact them.

[lverzosa]

Created a PR for Scala 2.12 support (upgrading to Apache Spark 2.4.5 since Apache Spark 2.4.5 supports Scala 2.12). Should be usable in Spark 3 https://github.com/springml/spark-salesforce/pull/64

@phatak-dev Also copied in source code from https://github.com/phatak-dev/java-sizeof. Not sure if you were still actively maintaining otherwise we could update that library to Scala 2.12 and wouldn't need to pull in the source.

[bijeshch]

+1. Waiting for options to use Spark3.0 to Sales force.

[bijeshch]

Thanks for com.springml:spark-salesforce_2.12:1.1.4. It works

[akhila-chanubala]

Hello Team,

We are currently facing connectivity issue with Salesforce server. Please find the below details for the cluster configuration:

8.2 (includes Apache Spark 3.1.1, Scala 2.12) Maven Libraries: com.springml:spark-salesforce_2.12:1.1.4

Error: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.DeserializationContext.handleSecondaryContextualization(Lcom/fasterxml/jackson/databind/JsonDeserializer;Lcom/fasterxml/jackson/databind/BeanProperty;)Lcom/fasterxml/jackson/databind/JsonDeserializer;

Please let us know on a possible fix for this issue.

[sandeep-golla]

@bijeshch can you please help us on this issue? @akhila-chanubala

[bijeshch]

Not sure. If it's related to Salesforce connectivity. Seems to be related to XML

[lverzosa]

Hi everybody - Spark 3 and connector have a jackson xml mismatch. I've excluded the dependency for jackson xml when I've included the connector and it seems to do the job.

Long term I think we need to change Spark 2 (and jackson) to provided scope to avoid these issues in the future.

On Tue, Aug 31, 2021 at 2:34 PM bijeshch @.***> wrote:

Not sure. If it's related to Salesforce connectivity. Seems to be related to XML

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/springml/spark-salesforce/issues/60#issuecomment-909658737, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDH7ORYQIOAXCPVNGYPVTDT7VDFRANCNFSM4TK5342Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.