search

vertica / spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Apache License 2.0
20 stars 23 forks source link

What's the difference between assets published in git releases and maven? #521

Closed vladpy8 closed 1 year ago

vladpy8 commented 1 year ago

There are several ways to obtain jars required:

https://github.com/vertica/spark-connector/releases/tag/v3.3.4

https://mvnrepository.com/artifact/com.vertica.spark/vertica-spark/3.3.4

What's the difference between them?

I am working with pyspark and have to manually (at least for now) add jars to spark distribution. And I am not sure which one is better

Thanks

jeremyprime commented 1 year ago

The JARs under the GitHub release page and the JARs in Maven should be exactly the same. So you can use whatever method works best for you.

We provide two kinds of JARs, shaded (default) and slim. The shaded JAR is much larger and also contains all of the dependencies (Spark, Hadoop, etc), as long as those dependencies don't clash with other versions you are already using. The slim JAR contains just the Spark Connector code, so you'll also have to download all of the dependency JARs separately.

For example, for version 3.3.4: