search

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.
Apache License 2.0
70 stars 28 forks source link

Problems with Guava version using spark 2.2 #50

Closed lbadi closed 6 years ago

lbadi commented 6 years ago

Hi Sam! How are you ?

I have been trying to use you library to write in batch mode into a big-query table on a local spark 2.2. I'm having problems with the infamous java.lang.NoSuchMethodError: com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;

The last version of spark-bigquery is compatible with spark 2.2 right ?

Also i tried to shade the google.common.base dependency with the maven shade plugin but din't work.

Do you have any idea what could be happening ?

samelamin commented 6 years ago

Hey @lbadi that error is really annoying! You are right its def Guava, I would guess you have a local installation of Guava 14 somewhere in your class path which is causing this issue

Can you check your spark lib or jar folders? its most likely there e.g. {spark-folder}/jars

lbadi commented 6 years ago

Thanks for the fast response sam!! Yes! You are right i have Guava 14 on /jars !! Are you saying that i can update that jar and it should work ? Also I solve this using properly the shade plugin but i think is not an elegant way.

Now i'm having another problem with java.lang.NoClassDefFoundError: com/google/api/client/http/HttpRequestInitializer. I think i'll be able to solve it with the same technique.

Really appreciate the fast response Sam!

samelamin commented 6 years ago

No worries, happy to help :)

Yeah I normally just overwrite the guava 14 with a newer version, you only need to do that locally as most clusters will have the latest