search

samelamin / spark-bigquery

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.
Apache License 2.0
70 stars 28 forks source link

Exception in thread "main" java.lang.NoSuchMethodError: #73

Closed badgujarsupriya closed 5 years ago

badgujarsupriya commented 5 years ago

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68) at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40)

Can you please tell me from which library you are getting 'checkArgument'?

fxofxo commented 5 years ago

I am facing the same problem. I was using the library without problems in google Dataproc environtment and from 22 of febrary is failing on the last version of dataproc.

samelamin commented 5 years ago

Sorry folks I am away on Holiday till Friday

I believe that error is due to a guava version mismatch, if you google it you will see where the problem is

Same issue appears here https://github.com/samelamin/spark-bigquery/issues/72

I think if you shade your libraries and create a fat jar it should work as long as guava is over version 14, i use 18 in my project and it seems to work but I have not tested with the latest dataproc

I shall be looking into this over the weekend, but if any of you want to do a headstart and test it out I would really appreciate it

fxofxo commented 5 years ago

Hi all, I have try, WITHOUT success, packing guava library on the jar adding dependency for guava v18, in the 

<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>18.0</version>
</dependency>

Any other think I could try? Thanks. Felix

samelamin commented 5 years ago

This issue is now fixed, please try the latest release and reopen if not the case

Thanks for raising it

fxofxo commented 5 years ago

Thanks for your time! but it seems not work for me. I have update pom file to get new version 0.2.6 how ever the error persist.

_BigQueryClient:141 - Executing query SELECT * FROM [business-intelligence-noticias:DS_DEV_minsait_dev.Rel_CanalCaptacion] 2019-03-27 15:12:22 INFO BigQueryClient:164 - Staging dataset datanoticiasdesarrollo:spark_bigquery_staging_eu already exists 2019-03-27 15:12:22 INFO BigQueryClient:144 - Destination table: {datasetId=spark_bigquery_staging_eu, projectId=datanoticiasdesarrollo, tableId=spark_bigquery_201903271412221158602716} 2019-03-27 15:12:22 INFO BigQueryClient:200 - Using legacy Sql: true Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68) . I include the maven tree output if it helps. mvnTres.txt Thanks in advance Felix

OsamaMaud commented 5 years ago

yes It still persists, I am also getting this error Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68) at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40) at com.samelamin.spark.bigquery.streaming.BigQuerySink.addBatch(BigQuerySink.scala:29) ...... .... ...

Using guava v19, spark 2.3 on dataproc.

ChitturiPadma commented 5 years ago

I also face the same issue with latest version.

AyWa commented 5 years ago

Similar issue on my side, didnt found work arround yet

dgouyette commented 5 years ago

I created a workflow with this dependencies provided by the cluster : And it solves differents versions issues

jobs:
  - sparkJob:
      jarFileUris:
        - gs://bucket/xxxx-0.1.0-SNAPSHOT.jar
      mainClass: jobs.BQ2GCS
    stepId: step1
placement:
  managedCluster:
    clusterName: xxx
    config:
      gceClusterConfig:
        metadata:
          bigquery-connector-version: 0.13.16
          gcs-connector-version: 1.9.16

        zoneUri: europe-west1-b
      initializationActions:
        - executableFile: gs://dataproc-initialization-actions/connectors/connectors.sh
          executionTimeout: 600s
      softwareConfig:
        imageVersion: '1.4'
      workerConfig:
        numInstances: 2
AyWa commented 5 years ago

@dgouyette Thank you it is working. I could also make it works by forcing "com.google.guava:guava:27.0.1-jre" to be use

mohanp062 commented 5 years ago

I also face the same issue with latest version.

were you able to fix this? If yes , can you please share the soluion. Thanks

mohanp062 commented 5 years ago

@dgouyette Thank you it is working. I could also make it works by forcing "com.google.guava:guava:27.0.1-jre" to be use

This still doesnt resolve the issue for me

michTalebzadeh commented 4 years ago

I came across this error when running on a newer version of Dataproc cluster.

To resolve it do the following in your build (mine assembly) with shade rules to get rid of old references to guava

assemblyShadeRules in assembly := Seq( ShadeRule.rename("com.google.common.**" -> "my_conf.@1").inAll )

Also see this dated email of mine

https://groups.google.com/forum/#!msg/cloud-dataproc-discuss/OICSgtN2oe8/bhOswni8EAAJ

currently I am using this dependency in my assembly

libraryDependencies += "com.google.guava" % "guava" % "27.0.1-jre"

HTH,

Mich

samelamin commented 4 years ago

Thanks Mich, guava is such a pain. Essentially this is fixed by shading the google common libraries

If the above doesnt work I am happy reopening this ticket