Closed 87sanchavan closed 6 years ago
Same error .. just coming to post this same problem. Code works fine from Google Cloud but when run from my desktop using credentials it fails with the above.
Java.lang.ClassNotFoundException: java.lang.ProcessEnvironment$Variable
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.samelamin.spark.bigquery.utils.EnvHacker$.setEnv(EnvHacker.scala:15)
at com.samelamin.spark.bigquery.BigQueryClient$.setGoogleBQEnvVariable(BigQueryClient.scala:51)
at com.samelamin.spark.bigquery.BigQueryClient$.getInstance(BigQueryClient.scala:34)
at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19)
at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19)
at com.samelamin.spark.bigquery.BigQuerySQLContext.setBigQueryDatasetLocation(BigQuerySQLContext.scala:65)
at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply$mcV$sp(DfpSdSupplyDemandTest.scala:35)
at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22)
at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
Hi folks
Can you please post the code snippet you are trying to save?
It seems like perhaps the environment variable hasn't been set up right
Is this Pyspark or scala? On Fri, 22 Sep 2017 at 14:17, Daryl Erwin notifications@github.com wrote:
Same error .. just coming to post this same problem. Code works fine from Google Cloud but when run from my desktop using credentials it fails with the above.
Java.lang.ClassNotFoundException: java.lang.ProcessEnvironment$Variable at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.samelamin.spark.bigquery.utils.EnvHacker$.setEnv(EnvHacker.scala:15) at com.samelamin.spark.bigquery.BigQueryClient$.setGoogleBQEnvVariable(BigQueryClient.scala:51) at com.samelamin.spark.bigquery.BigQueryClient$.getInstance(BigQueryClient.scala:34) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.setBigQueryDatasetLocation(BigQuerySQLContext.scala:65) at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply$mcV$sp(DfpSdSupplyDemandTest.scala:35) at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22) at com.bbmtek.external.DfpSdSupplyDemandTest$$anonfun$1.apply(DfpSdSupplyDemandTest.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/44#issuecomment-331443768, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLm0znODQPFB85zTWUSiDrDQAMNBfQks5sk7NVgaJpZM4PW7t9 .
Scala .. must be something in my desktop config - it works on the google cloud vm. On gcp box I run .. spark-shell --packages com.github.samelamin:spark-bigquery_2.11:0.2.2,com.github.databricks:spark-avro:204864b6cf < test.scala
If I run via intellij on my desktop .. It fails on the last line ... note that I dont have the JsonKey line in the code when I run on gcp.
import com.samelamin.spark.bigquery._ spark.sqlContext.setGcpJsonKeyFile(s"$resources/BIGDATA-CREDENTIALS.json") spark.sqlContext.setBigQueryProjectId("bbm-production-bigdata") spark.sqlContext.setBigQueryGcsBucket("bbm-production-bigdata") spark.sqlContext.setBigQueryDatasetLocation("US")
Looks like its an issue with setting the env variable, what os are you using? and whats your java version?
Perhaps install the latest jdk?
On Fri, Sep 22, 2017 at 3:34 PM, Daryl Erwin notifications@github.com wrote:
Scala .. must be something in my desktop config - it works on the google cloud vm. On gcp box I run .. spark-shell --packages com.github.samelamin:spark-bigquery_2.11:0.2.2,com. github.databricks:spark-avro:204864b6cf < test.scala
If I run via intellij on my desktop .. It fails on the last line ... note that I dont have the JsonKey line in the code when I run on gcp.
import com.samelamin.spark.bigquery._ spark.sqlContext.setGcpJsonKeyFile(s"$resources/BIGDATA-CREDENTIALS.json") spark.sqlContext.setBigQueryProjectId("bbm-production-bigdata") spark.sqlContext.setBigQueryGcsBucket("bbm-production-bigdata") spark.sqlContext.setBigQueryDatasetLocation("US")
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/44#issuecomment-331464513, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLmzRHKZIfa92omDbhcrtLmTN6d57Bks5sk8WHgaJpZM4PW7t9 .
Windows .. I tried setting the GOOGLE_APPLICATION_CREDENTIALS. First time it said it couldnt find the file then I renamed and it found it. Got the above error. So I removed it thinking I could just use the api call instead but still get the error.
using jdk1.8.0_144 and jre1.8.0_111
Are you able to share any intelij settings that might be relevant? I have your bigquery_2.11-0.2.2.jar along with the spark2.2.0-bin-hadoop2.7 jars
Is the json creds file something similar to ..
{ "type": "service_account", "project_id": "xxxx", "private_key_id": "xxxxxxa", "private_key": "-----BEGIN PRIVATE KEY-----\n blah blah blah
Hi Daryl,
that is the json file yes.
It is certainly a problem with env variables and I have not tested the library against windows so its hard for me to debug it for you without actually being there
It shouldn't matter about intellij because the issue seems to be separate to the IDE
I am assuming you are using a fat/uber jar, if you spark submit via the console/terminal do you get the same error?
On Fri, Sep 22, 2017 at 4:31 PM, Daryl Erwin notifications@github.com wrote:
Is the json creds file something similar to ..
{ "type": "service_account", "project_id": "xxxx", "private_key_id": "xxxxxxa", "private_key": "-----BEGIN PRIVATE KEY-----\n blah blah blah
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/44#issuecomment-331480666, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLm51mFbWXNsMct69PvnlIVMIRpndIks5sk9LkgaJpZM4PW7t9 .
Did you say you didnt use an environment variable? or you do? Did you do anything special? I did in windows environment VARIABLE="/path/to/.json"
Well ideally you should not need to this is what environment hacker does. But for some reason the jre on your environment dies when it tries to set the environment variable
I'd investigate why this line is failing on your os
Java.lang.ClassNotFoundException: java.lang.ProcessEnvironment
I believe it has to do with the jre On Fri, 22 Sep 2017 at 20:22, Daryl Erwin notifications@github.com wrote:
Did you say you didnt use an environment variable? or you do? Did you do anything special? I did in windows environment VARIABLE="/path/to/.json"
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/44#issuecomment-331538158, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLm3AQfc8Hl_A-QI51LnQaAXmmd2Z4ks5slAkTgaJpZM4PW7t9 .
I am closing this issue as I have not heard anything back and I cannot replicate it
A member of our team here found the issue . it is the hack for read/writing environment variables that doesnt work on windows. He commented out the code as it wasn't apparently required. setting a hadoop environment variable?
com.samelamin.spark.bigquery.utils.EnvHacker
ah yes, ironically i added that to make it simpler to use to connector. It uses the Google Hadoop connector to move data in and out of Google storage and that needs the Google credentials file in a env variable
Thanks for raising it, if you send a pr ill review and merge. I would do it myself but i do not have a windows machine to test with
On Thu, Oct 5, 2017 at 4:35 PM, Daryl Erwin notifications@github.com wrote:
A member of our team here found the issue . it is the hack for read/writing environment variables that doesnt work on windows. He commented out the code as it wasn't apparently required. setting a hadoop environment variable?
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/44#issuecomment-334503704, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLm6fn5eM4d1gDLRvyPL-TxPDSHbwQks5spPdKgaJpZM4PW7t9 .
Hi Sam , I m facing this issue.. I think this could be versioning issue. I am same pyspark code you have given in another issue which I had raised. have you ever faced this or you have any idea about it.
java.lang.ClassNotFoundException: java.lang.ProcessEnvironment$Variable at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:360) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:260) at com.samelamin.spark.bigquery.utils.EnvHacker$.setEnv(EnvHacker.scala:15) at com.samelamin.spark.bigquery.BigQueryClient$.setGoogleBQEnvVariable(BigQueryClient.scala:51) at com.samelamin.spark.bigquery.BigQueryClient$.getInstance(BigQueryClient.scala:34) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.bigQuerySelect(BigQuerySQLContext.scala:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) Traceback (most recent call last): File "C:/Users/IT000493/PycharmProjects/sc-ln/pyspark-examples/hello_pyspark.py", line 30, in
File "D:\software\spark-2.2.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py", line 1133, in call File "D:\software\spark-2.2.0-bin-hadoop2.6\python\pyspark\sql\utils.py", line 63, in deco return f(*a, **kw) File "D:\software\spark-2.2.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o34.bigQuerySelect. : java.lang.NoSuchMethodError: com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List; at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase$ParentTimestampUpdateIncludePredicate.create(GoogleHadoopFileSystemBase.java:655) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.createOptionsBuilderFromConfig(GoogleHadoopFileSystemBase.java:2005) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1697) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:878) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:841) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at com.google.cloud.hadoop.io.bigquery.BigQueryConfiguration.getTemporaryPathRoot(BigQueryConfiguration.java:283) at com.google.cloud.hadoop.io.bigquery.AbstractBigQueryInputFormat.getSplits(AbstractBigQueryInputFormat.java:114) at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:125) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.rdd.RDD.partitions(RDD.scala:250) at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1333) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.take(RDD.scala:1327) at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1368) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.first(RDD.scala:1367) at com.samelamin.spark.bigquery.BigQuerySQLContext.bigQuerySelect(BigQuerySQLContext.scala:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745)