search

Stratio / Spark-MongoDB

Spark library for easy MongoDB access
http://www.stratio.com
Apache License 2.0
307 stars 99 forks source link

Caused by: java.lang.IllegalArgumentException: state should be: w >= 0 #147

Open a4712020502 opened 8 years ago

a4712020502 commented 8 years ago

2016-06-30 13:04:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.ExceptionInInitializerError at com.mongodb.casbah.BaseImports$class.$init$(Implicits.scala:162) at com.mongodb.casbah.Imports$.(Implicits.scala:142) at com.mongodb.casbah.Imports$.(Implicits.scala) at com.mongodb.casbah.MongoClient.apply(MongoClient.scala:219) at com.stratio.datasource.mongodb.partitioner.MongodbPartitioner.isShardedCollection(MongodbPartitioner.scala:78) at com.stratio.datasource.mongodb.partitioner.MongodbPartitioner$$anonfun$computePartitions$1.apply(MongodbPartitioner.scala:67) at com.stratio.datasource.mongodb.partitioner.MongodbPartitioner$$anonfun$computePartitions$1.apply(MongodbPartitioner.scala:66) at com.stratio.datasource.util.using$.apply(using.scala:38) at com.stratio.datasource.mongodb.partitioner.MongodbPartitioner.computePartitions(MongodbPartitioner.scala:66) at com.stratio.datasource.mongodb.rdd.MongodbRDD.getPartitions(MongodbRDD.scala:42) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:330) at com.stratio.datasource.mongodb.schema.MongodbSchema.schema(MongodbSchema.scala:47) at com.stratio.datasource.mongodb.MongodbRelation.com$stratio$datasource$mongodb$MongodbRelation$$lazySchema$lzycompute(MongodbRelation.scala:63) at com.stratio.datasource.mongodb.MongodbRelation.com$stratio$datasource$mongodb$MongodbRelation$$lazySchema(MongodbRelation.scala:60) at com.stratio.datasource.mongodb.MongodbRelation$$anonfun$1.apply(MongodbRelation.scala:65) at com.stratio.datasource.mongodb.MongodbRelation$$anonfun$1.apply(MongodbRelation.scala:65) at scala.Option.getOrElse(Option.scala:120) at com.stratio.datasource.mongodb.MongodbRelation.(MongodbRelation.scala:65) at com.stratio.datasource.mongodb.DefaultSource.createRelation(DefaultSource.scala:36) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) at com.askingdata.test.TestSpark.main(TestSpark.java:23) Caused by: java.lang.IllegalArgumentException: state should be: w >= 0 at com.mongodb.assertions.Assertions.isTrueArgument(Assertions.java:99) at com.mongodb.WriteConcern.(WriteConcern.java:316) at com.mongodb.WriteConcern.(WriteConcern.java:227) at com.mongodb.casbah.WriteConcern$.(WriteConcern.scala:41) at com.mongodb.casbah.WriteConcern$.(WriteConcern.scala) ... 37 more

gohilankit commented 8 years ago

Any update on this? Even I'm getting this issue with PySpark. I'm using the version 2.10:0.11.0 and running the followint piece of code

reader = sqlContext.read.format('com.stratio.datasource.mongodb')
df = reader.options(host=<ip>, database=<db>, collection=<collection>).load()
ishare commented 8 years ago

It seems stratio mgo is not compatible with the new official mgo driver. I try to remove the offical driver dependency, then it works. I wanna is there any other way to make both available?