search

Stratio / Spark-MongoDB

Spark library for easy MongoDB access
http://www.stratio.com
Apache License 2.0
307 stars 99 forks source link

SplitkeyMin and SplitKeyMax is not working for isodate column #170

Open bbnsumanth opened 7 years ago

bbnsumanth commented 7 years ago

I'm using following config to read data using a splitkeyMin and Max.But whole data is getting loaded into dataframe.Im using latest version of spark-MonogDB and spark 2.0.0

val mongoConfig = MongodbConfigBuilder( Map( Credentials -> List(slaveCredentials), Host -> mongoHost, Database -> mongoDatabase, Collection -> mongoCollection, SamplingRatio -> 1.0, WriteConcern -> "normal", SplitSize -> "10", SplitKey -> "created_at", SplitKeyMin -> "2016-11-20T10:01:32.239Z", SplitKeyMax -> "2016-11-23T10:01:32.239Z", SplitKeyType -> "isoDate" ) ).build()

val mongoDF = spark.sqlContext.fromMongoDB(mongoConfig)

bbnsumanth commented 7 years ago

It is not working as there is no index on "created_at" field in my mongoCollection. I would be very helpful if this is included in the documentation and also what is the format we must use in splitkeyMin/Max for isodate data type.