Stratio / deep-spark

Connecting Apache Spark with different data stores [DEPRECATED]
http://stratio.github.io/deep-spark
Apache License 2.0
197 stars 42 forks source link

ERROR: parameter class java.util.ArrayList does not have a Cassandra marshaller (v0.4.0) #7

Closed eraffenne closed 9 years ago

eraffenne commented 10 years ago

This error occurs in version 0.4.0 when writing a Cells RDD to Cassandra that contains a collection (List, Map or Set). Here is the trace when trying to write a ArrayList:

com.stratio.deep.exception.DeepGenericException: parameter class java.util.ArrayList does not have a Cassandra marshaller at com.stratio.deep.rdd.CassandraRDDUtils.marshallerInstance(CassandraRDDUtils.java:167) at com.stratio.deep.entity.CellValidator.cellValidator(CellValidator.java:229) at com.stratio.deep.entity.CassandraCell.getValueType(CassandraCell.java:139) at com.stratio.deep.entity.CassandraCell.(CassandraCell.java:153) at com.stratio.deep.entity.CassandraCell.create(CassandraCell.java:91) at com.stratio.deep.entity.CassandraCell.create(CassandraCell.java:66) at com.stratio.quantum.QuantumALS$1.call(QuantumALS.java:131) at com.stratio.quantum.QuantumALS$1.call(QuantumALS.java:122) at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:923) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at com.stratio.deep.rdd.CassandraRDDUtils$1.apply(CassandraRDDUtils.java:132) at com.stratio.deep.rdd.CassandraRDDUtils$1.apply(CassandraRDDUtils.java:125) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

The code snippet:

    JavaRDD<Cells> outputRDD = features.map(new Function<Tuple2<Object, double[]>, Cells>() {
        @Override
        public Cells call(Tuple2<Object, double[]> t) throws Exception {
            List<Double> dl = new ArrayList<Double>();
            for (double d : t._2()) {
                dl.add(d);
            }
            Cell c1 = CassandraCell.create("id", t._1(), true, false);
            Cell c2 = CassandraCell.create("features", dl); <== ERROR 
            return new Cells(c1, c2);
        }
    });
    CassandraRDD.saveRDDToCassandra(outputRDD, featuresConfig);
rcrespodelosreyes commented 9 years ago

Solved in version 0.6.

Thanks :)