search

NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

ConversionException when joining DObject #267

Closed ebastien closed 11 years ago

ebastien commented 11 years ago

I am trying to join a DObject with a DList and it fails when running locally. It only works for an in-memory run.

The minimized test case is as follows:

    val ints: DList[Int] = DList(1,2,3)
    val obj: DObject[Iterable[Int]] = ints.materialise

    val res = obj.join(ints).map {
      case (o: Iterable[Int], r: Int) => r + o.sum
    }

    persist(res.toTextFile(args(1), overwrite = true))

My build.sbt file:


import AssemblyKeys._

assemblySettings

assemblyCacheOutput in assembly := false

assemblyCacheUnzip in assembly := true

name := "playground"

version := "0.1"

scalaVersion := "2.10.1"

libraryDependencies += "com.nicta" %% "scoobi" % "0.7.0-RC1-cdh3"

libraryDependencies += "org.specs2" %% "specs2" % "2.0-RC1" % "test"

libraryDependencies += "joda-time" % "joda-time" % "2.2"

libraryDependencies += "org.joda" % "joda-convert" % "1.3.1"

resolvers ++= Seq(
    "cloudera" at "https://repository.cloudera.com/content/repositories/releases",
    "Sonatype-snapshots" at "http://oss.sonatype.org/content/repositories/snapshots")

mergeStrategy in assembly <<= (mergeStrategy in assembly) { mergeStrategy => {
    case entry => {
      val strategy = mergeStrategy(entry)
      if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
      else strategy
    }
  }
}

The exception I get:


com.thoughtworks.xstream.converters.ConversionException: Could not call scala.collection.immutable.$colon$colon.readObject() : sbt.classpath.ClasspathUtilities$$anon$1 : sbt.classpath.ClasspathUtilities$$anon$1
---- Debugging information ----
message             : sbt.classpath.ClasspathUtilities$$anon$1
cause-exception     : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message       : sbt.classpath.ClasspathUtilities$$anon$1
class               : java.util.ResourceBundle$LoaderReference
required-type       : java.util.ResourceBundle$LoaderReference
converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
path                : /com.nicta.scoobi.impl.plan.mscr.InputChannels/channels/scala.collection.immutable.$colon$colon/com.nicta.scoobi.impl.plan.mscr.FloatingInputChannel/nodes/sc/counters/groups/entry[2]/org.apache.hadoop.mapreduce.CounterGroup/bundle/cacheKey/loaderRef/referent
class[1]            : java.util.ResourceBundle$CacheKey
class[2]            : java.util.PropertyResourceBundle
class[3]            : org.apache.hadoop.mapreduce.CounterGroup
class[4]            : java.util.TreeMap
converter-type[1]   : com.thoughtworks.xstream.converters.collections.TreeMapConverter
class[5]            : org.apache.hadoop.mapreduce.Counters
class[6]            : com.nicta.scoobi.impl.ScoobiConfigurationImpl
class[7]            : com.nicta.scoobi.impl.exec.HadoopMode
class[8]            : com.nicta.scoobi.impl.plan.mscr.FloatingInputChannel
-------------------------------
message             : Could not call scala.collection.immutable.$colon$colon.readObject()
cause-exception     : com.thoughtworks.xstream.converters.ConversionException
cause-message       : sbt.classpath.ClasspathUtilities$$anon$1 : sbt.classpath.ClasspathUtilities$$anon$1
class               : scala.collection.immutable.$colon$colon
required-type       : java.util.ResourceBundle$LoaderReference
converter-type      : com.thoughtworks.xstream.converters.reflection.SerializableConverter
path                : /com.nicta.scoobi.impl.plan.mscr.InputChannels/channels/scala.collection.immutable.$colon$colon/com.nicta.scoobi.impl.plan.mscr.FloatingInputChannel/nodes/sc/counters/groups/entry[2]/org.apache.hadoop.mapreduce.CounterGroup/bundle/cacheKey/loaderRef/referent
class[1]            : com.nicta.scoobi.impl.plan.mscr.InputChannels
converter-type[1]   : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
version             : null
-------------------------------
    at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadObject(SerializationMethodInvoker.java:119)
    at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doUnmarshal(SerializableConverter.java:425)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1058)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1042)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:922)
    at com.nicta.scoobi.impl.util.Serialiser$class.deserialise(Serialiser.scala:50)
    at com.nicta.scoobi.impl.util.Serialiser$.deserialise(Serialiser.scala:62)
    at com.nicta.scoobi.impl.util.DistCache$$anonfun$deserialise$1.apply(DistCache.scala:83)
    at com.nicta.scoobi.impl.util.DistCache$$anonfun$deserialise$1.apply(DistCache.scala:80)
    at com.nicta.scoobi.impl.util.DistCache$$anonfun$pullObject$2.apply(DistCache.scala:73)
    at com.nicta.scoobi.impl.util.DistCache$$anonfun$pullObject$2.apply(DistCache.scala:72)
    at scala.Option.flatMap(Option.scala:170)
    at com.nicta.scoobi.impl.util.DistCache$.pullObject(DistCache.scala:72)
    at com.nicta.scoobi.impl.mapreducer.MscrMapper.setup(MscrMapper.scala:47)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
Caused by: com.thoughtworks.xstream.converters.ConversionException: sbt.classpath.ClasspathUtilities$$anon$1 : sbt.classpath.ClasspathUtilities$$anon$1
---- Debugging information ----
message             : sbt.classpath.ClasspathUtilities$$anon$1
cause-exception     : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message       : sbt.classpath.ClasspathUtilities$$anon$1
class               : java.util.ResourceBundle$LoaderReference
required-type       : java.util.ResourceBundle$LoaderReference
converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
path                : /com.nicta.scoobi.impl.plan.mscr.InputChannels/channels/scala.collection.immutable.$colon$colon/com.nicta.scoobi.impl.plan.mscr.FloatingInputChannel/nodes/sc/counters/groups/entry[2]/org.apache.hadoop.mapreduce.CounterGroup/bundle/cacheKey/loaderRef/referent
class[1]            : java.util.ResourceBundle$CacheKey
class[2]            : java.util.PropertyResourceBundle
class[3]            : org.apache.hadoop.mapreduce.CounterGroup
class[4]            : java.util.TreeMap
converter-type[1]   : com.thoughtworks.xstream.converters.collections.TreeMapConverter
class[5]            : org.apache.hadoop.mapreduce.Counters
class[6]            : com.nicta.scoobi.impl.ScoobiConfigurationImpl
class[7]            : com.nicta.scoobi.impl.exec.HadoopMode
class[8]            : com.nicta.scoobi.impl.plan.mscr.FloatingInputChannel
-------------------------------
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
    at com.thoughtworks.xstream.converters.collections.MapConverter.putCurrentEntryIntoMap(MapConverter.java:90)
    at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:78)
    at com.thoughtworks.xstream.converters.collections.TreeMapConverter.populateTreeMap(TreeMapConverter.java:141)
    at com.thoughtworks.xstream.converters.collections.TreeMapConverter.unmarshal(TreeMapConverter.java:104)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshallField(AbstractReflectionConverter.java:355)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:306)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.converters.reflection.SerializableConverter$2.readFromStream(SerializableConverter.java:311)
    at com.thoughtworks.xstream.core.util.CustomObjectInputStream.readObjectOverride(CustomObjectInputStream.java:104)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:363)
    at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
    at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadObject(SerializationMethodInvoker.java:113)
    ... 30 more
Caused by: com.thoughtworks.xstream.mapper.CannotResolveClassException: sbt.classpath.ClasspathUtilities$$anon$1
    at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:56)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
    at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:45)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.determineType(AbstractReflectionConverter.java:458)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:294)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:234)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
    ... 92 more
ebastien commented 11 years ago

It seems to work with 0.7.0-RC2-cdh3-SNAPSHOT