search

NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

Fix for NullPointerException while updating hadoop job counters. #356

Closed deepujain closed 9 years ago

deepujain commented 9 years ago

1) Error seen in scoobi Version: 0.8.3

We have seen this error repeatedly. Even if the job succeeded with few failed map attempts, scoobi threw null pointer exception. Hence adding null checks.

Logs

[INFO] OutputChannel - scoobi.tmp.copy : mrename [INFO] OutputChannel - Rename Time : 1 mins [INFO] OutputChannel - scoobi.tmp.copy : mrename [INFO] OutputChannel - Rename Time : 0 mins Exception in thread "main" java.lang.NullPointerException at com.nicta.scoobi.impl.ScoobiConfigurationImpl.updateCounters(ScoobiConfigurationImpl.scala:99) at com.nicta.scoobi.impl.ScoobiConfigurationImpl.updateCounters(ScoobiConfigurationImpl.scala:45) at com.nicta.scoobi.impl.exec.HadoopMode$Execution$$anonfun$reportMscr$1.apply(HadoopMode.scala:173) at com.nicta.scoobi.impl.exec.HadoopMode$Execution$$anonfun$reportMscr$1.apply(HadoopMode.scala:171) at scala.Function2$$anonfun$tupled$1.apply(Function2.scala:54) at scala.Function2$$anonfun$tupled$1.apply(Function2.scala:53) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at com.nicta.scoobi.impl.exec.HadoopMode$Execution.runMscrs(HadoopMode.scala:146) at com.nicta.scoobi.impl.exec.HadoopMode$Execution.execute(HadoopMode.scala:125) at com.nicta.scoobi.impl.exec.HadoopMode$$anonfun$com$nicta$scoobi$impl$exec$HadoopMode$$executeLayer$1.apply(HadoopMode.scala:115) at com.nicta.scoobi.impl.exec.HadoopMode$$anonfun$com$nicta$scoobi$impl$exec$HadoopMode$$executeLayer$1.apply(HadoopMode.scala:114) at org.kiama.attribution.AttributionCore$CachedAttribute.apply(AttributionCore.scala:62) at com.nicta.scoobi.impl.exec.HadoopMode$$anonfun$com$nicta$scoobi$impl$exec$HadoopMode$$executeLayers$1$1.apply(HadoopMode.scala:73) at com.nicta.scoobi.impl.exec.HadoopMode$$anonfun$com$nicta$scoobi$impl$exec$HadoopMode$$executeLayers$1$1.apply(HadoopMode.scala:73) at scala.collection.immutable.List.foreach(List.scala:318)

Travis Build is Success = https://travis-ci.org/deepujain/scoobi

2) Updated .travis* file to include compilation and packaging.

etorreborre commented 9 years ago

Thanks for you PR. I didn't merge it in straight away because:

I have published this fix in 0.9.2-SNAPSHOT if you want to try it out.

deepujain commented 9 years ago

I do not see a branch/tag of 0.9.2-SNAPSHOT. How do i see your changes and which repository contains 0.9.2-SNAPSHOT jar ?

etorreborre commented 9 years ago

You can get it here.

deepujain commented 9 years ago

Thanks. I will try it out. I have sent you few tweets regarding the help i need in setting up my scala project on eclipse. Could you please help me out there.

deepujain commented 9 years ago

Can snapshot be used with 2.10 scala version ?

We use. scala.version = 2.10.3 scala.binary.version = 2.10 scala.tools.version = 2.10

etorreborre commented 9 years ago

Can you bring that to the mailing-list? I am not a Eclipse user but maybe other people can help you out there.

etorreborre commented 9 years ago

I have now published a Scoobi version for Scala 2.10: https://oss.sonatype.org/content/repositories/snapshots/com/nicta/scoobi_2.10/0.9.2-SNAPSHOT/