NICTA / scoobi

A Scala productivity framework for Hadoop.
http://nicta.github.com/scoobi/
482 stars 97 forks source link

Object field deleted when extending ScoobiApp #331

Closed plafl closed 10 years ago

plafl commented 10 years ago

The problem can be replicated from the sbt console:

Welcome to Scala version 2.10.3 (OpenJDK 64-Bit Server VM, Java 1.7.0_55).
Type in expressions to have them evaluated.
Type :help for more information.

scala> object Foo {val x = 10}
defined module Foo

scala> Foo.x
res0: Int = 10

scala> object Bar extends com.nicta.scoobi.Scoobi.ScoobiApp {val x = 10; def run() {}};
defined module Bar

scala> Bar.x
res1: Int = 0

A possible workaround is defining the field as a def or as lazy:

scala> object Spam extends com.nicta.scoobi.Scoobi.ScoobiApp {def x = 10; def run() {}};
defined module Spam

scala> Spam.x
res2: Int = 10

My build.sbt:

name := "scoobi-bug"

version := "1.0"

scalaVersion := "2.10.3"

libraryDependencies += "com.nicta" %% "scoobi" % "0.8.4"

resolvers ++= Seq(Resolver.sonatypeRepo("releases"),
                  Resolver.sonatypeRepo("snapshots"))

I have found this bug while executing in a cluster with CDH 4.6.0 (with other build configuration). The bug can be replicated locally as shown above inside the console, but surprisingly if I execute the scoobi application locally in my laptop, the bug doesn't manifest. I have been using scala and scoobi for just two weeks, I hope I'm not missing anything obvious.Thanks!

markhibberd commented 10 years ago

Regrettably this is standard scala behaviour when using DelayedInit.

scala> object Bar extends App { val x = 10 }
defined module Bar

scala> Bar.x
res4: Int = 0

The problem you are seeing is that syntactically it looks like you are assigning the value 10 to x when the object is initialized, but when you extend DelayedInit (App, and ScoobiApp both do this) your object initializtion code has to be run explicitly. Both App and ScoobiApp do this when the main method is called as this is their primary use.

For example, in normal scala you can see this with App:

scala> object Bar extends App { var x = 10 }
defined module Bar

scala> Bar.x
res5: Int = 0

scala> Bar.main(Array())

scala> Bar.x
res7: Int = 10

As to whether this code has to use DelayedInit, is a question for @etorreborre.

plafl commented 10 years ago

Ok, thanks for the explanation. Reading about traits I could not think how this behaviour could be possibly be implemented, searching a little further it seems that DelayedInit receives special treatment by the compiler.

Today I have learnt something (even if I don't like it :) )