Repl seems broken in local mode

[johnynek]

There is also a mailing list thread:

https://groups.google.com/forum/#!topic/scalding-dev/y582r5zmr18

I guess a dependency is missing from the repl. We have a test, but the approach to run it is not just calling run --local on the repl target.

t-oscar1:scalding oscar$ ./sbt "scalding-repl/run --local"
[info] Loading project definition from /Users/oscar/oss/scalding/project
[info] Set current project to scalding (in build file:/Users/oscar/oss/scalding/)
[info] Running com.twitter.scalding.ScaldingShell --local
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[error] (run-main-0) java.lang.NoClassDefFoundError: cascading/flow/FlowConnector
java.lang.NoClassDefFoundError: cascading/flow/FlowConnector
        at com.twitter.scalding.Mode$.apply(Mode.scala:87)
        at com.twitter.scalding.BaseScaldingShell$class.parseModeArgs(ScaldingShell.scala:117)
        at com.twitter.scalding.ScaldingShell$.parseModeArgs(ScaldingShell.scala:196)
        at com.twitter.scalding.BaseScaldingShell$class.process(ScaldingShell.scala:66)
        at com.twitter.scalding.ScaldingShell$.process(ScaldingShell.scala:196)
        at com.twitter.scalding.BaseScaldingShell$class.main(ScaldingShell.scala:127)
        at com.twitter.scalding.ScaldingShell$.main(ScaldingShell.scala:196)
        at com.twitter.scalding.ScaldingShell.main(ScaldingShell.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: cascading.flow.FlowConnector
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.twitter.scalding.Mode$.apply(Mode.scala:87)
        at com.twitter.scalding.BaseScaldingShell$class.parseModeArgs(ScaldingShell.scala:117)
        at com.twitter.scalding.ScaldingShell$.parseModeArgs(ScaldingShell.scala:196)
        at com.twitter.scalding.BaseScaldingShell$class.process(ScaldingShell.scala:66)
        at com.twitter.scalding.ScaldingShell$.process(ScaldingShell.scala:196)
        at com.twitter.scalding.BaseScaldingShell$class.main(ScaldingShell.scala:127)
        at com.twitter.scalding.ScaldingShell$.main(ScaldingShell.scala:196)
        at com.twitter.scalding.ScaldingShell.main(ScaldingShell.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last scalding-repl/unprovided:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last scalding-repl/*:run for the full output.
[error] (scalding-repl/*:run) Nonzero exit code: 1
[error] Total time: 3 s, completed Oct 26, 2016 7:12:53 AM

[davidgetchell]

I am also manifesting this issue. A couple of other notes:

This works:

/ap_data/spectra/scalding$ ./sbt scalding-repl/console [info] Loading project definition from /ap_data/spectra/scalding/project [info] Resolving key references (21992 settings) ... [info] Set current project to scalding (in build file:/apdata/spectra/scalding/) [info] Starting scala interpreter... [info] import com.twitter.scalding. import com.twitter.scalding.ReplImplicits. import com.twitter.scalding.ReplImplicitContext. Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67). Type in expressions for evaluation. Or try :help.

It reports Scalding 2.11.8, whereas the system Scala is 2.10.4

/ap_data/spectra/scalding$ scala Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67). Type in expressions to have them evaluated. Type :help for more information.

And Spark is using 2.10.5:

/ap_data/spectra/scalding$ spark-shell /usr/bin/spark-shell: fork: retry: Resource temporarily unavailable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). Welcome to

---

 / **/**  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/

// ./,// //_\ version 1.6.0 //

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67) Type in expressions to have them evaluated. Type :help for more information. 16/10/26 14:58:47 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. Spark context available as sc (master = yarn-client, app id = application_1473161870114_15336). SQL context available as sqlContext.

scala>

The issue that brought me here is my inability to assemble a fat jar with ./sbt assemble

[error] (scalding-parquet/test:test) sbt.TestsFailedException: Tests unsuccessful [error] (scalding-thrift-macros/test:test) sbt.TestsFailedException: Tests unsuccessful [error] (scalding-hadoop-test/test:test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 715 s, completed Oct 26, 2016 3:00:13 PM

[johnynek]

you can disable running the tests to build an assembly jar:

https://github.com/twitter/scalding/blob/develop/build.sbt#L122

[johnynek]

Also, note, you don't need scala installed at all to run sbt. The version of scala is set in the repo. It totally ignores any local state other than the local ivy/maven cache which may be used.